Рубрики
ovs / openvswitch / dpdk

openvswitch / ovs-vsct / dpdk / debian 11 / ubuntu / modprobe / apt /

Ссылки:

http://core.dpdk.org/doc/
https://doc.dpdk.org/guides/tools/devbind.html


https://ubuntu.com/server/docs/network-dpdk
https://ubuntu.com/server/docs/openvswitch-dpdk
https://ubuntu.com/server/docs/search?q=openvswitch&siteSearch=https%3A%2F%2Fubuntu.com%2Fserver%2Fdocs
https://askubuntu.com/questions/1128111/how-to-install-openvswitch-in-ubuntu-16-04

https://habr.com/ru/company/huawei/blog/653839/
https://habr.com/ru/company/intel/blog/280502/


https://docs.microsoft.com/ru-ru/azure/virtual-network/setup-dpdk

https://selectel.ru/blog/vvedenie-v-dpdk-arxitektura-i-princip-raboty/
https://habr.com/ru/company/selectel/blog/313150/


https://docs.openvswitch.org/en/latest/intro/

https://www.intel.com/content/www/us/en/developer/articles/technical/open-vswitch-with-dpdk-overview.html


https://forum.proxmox.com/threads/tutorial-run-open-vswitch-ovs-dpdk-on-pve-7-0.97116/

https://ovs.readthedocs.io/en/latest/howto/dpdk.html

https://doc.dpdk.org/guides/sample_app_ug/l2_forward_real_virtual.html

https://docs.openvswitch.org/en/latest/intro/install/dpdk/

https://habr.com/ru/post/267591/

http://core.dpdk.org/supported/

Установка openvswitch \ debian:

Установка:
sudo apt update
sudo apt upgrade
sudo apt install openvswitch-switch

Установка зависимостей:
sudo apt install build-essential fakeroot graphviz autoconf automake bzip2 debhelper dh-autoreconf libssl-dev libtool openssl procps python-all python-qt4 python-twisted-conch python-zopeinterface module-assistant dkms make libc6-dev python-argparse uuid-runtime netbase kmod python-twisted-web iproute2 ipsec-tools openvswitch-switch racoon

sudo dpkg-checkbuilddeps

Установка openvswitch dpdk \debian:

Удаляем обычный openvswitch
apt remove openvswitch-* --purge
apt autoremove -f 

Удаляем конфигурации openvswitch: 
rm usr/local/var/run/openvswitch /usr/local/etc/openvswitch /var/log/openvswitch /var/lib/openvswitch -rf 

Устанавливаем openvswitch-switch-dpdk:
apt install openvswitch-switch-dpdk

Пример использование DPDK

0. добавили драйверы с поддержкой dpdk: 
vfio-pci 
uio_pci_generic 
ixgbe

1. Создали бридж в openvswitch-switch-dpdk

2. Добавили в него сетевую карту dpdk-p1

Добавление сетевой карты в управление DPDK:

dpdk-devbind --status - проверяем состояние dpdk (доступные \ подключенные устройства)

Загружаем модули ядра 
modprobe vfio-pci
modprobe uio_pci_generic
modprobe ixgbe

Драйверы для устройств:
vfio-pci
uio_pci_generic
Чтобы понять какой IO нужен для работы с картой, сначала пробуем vfio-pci, а если в dmesg будет ошибка - "привязываем" к uio_pci_generic.
dpdk-devbind --bind=vfio-pci 09:00.0
dmesg 
dpdk-devbind --bind=uio_pci_generic 09:00.0

DPDK device config

/etc/dpdk/interfaces
--------------------
pci 0000:09:00.0 uio_pci_generic
--------------------

dpdk-devbind --status - проверяем состояние dpdk (доступные \ подключенные устройства)

DPDK hugapages

vim /etc/dpdk/dpdk.conf
-----------------------
NR_2M_PAGES=1024
NR_1G_PAGES=4
------------------------

dpdk-hugepages -p 1G --setup 2G

Подготовка OVS для использования DPDK

# Enable DPDK support
ovs-vsctl set Open_vSwitch . "other_config:dpdk-init=true"
# run on core 1 only
ovs-vsctl set Open_vSwitch . "other_config:dpdk-lcore-mask=0x1"
# allocate 2G huge pages
ovs-vsctl set Open_vSwitch . "other_config:dpdk-socket-mem=2048"
# enable vhost-user-client IOMMU support
ovs-vsctl set Open_vSwitch . "other_config:vhost-iommu-support=true"
# restart OVS
systemctl restart ovs-vswitchd.service

# problem 
ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true
# restart OVS
systemctl restart ovs-vswitchd.service

Создание br0 и добавление dpdk-p0

ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev
ovs-vsctl add-port br0 dpdk-p0 -- set Interface dpdk-p0 type=dpdk options:dpdk-devargs=0000:09:00.1

PMD Thread Statistics:

To show current stats:
ovs-appctl dpif-netdev/pmd-stats-show

To clear previous stats:
ovs-appctl dpif-netdev/pmd-stats-clear

Port/RXQ Assigment to PMD Threads:

To show port/rxq assignment:
ovs-appctl dpif-netdev/pmd-rxq-show

To change default rxq assignment to pmd threads, rxqs may be manually pinned to desired cores using:
ovs-vsctl set Interface  other_config:pmd-rxq-affinity=

where:
 is a CSV list of : values

For example:
$ ovs-vsctl set interface dpdk-p0 options:n_rxq=4 other_config:pmd-rxq-affinity="0:3,1:7,3:8"

Adding an OVS port to a KVM guest:

# create a directory for vhost-user sockets
mkdir -p /var/run/vhostuserclient

# add a vhost-user-client port
ovs-vsctl add-port br0 vhost-user-1 -- set Interface vhost-user-1 type=dpdkvhostuserclient "options:vhost-server-path=/var/run/vhostuserclient/vhost-user-client-1"

# vm setup
- Ensure the VM instance is powered off.
- Change the machine type to `q35`.
- Enable NUMA on guest CPU.
- Edit `/etc/pve/qemu-server/.conf`, add the following lines to add the port to guest with mac address `00:00:00:00:00:01` and enable hugepages and (optional) vIOMMU on the guest:
```
args: -machine q35+pve0,kernel_irqchip=split -device intel-iommu,intremap=on,caching-mode=on -chardev socket,id=char1,path=/var/run/vhostuserclient/vhost-user-client-1,server=on -netdev type=vhost-user,id=mynet1,chardev=char1,vhostforce=on -device virtio-net-pci,mac=00:00:00:00:00:01,netdev=mynet1
hugepages: 1024
```