Ссылки:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
http://core.dpdk.org/doc/ https://doc.dpdk.org/guides/tools/devbind.html https://ubuntu.com/server/docs/network-dpdk https://ubuntu.com/server/docs/openvswitch-dpdk https://ubuntu.com/server/docs/search?q=openvswitch&siteSearch=https%3A%2F%2Fubuntu.com%2Fserver%2Fdocs https://askubuntu.com/questions/1128111/how-to-install-openvswitch-in-ubuntu-16-04 https://habr.com/ru/company/huawei/blog/653839/ https://habr.com/ru/company/intel/blog/280502/ https://docs.microsoft.com/ru-ru/azure/virtual-network/setup-dpdk https://selectel.ru/blog/vvedenie-v-dpdk-arxitektura-i-princip-raboty/ https://habr.com/ru/company/selectel/blog/313150/ https://docs.openvswitch.org/en/latest/intro/ https://www.intel.com/content/www/us/en/developer/articles/technical/open-vswitch-with-dpdk-overview.html https://forum.proxmox.com/threads/tutorial-run-open-vswitch-ovs-dpdk-on-pve-7-0.97116/ https://ovs.readthedocs.io/en/latest/howto/dpdk.html https://doc.dpdk.org/guides/sample_app_ug/l2_forward_real_virtual.html https://docs.openvswitch.org/en/latest/intro/install/dpdk/ |
Установка openvswitch \ debian:
|
1 2 3 4 5 6 7 8 9 |
Установка: sudo apt update sudo apt upgrade sudo apt install openvswitch-switch Установка зависимостей: sudo apt install build-essential fakeroot graphviz autoconf automake bzip2 debhelper dh-autoreconf libssl-dev libtool openssl procps python-all python-qt4 python-twisted-conch python-zopeinterface module-assistant dkms make libc6-dev python-argparse uuid-runtime netbase kmod python-twisted-web iproute2 ipsec-tools openvswitch-switch racoon sudo dpkg-checkbuilddeps |
Установка openvswitch dpdk \debian:
|
1 2 3 4 5 6 7 8 9 |
Удаляем обычный openvswitch apt remove openvswitch-* --purge apt autoremove -f Удаляем конфигурации openvswitch: rm usr/local/var/run/openvswitch /usr/local/etc/openvswitch /var/log/openvswitch /var/lib/openvswitch -rf Устанавливаем openvswitch-switch-dpdk: apt install openvswitch-switch-dpdk |
Пример использование DPDK
|
1 2 3 4 5 6 7 8 |
0. добавили драйверы с поддержкой dpdk: vfio-pci uio_pci_generic ixgbe 1. Создали бридж в openvswitch-switch-dpdk 2. Добавили в него сетевую карту dpdk-p1 |
Добавление сетевой карты в управление DPDK:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
dpdk-devbind --status - проверяем состояние dpdk (доступные \ подключенные устройства) Загружаем модули ядра modprobe vfio-pci modprobe uio_pci_generic modprobe ixgbe Драйверы для устройств: vfio-pci uio_pci_generic Чтобы понять какой IO нужен для работы с картой, сначала пробуем vfio-pci, а если в dmesg будет ошибка - "привязываем" к uio_pci_generic. dpdk-devbind --bind=vfio-pci 09:00.0 dmesg dpdk-devbind --bind=uio_pci_generic 09:00.0 |
DPDK device config
|
1 2 3 4 5 6 |
/etc/dpdk/interfaces -------------------- pci 0000:09:00.0 uio_pci_generic -------------------- dpdk-devbind --status - проверяем состояние dpdk (доступные \ подключенные устройства) |
DPDK hugapages
|
1 2 3 4 5 6 7 |
vim /etc/dpdk/dpdk.conf ----------------------- NR_2M_PAGES=1024 NR_1G_PAGES=4 ------------------------ dpdk-hugepages -p 1G --setup 2G |
Подготовка OVS для использования DPDK
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
# Enable DPDK support ovs-vsctl set Open_vSwitch . "other_config:dpdk-init=true" # run on core 1 only ovs-vsctl set Open_vSwitch . "other_config:dpdk-lcore-mask=0x1" # allocate 2G huge pages ovs-vsctl set Open_vSwitch . "other_config:dpdk-socket-mem=2048" # enable vhost-user-client IOMMU support ovs-vsctl set Open_vSwitch . "other_config:vhost-iommu-support=true" # restart OVS systemctl restart ovs-vswitchd.service # problem ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true # restart OVS systemctl restart ovs-vswitchd.service |
Создание br0 и добавление dpdk-p0
|
1 2 |
ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev ovs-vsctl add-port br0 dpdk-p0 -- set Interface dpdk-p0 type=dpdk options:dpdk-devargs=0000:09:00.1 |
PMD Thread Statistics:
|
1 2 3 4 5 |
To show current stats: ovs-appctl dpif-netdev/pmd-stats-show To clear previous stats: ovs-appctl dpif-netdev/pmd-stats-clear |
Port/RXQ Assigment to PMD Threads:
|
1 2 3 4 5 6 7 8 9 10 11 |
To show port/rxq assignment: ovs-appctl dpif-netdev/pmd-rxq-show To change default rxq assignment to pmd threads, rxqs may be manually pinned to desired cores using: ovs-vsctl set Interface <iface> other_config:pmd-rxq-affinity=<rxq-affinity-list> where: <rxq-affinity-list> is a CSV list of <queue-id>:<core-id> values For example: $ ovs-vsctl set interface dpdk-p0 options:n_rxq=4 other_config:pmd-rxq-affinity="0:3,1:7,3:8" |
Adding an OVS port to a KVM guest:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
# create a directory for vhost-user sockets mkdir -p /var/run/vhostuserclient # add a vhost-user-client port ovs-vsctl add-port br0 vhost-user-1 -- set Interface vhost-user-1 type=dpdkvhostuserclient "options:vhost-server-path=/var/run/vhostuserclient/vhost-user-client-1" # vm setup - Ensure the VM instance is powered off. - Change the machine type to `q35`. - Enable NUMA on guest CPU. - Edit `/etc/pve/qemu-server/<ID>.conf`, add the following lines to add the port to guest with mac address `00:00:00:00:00:01` and enable hugepages and (optional) vIOMMU on the guest: ``` args: -machine q35+pve0,kernel_irqchip=split -device intel-iommu,intremap=on,caching-mode=on -chardev socket,id=char1,path=/var/run/vhostuserclient/vhost-user-client-1,server=on -netdev type=vhost-user,id=mynet1,chardev=char1,vhostforce=on -device virtio-net-pci,mac=00:00:00:00:00:01,netdev=mynet1 hugepages: 1024 ``` |