NIC: one Intel Corporation Ethernet Controller x710 for 10GbE SFP+ with 4 ports (PF) Distribution version: 18.04.1 LTS (Bionic Beaver) Kernel version: 5.4.0-99-generic (also reproduced with 5.4.0-94-generic) DPDK version: 20.11.4 LTS Setup created before launching testpmd: 2 bondings (LAG/LACP) called access_lag and backbone_lag are created, each bond regroups 2 ports. 200 vlan are added to bond access_lag. interface/port_id: PCI-SLOT IFNAME MAC-ADDRESS KMOD DEVICE 0000:8a:00.0 eno12399 68:05:ca:d2:df:2e i40e Intel Corporation Ethernet Controller X710 for 10GbE SFP+ 0000:8a:00.1 eno12409 68:05:ca:d2:df:2f i40e Intel Corporation Ethernet Controller X710 for 10GbE SFP+ 0000:8a:00.2 eno12419 68:05:ca:d2:df:30 i40e Intel Corporation Ethernet Controller X710 for 10GbE SFP+ 0000:8a:00.3 eno12429 68:05:ca:d2:df:31 i40e Intel Corporation Ethernet Controller X710 for 10GbE SFP+ Steps to reproduce: 1/ Create the setup **Initial state root@ubuntu1804hwe:~# ip -c l show 4: eno12399: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qle0 link/ether 68:05:ca:d2:df:2e brd ff:ff:ff:ff:ff:ff 5: eno12409: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qle0 link/ether 68:05:ca:d2:df:2f brd ff:ff:ff:ff:ff:ff 6: eno12419: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qle0 link/ether 68:05:ca:d2:df:30 brd ff:ff:ff:ff:ff:ff 7: eno12429: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qle0 link/ether 68:05:c **setup root@ubuntu1804hwe:~# ip l add backbone_lag type bond mode 802.3ad lacp_rate 1 xmit_hash_policy 2 root@ubuntu1804hwe:~# ip l add access_lag type bond mode 802.3ad lacp_rate 1 xmit_hash_policy 2 root@ubuntu1804hwe:~# ip l set eno12399 master backbone_lag root@ubuntu1804hwe:~# ip l set eno12409 master backbone_lag root@ubuntu1804hwe:~# ip l set eno12419 master access_lag root@ubuntu1804hwe:~# ip l set eno12429 master access_lag root@ubuntu1804hwe:~# ip l set access_lag up root@ubuntu1804hwe:~# ip l set backbone_lag up root@ubuntu1804hwe:~# for i in $(seq 1 200); do ip l add link access_lag name vlan$i type vlan id $i; ip link set vlan$i up ; done root@ubuntu1804hwe:~# ip l show | grep vlan | wc -l 200 root@ubuntu1804hwe:~/dpdk-stable-20.11.4# ip -c l show access_lag 9: access_lag: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 68:05:ca:d2:df:30 brd ff:ff:ff:ff:ff:ff root@ubuntu1804hwe:~/dpdk-stable-20.11.4# ip -c l show backbone_lag 8: backbone_lag: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 68:05:ca:d2:df:2e brd ff:ff:ff:ff:ff:ff root@ubuntu1804hwe:~/dpdk-stable-20.11.4# ip -c l show eno12399 4: eno12399: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master backbone_lag state UP mode DEFAULT group default qlen 1000 link/ether 68:05:ca:d2:df:2e brd ff:ff:ff:ff:ff:ff root@ubuntu1804hwe:~/dpdk-stable-20.11.4# ip -c l show eno12409 5: eno12409: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master backbone_lag state UP mode DEFAULT group default qlen 1000 link/ether 68:05:ca:d2:df:2e brd ff:ff:ff:ff:ff:ff root@ubuntu1804hwe:~/dpdk-stable-20.11.4# ip -c l show eno12419 6: eno12419: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master access_lag state UP mode DEFAULT group default qlen 1000 link/ether 68:05:ca:d2:df:30 brd ff:ff:ff:ff:ff:ff root@ubuntu1804hwe:~/dpdk-stable-20.11.4# ip -c l show eno12429 7: eno12429: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master access_lag state UP mode DEFAULT group default qlen 1000 link/ether 68:05:ca:d2:df:30 brd ff:ff:ff:ff:ff:ff root@ubuntu1804hwe:~/dpdk-stable-20.11.4# ./usertools/dpdk-devbind.py --status Network devices using kernel driver =================================== 0000:8a:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572' if=eno12399 drv=i40e unused=vfio-pci 0000:8a:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572' if=eno12409 drv=i40e unused=vfio-pci 0000:8a:00.2 'Ethernet Controller X710 for 10GbE SFP+ 1572' if=eno12419 drv=i40e unused=vfio-pci 0000:8a:00.3 'Ethernet Controller X710 for 10GbE SFP+ 1572' if=eno12429 drv=i40e unused=vfio-pci 2/ Launch testpmd root@ubuntu1804hwe:~/dpdk-stable-20.11.4# ./usertools/dpdk-devbind.py -u 8a:00.0 8a:00.1 8a:00.2 8a:00.3 root@ubuntu1804hwe:~/dpdk-stable-20.11.4# ./usertools/dpdk-devbind.py --bind=vfio-pci 8a:00.0 8a:00.1 8a:00.2 8a:00.3 root@ubuntu1804hwe:~/dpdk-stable-20.11.4# ./build/app/dpdk-testpmd -l 4,5 -- --total-num-mbufs=65563 -i EAL: Detected 48 lcore(s) EAL: Detected 1 NUMA nodes EAL: Detected static linkage of DPDK EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'VA' EAL: Probing VFIO support... EAL: VFIO support initialized EAL: using IOMMU type 1 (Type 1) EAL: Probe PCI driver: net_i40e (8086:1572) device: 0000:8a:00.0 (socket 0) eth_i40e_dev_init(): Failed to reset pf: -15 ethdev initialisation failed EAL: Releasing pci mapped resource for 0000:8a:00.0 EAL: Calling pci_unmap_resource for 0000:8a:00.0 at 0x1180000000 EAL: Calling pci_unmap_resource for 0000:8a:00.0 at 0x1181000000 EAL: Requested device 0000:8a:00.0 cannot be used EAL: using IOMMU type 1 (Type 1) EAL: Probe PCI driver: net_i40e (8086:1572) device: 0000:8a:00.1 (socket 0) eth_i40e_dev_init(): Failed to reset pf: -15 ethdev initialisation failed EAL: Releasing pci mapped resource for 0000:8a:00.1 EAL: Calling pci_unmap_resource for 0000:8a:00.1 at 0x1181008000 EAL: Calling pci_unmap_resource for 0000:8a:00.1 at 0x1182008000 EAL: Requested device 0000:8a:00.1 cannot be used EAL: using IOMMU type 1 (Type 1) EAL: Probe PCI driver: net_i40e (8086:1572) device: 0000:8a:00.2 (socket 0) i40e_GLQF_reg_init(): i40e device 0000:8a:00.2 changed global register [0x002689a0]. original: 0x00000000, new: 0x00000029 i40e_GLQF_reg_init(): i40e device 0000:8a:00.2 changed global register [0x00268ca4]. original: 0x00001840, new: 0x00009420 i40e_aq_debug_write_global_register(): i40e device 0000:8a:00.2 changed global register [0x0026c7a0]. original: 0xa8, after: 0x28 EAL: Probe PCI driver: net_i40e (8086:1572) device: 0000:8a:00.3 (socket 0) Interactive-mode selected testpmd: create a new mbuf pool <mb_pool_0>: n=65563, size=2176, socket=0 testpmd: preferred mempool ops selected: ring_mp_mc Configuring Port 0 (socket 0) Port 0: 68:05:CA:D2:DF:30 Configuring Port 1 (socket 0) Port 1: 68:05:CA:D2:DF:31 Checking link statuses... Done testpmd> show port summary all Number of available ports: 2 Port MAC Address Name Driver Status Link 0 68:05:CA:D2:DF:30 0000:8a:00.2 net_i40e up 10 Gbps 1 68:05:CA:D2:DF:31 0000:8a:00.3 net_i40e up 10 Gbps ===> ethdev initialization fails for the ports enslaved to bonding access_lag : EAL: Probe PCI driver: net_i40e (8086:1572) device: 0000:8a:00.0 (socket 0) eth_i40e_dev_init(): Failed to reset pf: -15 ethdev initialisation failed EAL: Probe PCI driver: net_i40e (8086:1572) device: 0000:8a:00.1 (socket 0) eth_i40e_dev_init(): Failed to reset pf: -15 ethdev initialisation failed FYI: 1/ Once the bug is triggered we can no longer bind the ports back to the kernel driver i40e, and a reboot is the only way to recover. 2/ The issue is not reproduced if we don’t create the 2 bonds and instead we attach the vlans to one of the interfaces. 3/ This issue was not reproduced with Redhat7 ***Additional information about interface eno12399 (portid 0000:8a:00.0) & eno12409 (portid 0000:8a:00.0): root@ubuntu1804hwe:~# ethtool -i eno12399 driver: i40e version: 2.8.20-k firmware-version: 8.40 0x8000af7e 20.5.13 expansion-rom-version: bus-info: 0000:8a:00.0 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes root@ubuntu1804hwe:~# ethtool eno12399 Settings for eno12399: Supported ports: [ FIBRE ] Supported link modes: 10000baseSR/Full Supported pause frame use: Symmetric Receive-only Supports auto-negotiation: Yes Supported FEC modes: Not reported Advertised link modes: 10000baseSR/Full Advertised pause frame use: No Advertised auto-negotiation: Yes Advertised FEC modes: Not reported Speed: 10000Mb/s Duplex: Full Port: FIBRE PHYAD: 0 Transceiver: internal Auto-negotiation: off Supports Wake-on: g Wake-on: g Current message level: 0x00000007 (7) drv probe link Link detected: no root@ubuntu1804hwe:~# ethtool -i eno12409 driver: i40e version: 2.8.20-k firmware-version: 8.40 0x8000af7e 20.5.13 expansion-rom-version: bus-info: 0000:8a:00.1 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes root@ubuntu1804hwe:~# ethtool eno12409 Settings for eno12409: Supported ports: [ FIBRE ] Supported link modes: 10000baseSR/Full Supported pause frame use: Symmetric Receive-only Supports auto-negotiation: Yes Supported FEC modes: Not reported Advertised link modes: 10000baseSR/Full Advertised pause frame use: No Advertised auto-negotiation: Yes Advertised FEC modes: Not reported Speed: 10000Mb/s Duplex: Full Port: FIBRE PHYAD: 0 Transceiver: internal Auto-negotiation: off Supports Wake-on: g Wake-on: g Current message level: 0x00000007 (7) drv probe link Link detected: no
It seems that it has nothing to do with the dpdk. Have you tried to bind to the i40e without starting testpmd after binding to the vfio-pci? If you look at the dmesg, there should be an error message.
Hi All, No reply for a long time, I'm going to close this ticket for now, so please contact me if you still have questions. Thanks Deng Kaiwen