[dpdk-dev] ixgbevf does not recover from pf reset

[Wu, Jingjing]

Hi, David

We also noticed this issue before. And we are planning to fix this issue.
And the patch for i40e is ready:
http://dpdk.org/dev/patchwork/patch/9832/
http://dpdk.org/dev/patchwork/patch/9833/

The solution is that: PF reset interrupt will be captured by DPDK VF, then DPDK
Call the application's callback function to reset the VF device. For i40e device,
application need to detach and re-attach the device.

For ixgbe, I think the solution would be similar.

Thanks
Jingjing

> -----Original Message-----
> From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of David Marchand
> Sent: Friday, December 11, 2015 10:47 PM
> To: Zhang, Helin; Ananyev, Konstantin
> Cc: dev at dpdk.org
> Subject: [dpdk-dev] ixgbevf does not recover from pf reset
> 
> Hello Helin, Konstantin,
> 
> I noticed this issue quite some time ago (maybe around dpdk-1.3.0, not
> sure) but had no time to investigate/report.
> I hit it again with dpdk-2.2.0-rc2, so maybe all dpdk versions are impacted.
> 
> The test is quite simple :
> - send icmp packets from a system (tn) to itself using two 10G interfaces
> connected to two VF ports of a dual port 82599 nic,
> - run testpmd in a vm owning those two VF ports,
> - reset a pf interface in the host,
> - then no connectivity on the associated VF port unless it is completely
> restarted.
> 
> In my previous test, I think I also unplugged the fiber then replugged
> (without touching the pf interface status on the host) and observed the
> same issue.
> 
> I looked at ixgbevf pmd, but I can't find a place where the PF asking for reset
> would be handled except at devinit.
> 
> 
> 
> Here is the full description of my test :
> 
> The system owning the two VF ports is a ubuntu 14.04 virtual machine
> (3.13.0-71-generic kernel).
> The host running this virtual machine is running ubuntu 14.04 as well (but
> kernel is newer 3.19.0-39-generic).
> 
> 
> - host setup :
> echo 3072
> >/sys/devices/system/node/node0/hugepages/hugepages-
> 2048kB/nr_hugepages
> echo 3072
> >/sys/devices/system/node/node1/hugepages/hugepages-
> 2048kB/nr_hugepages
> mkdir -p
> /mnt/huge
> mount -t hugetlbfs none /mnt/huge
> 
> echo 1 >
> "/sys/bus/pci/devices/0000:83:00.0/sriov_numvfs"
> bridge link set dev ixgbe0 hwmode veb
> ip link set dev ixgbe0 vf 0 mac 00:09:c0:0e:4e:64 ip link set dev ixgbe0 vf 0
> spoofchk off ip link set dev ixgbe0 mtu 9000 ip link set dev ixgbe0 up ip link
> set dev ixgbe0 address 00:09:c0:0e:4e:64
> 
> echo 1 >
> "/sys/bus/pci/devices/0000:83:00.1/sriov_numvfs"
> bridge link set dev ixgbe1 hwmode veb
> ip link set dev ixgbe1 vf 0 mac 00:09:c0:0e:4e:65 ip link set dev ixgbe1 vf 0
> spoofchk off ip link set dev ixgbe1 mtu 9000 ip link set dev ixgbe1 up ip link
> set dev ixgbe1 address 00:09:c0:0e:4e:65
> 
> qemu-system-x86_64 -k fr -daemonize -S --enable-kvm -m 3G -mem-path
> /mnt/huge -cpu host -smp cores=6,threads=2,sockets=1 -serial
> telnet::53714,server,nowait -serial null -monitor telnet::36576,server,nowait
> -device
> virtio-net,mac=DE:AD:DE:01:02:03,netdev=user.0,addr=03 -netdev
> user,id=user.0 -device pci-assign,host=0000:83:10.0,addr=04 -device
> pci-assign,host=0000:83:10.1,addr=05 -hda ubuntu-14.04-template.qcow2 -
> snapshot -vga none -display none
> 
> 
> - guest setup :
> echo 1024 >
> /sys/devices/system/node/node0/hugepages/hugepages-
> 2048kB/nr_hugepages
> mkdir -p
> /mnt/huge
> mount -t hugetlbfs none /mnt/huge
> 
> modprobe uio
> modprobe igb_uio
> echo 0000:00:04.0 > /sys/m/bus/pci/drivers/ixgbevf/unbind
> echo 0000:00:05.0 > /sys/m/bus/pci/drivers/ixgbevf/unbind
> echo 8086 10ed > /sys/bus/pci/drivers/igb_uio/new_id
> 
> testpmd -w0000:00:04.0 -w0000:00:05.0 --huge-dir=/mnt/huge -n 4 -c 0xe -- -i
> [snip] Port 0:
> 00:09:C0:0E:4E:64
> [snip]
> Port 1:
> 00:09:C0:0E:4E:65
> Checking link
> statuses...
> Port 0 Link Up - speed 10000 Mbps -
> full-duplex
> Port 1 Link Up - speed 10000 Mbps -
> full-duplex
> Done
> 
> testpmd>
> start
> 
> 
> - tn setup :
> ip link set dev xaui0 up
> ip link set dev xaui1 up
> ip addr add 1.1.1.1/24 dev xaui0
> arp -i xaui0 -s 1.1.1.2 00:09:c0:0e:4e:64
> 
> 
> 
> - From here, on tn, send icmp packets with a tcpdump running background
> on the other iface :
> ping 1.1.1.2
> PING 1.1.1.2 (1.1.1.2): 56 data
> bytes
> 03:10:15.003047 00:02:bb:a8:0d:10 > 00:09:c0:0e:4e:64, ethertype IPv4
> (0x0800), length 98: IP 1.1.1.1 > 1.1.1.2: icmp 64: echo request seq 1
> 03:10:16.003151 00:02:bb:a8:0d:10 > 00:09:c0:0e:4e:64, ethertype IPv4
> (0x0800), length 98: IP 1.1.1.1 > 1.1.1.2: icmp 64: echo request seq 2
> 03:10:17.003246 00:02:bb:a8:0d:10 > 00:09:c0:0e:4e:64, ethertype IPv4
> (0x0800), length 98: IP 1.1.1.1 > 1.1.1.2: icmp 64: echo request seq 3
> 
> 
> - Ok, now, kick ixgbe0 pf interface down / up on host system.
> ip link set dev ixgbe0 down
> ip link set dev ixgbe0 up
> 
> 
> - On tn, no packet is received anymore, the only workaround I found is to
> stop and restart the port 0 :
> 
> testpmd>
> stop
> [snip]
> Done.
> 
> testpmd> port stop
> all
> Stopping
> ports...
> Checking link
> statuses...
> Port 0 Link
> Down
> Port 1 Link Up - speed 10000 Mbps -
> full-duplex
> Done
> 
> 
> Funny thing here, the link is reported down, while the pf is already back up.
> 
> testpmd> port start all
> PMD: ixgbe_set_rx_function(): Vector rx enabled, please make sure RX burst
> size no less than 4 (port=0).
> Port 0:
> 00:09:C0:0E:4E:64
> PMD: ixgbe_set_rx_function(): Vector rx enabled, please make sure RX burst
> size no less than 4 (port=1).
> Port 1:
> 00:09:C0:0E:4E:65
> Checking link
> statuses...
> Port 0 Link Up - speed 10000 Mbps -
> full-duplex
> Port 1 Link Up - speed 10000 Mbps -
> full-duplex
> Done
> 
> 
> 
> Thanks.
> 
> --
> David Marchand