Hi, Sometimes XL710 VF cannot Rx packets when startup with DPDK 22.03. It have been worked well before DPDK 22.03. 1. Reproduction : Send arp flooding packets in same subnet during DPDK driver initialization, it happens randomly. #!/bin/bash while [ true ] do sudo arping -i ethvf -w 0 -B -q done Rx stopped after "fault addr 0 [fault reason 05] PTE Write access is not set" message found. 2. Test environment 1) OS : Ubuntu 18.04 2) NIC : Intel XL710 SRIOV-VF 3) Used Kernel driver and F/W version Driver version: 2.1.14-k firmware-version: 7.10 0x800075e6 19.5.12 -> It happened other several other different drivers and F/W versions 3. Syslog May 23 03:55:32 xx[31168]: EAL: VFIO support initialized May 23 03:55:32 xx[31168]: EAL: Using IOMMU type 1 (Type 1) May 23 03:55:32 kernel: vfio-pci 0000:3b:02.3: enabling device (0000 -> 0002) May 23 03:55:33 xx[31168]: EAL: Probe PCI driver: net_iavf (8086:154c) device: 0000:3b:02.3 (socket 0) May 23 03:55:33 kernel: vfio-pci 0000:3b:02.4: enabling device (0000 -> 0002) May 23 03:55:33 xx[31168]: EAL: Probe PCI driver: net_iavf (8086:154c) device: 0000:3b:02.4 (socket 0) May 23 03:55:33 xx[31168]: TELEMETRY: No legacy callbacks, legacy socket not created May 23 03:55:33 xx[31168]: iavf_configure_queues(): RXDID[22] is not supported, request default RXDID[1] in Queue[0] May 23 03:55:34 kernel: i40e 0000:3b:00.0: Unprivileged VF 3 is attempting to configure promiscuous mode May 23 03:55:34 kernel: i40e 0000:3b:00.0: Unprivileged VF 3 is attempting to configure promiscuous mode May 23 03:55:34 xx[31168]: iavf_configure_queues(): RXDID[22] is not supported, request default RXDID[1] in Queue[0] May 23 03:55:34 kernel: DMAR: DRHD: handling fault status reg 302 May 23 03:55:34 kernel: DMAR: [DMA Write] Request device [3b:02.4] fault addr 0 [fault reason 05] PTE Write access is not set May 23 03:55:34 kernel: i40e 0000:3b:00.0: Unprivileged VF 4 is attempting to configure promiscuous mode May 23 03:55:34 kernel: DMAR: [DMA Write] Request device [3b:02.4] fault addr 0 [fault reason 05] PTE Write access is not set May 23 03:55:34 kernel: DMAR: DRHD: handling fault status reg 500 4
Hi, It always reproducible with DPDK 22.03 version(iavf used). No issue with DPDK 21.02(i40evf used) version. It would be issue on new iavf driver.
please offer the reproduce steps with standard DPDK API like testpmd, l3fwd and other examples.
Hi Kevin Here are reproduction steps. I tested it with 1 Intel XL710 40G NIC. Step1. Create 3 SRIOV-VF NICs in XL710 NIC. Step2. Bind 2 NICs for DPDK testpmd, 1 for Linux Kernel driver(enp59s13f6) Step3. Run flood ping to broadcast address with SRIOV-VF NIC It'll broadcast packets to 2 DPDK NICs. Linux# sudo ping -I enp59s13f6 -b -q -f 255.255.255.255 Step4. Run testpmd I used 4 cores and 2 NICs Linux#sudo ./dpdk-testpmd -l 72,74,76,78 -a 3b:0d.2 -a 3b:0d.3 -n 4 -- -i Step5. Check forwarding stats in testpmd CLI All packets will be "Rx-dropped". testpmd> show fwd stats all ---------------------- Forward statistics for port 0 ---------------------- RX-packets: 451 RX-dropped: 422 RX-total: 873 TX-packets: 32 TX-dropped: 0 TX-total: 32 ---------------------------------------------------------------------------- ---------------------- Forward statistics for port 1 ---------------------- RX-packets: 485 RX-dropped: 390 RX-total: 875 TX-packets: 32 TX-dropped: 0 TX-total: 32 ---------------------------------------------------------------------------- +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++ RX-packets: 936 RX-dropped: 812 RX-total: 1748 TX-packets: 64 TX-dropped: 0 TX-total: 64 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Since this issue happen when initializing DPDK driver. (There are many DPDK error messages visible.) It does not issue if flood ping done after testpmd started. Step6. Normal case testing Run testpmd first, after then flood ping. In normal case, all packets visible as "Tx-dropped". And no DPDK error messages. testpmd> show fwd stats all ---------------------- Forward statistics for port 0 ---------------------- RX-packets: 1077 RX-dropped: 0 RX-total: 1077 TX-packets: 32 TX-dropped: 0 TX-total: 32 ---------------------------------------------------------------------------- ---------------------- Forward statistics for port 1 ---------------------- RX-packets: 1077 RX-dropped: 0 RX-total: 1077 TX-packets: 32 TX-dropped: 0 TX-total: 32 ---------------------------------------------------------------------------- +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++ RX-packets: 2154 RX-dropped: 0 RX-total: 2154 TX-packets: 64 TX-dropped: 2154 TX-total: 2218 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
I tried to test with the latest version of the dpdk, but your errors did not reappear. I looked at your driver and firmware versions, which seem to be a little old. The following are recommended by dpdk22.03： * Firmware version(PF): 8.30 0x8000a49d 1.2926.0 * Firmware version(VF): 8.50 0x8000b6d9 1.3082.0 * Device id (pf/vf): 8086:1572 / 8086:154c * Driver version: 2.17.15 (i40e)
Hi, It's reproducible in the latest version of DPDK and any version of Linux kernel driver combinations including iavf 4.9.1 and DPDK 23.07 version. Following error messages visible in Linux kernel if broadcast packet received when iavf pmd enable queue. 261863.529942] DMAR: [DMA Write] Request device [3b:0d.1] fault addr 0 [fault reason 05] PTE Write access is not set [261863.544965] DMAR: DRHD: handling fault status reg 602 [261863.545624] DMAR: [DMA Write] Request device [3b:0d.1] fault addr 0 [fault reason 05] PTE Write access is not set [261863.560952] DMAR: DRHD: handling fault status reg 702 [261863.561626] DMAR: [DMA Write] Request device [3b:0d.1] fault addr 0 [fault reason 05] PTE Write access is not set [261863.576950] DMAR: DRHD: handling fault status reg 2 [261863.576959] DMAR: [DMA Write] Request device [3b:0d.1] fault addr 0 [fault reason 05] PTE Write access is not set
This issue is not visible, if force to use old i40e PMD. iavf_ethdev.c static const struct rte_pci_id pci_id_iavf_map[] = { { RTE_PCI_DEVICE(IAVF_INTEL_VENDOR_ID, IAVF_DEV_ID_ADAPTIVE_VF) }, #if 0 /* force to use i40evf */ { RTE_PCI_DEVICE(IAVF_INTEL_VENDOR_ID, IAVF_DEV_ID_VF) }, { RTE_PCI_DEVICE(IAVF_INTEL_VENDOR_ID, IAVF_DEV_ID_VF_HV) }, { RTE_PCI_DEVICE(IAVF_INTEL_VENDOR_ID, IAVF_DEV_ID_X722_VF) }, { RTE_PCI_DEVICE(IAVF_INTEL_VENDOR_ID, IAVF_DEV_ID_X722_A0_VF) }, #endif { .vendor_id = 0, /* sentinel */ }, };
it's reproducible when VF devices in same PF are used by both Linux kernel driver(iavf 4.9.1) and DPDK(dpdk 23.07) in same host.
It's 100% reproducible with flood ping with broadcast address. For example # sudo ping -f -b -q 1.1.1.255 -I enp59s13f5