Using the openvswitch+dpdk method, for the ixgbe driver type network card added to the bridge, when two VM virtual machine interfaces are sent, the netperf tool is used to send TCP_ After a period of time, the message of CRR will undergo rte_ Eth_ Tx_ Burst stops sending packets and discovers rte through localization_ Pktmbuf_ Alloc cannot allocate mbuf. Openvswitch: DPDK: 22.11 Network card: 82599ES Network card queue: 8 Network card receiving descriptor: 4096
This sounds like a packet leak. So it could be an issue in the ixgbe driver (bug in the code), or in the application (bug in the code or wrong configuration). I suspect the current bug description won't be enough to try to reproduce this issue, so please share more information: - did you apply patches on OVS? in any case, which version of OVS are you using? - did you apply patches on dpdk? please share this version too, - what configuration did you use for OVS? Assigning to the ixgbe driver maintainer for now.
(In reply to David Marchand from comment #1) > This sounds like a packet leak. > So it could be an issue in the ixgbe driver (bug in the code), or in the > application (bug in the code or wrong configuration). > > > I suspect the current bug description won't be enough to try to reproduce > this issue, so please share more information: > > - did you apply patches on OVS? in any case, which version of OVS are you > using? > - did you apply patches on dpdk? please share this version too, > - what configuration did you use for OVS? > > > Assigning to the ixgbe driver maintainer for now. (ovsdpdk-vswitchd)[root@node61 /]# lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 80 On-line CPU(s) list: 0-79 Thread(s) per core: 2 Core(s) per socket: 20 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel BIOS Vendor ID: Intel(R) Corporation CPU family: 6 Model: 106 Model name: Intel(R) Xeon(R) Silver 4316 CPU @ 2.30GHz BIOS Model name: Intel(R) Xeon(R) Silver 4316 CPU @ 2.30GHz Stepping: 6 CPU MHz: 3400.000 CPU max MHz: 3400.0000 CPU min MHz: 800.0000 BogoMIPS: 4600.00 Virtualization: VT-x L1d cache: 48K L1i cache: 32K L2 cache: 1280K L3 cache: 30720K NUMA node0 CPU(s): 0-19,40-59 NUMA node1 CPU(s): 20-39,60-79 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 invpcid_single intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect wbnoinvd dtherm ida arat pln pts avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid fsrm md_clear pconfig flush_l1d arch_capabilities (ovsdpdk-vswitchd)[root@node61 /]# ovs-vsctl list open _uuid : 68288537-9941-4789-9555-e1fc2e2834d7 bridges : [07155ac5-825c-4a52-8a10-4b7bd5e90265, 97de9721-274e-4538-8150-882df8f1149b, 9a14f379-6914-4255-9274-91751b9b7848] cur_cfg : 71 datapath_types : [netdev, system] datapaths : {netdev=267c7904-6f77-405b-9577-f26c44c8649f, system=60e141bc-2211-49bf-8225-a44908ad262f} db_version : [] dpdk_initialized : true dpdk_version : "DPDK 21.11.2" external_ids : {hostname=testnode, ovn-bridge-datapath-type=netdev, ovn-bridge-mappings="public:br-ex", ovn-cms-options=enable-chassis-as-gw, ovn-encap-ip="192.168.50.61", ovn-encap-type=geneve, ovn-remote="tcp:100.180.150.152:6642", ovn-remote-probe-interval="60000", system-id="5fbe081604db8aa0c8476b62f258f3fb"} iface_types : [bareudp, dpdk, dpdkvhostuser, dpdkvhostuserclient, erspan, geneve, gre, gtpu, internal, ip6erspan, ip6gre, lisp, patch, stt, system, tap, vxlan] manager_options : [] next_cfg : 71 other_config : {dpdk-hugepage-dir="/dev/hugepages", dpdk-init=True, dpdk-lcore-mask="0x3", dpdk-socket-mem="4096,4096", pmd-cpu-mask="0xffff0ffff0ffff0ffff0", vlan-limit="0"} ovs_version : [] ssl : [] statistics : {} system_type : [] system_version : [] (ovsdpdk-vswitchd)[root@node61 /]# lscpu Bridge br-dpdk datapath_type: netdev Port bond2 Interface ens29f1 type: dpdk options: {dpdk-devargs="0000:e3:00.1", n_rxq="8", n_rxq_desc="4096", n_txq="8", n_txq_desc="4096"} Interface ens29f0 type: dpdk options: {dpdk-devargs="0000:e3:00.0", n_rxq="8", n_rxq_desc="4096", n_txq="8", n_txq_desc="4096"} Port br-dpdk Interface br-dpdk type: internal (ovsdpdk-vswitchd)[root@node61 /]# ovs-vsctl -V ovs-vsctl (Open vSwitch) 3.0.5 DB Schema 8.3.0 (ovsdpdk-vswitchd)[root@node61 /]# dpdk-devbind.py -s Network devices using DPDK-compatible driver ============================================ 0000:cb:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe 0000:cb:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe 0000:e3:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=vfio-pci unused=i40e 0000:e3:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=vfio-pci unused=i40e
and also， This issue also exists on the i40e network card
I've encountered a similar issue before, and I think there might be a problem with the logic when initializing the mbuf. So, I modified the logic here in OVS to dynamically configure the mbuf based on the number of physical ports bound by DPDK. The modification logic I provided might be helpful for you. You may consider adapting it here. diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c index 58240d26c5e6559f8362d81ca1ad9e03e4deb459..7966af2cd57633f96a05c15718c22f48c9f049e3 100644 --- a/lib/netdev-dpdk.c +++ b/lib/netdev-dpdk.c @@ -703,6 +703,7 @@ static uint32_t dpdk_calculate_mbufs(struct netdev_dpdk *dev, int mtu) { uint32_t n_mbufs; + int port_num; if (!per_port_memory) { /* Shared memory are being used. @@ -725,10 +726,17 @@ dpdk_calculate_mbufs(struct netdev_dpdk *dev, int mtu) * + <packets in the pmd threads> * + <additional memory for corner cases> */ - n_mbufs = dev->requested_n_rxq * dev->requested_rxq_size - + dev->requested_n_txq * dev->requested_txq_size - + MIN(RTE_MAX_LCORE, dev->requested_n_rxq) * NETDEV_MAX_BURST - + MIN_NB_MBUF; + port_num = dpdk_eth_port_num(); + if (port_num <= 1) + n_mbufs = dev->requested_n_rxq * dev->requested_rxq_size + + dev->requested_n_txq * dev->requested_txq_size + + MIN(RTE_MAX_LCORE, dev->requested_n_rxq) * NETDEV_MAX_BURST + + MIN_NB_MBUF; + else + n_mbufs = dev->requested_n_rxq * dev->requested_rxq_size + + dev->requested_n_txq * dev->requested_txq_size * port_num + + MIN(RTE_MAX_LCORE, dev->requested_n_rxq) * NETDEV_MAX_BURST + + MIN_NB_MBUF; } return n_mbufs;
@reporter If this proposed patch changes something in your setup, then maybe the heuristic for creating a *per port* mempool can be enhanced but it is hard to tell what is problematic with just the initial report. There was no activity by Intel on this bug on the DPDK side. If this issue still affects you, I recommend creating a OVS issue at https://github.com/openvswitch/ovs-issues/issues/. @Jun Wang About the change itself, this part of the code provides the size of a mempool created for a single port. Adding the number of DPDK ports in the formula is just a workaround which results in a higher mbuf count, but without a relation with the actual issue.
I analyzed based on this code comment. Here it seems to want to count the txqs of all ports, but in reality the code logic here only initializes one port. So I changed it to dynamically initialize based on the number of physical ports, and it indeed resolves the same issue I encountered. /* Per port memory is being used. * XXX: rough estimation of number of mbufs required for this port: * <packets required to fill the device rxqs> * + <packets that could be stuck on other ports txqs> * + <packets in the pmd threads> * + <additional memory for corner cases> */ The current code is: dev->requested_n_txq * dev->requested_txq_size I believe what is actually desired is: dev->requested_n_txq * dev->requested_txq_size * port_num
other_config : {dpdk-hugepage-dir="/dev/hugepages", dpdk-init=True, dpdk-lcore-mask="0x3", dpdk-socket-mem="4096,4096", pmd-cpu-mask="0xffff0ffff0ffff0ffff0", vlan-limit="0"} Per port mempool is not used in this setup. It would say 'per-port-memory="true"' if used, so size of per-port allocation is not relevant to this issue. So if not per-port than it is using shared mempools. The minimum size it will run with if there is not enough memory for more is MIN_NB_MBUF = 16K. 2 ports * 8 Rx queues * 4K mbufs per queue = 64K mbuffs So it could be that your config is requiring more mbuffs than are available. Suggest to set 'ovs-appctl vlog/set netdev_dpdk:dbg' before adding the ports and it will give info on the mempools creation. Also, you can run, 'ovs-appctl netdev-dpdk/get-mempool-info' options: {dpdk-devargs="0000:e3:00.1", n_rxq="8", n_rxq_desc="4096", n_txq="8", n_txq_desc="4096"} The default Rx descriptor size is 2048, unless you have a need to have 4096 above, then you could remove n_rxq_desc="4096" and it would half the amount of mbuffs need. As an aside, n_txq is not used by OVS, that is calculated based on datapath cores. Setting it is a noop.