[dpdk-dev] Polling too often at lower packet rates?

[Ananyev, Konstantin]

> -----Original Message-----
> From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of Aaron Campbell
> Sent: Wednesday, April 08, 2015 5:36 PM
> To: dev at dpdk.org
> Subject: [dpdk-dev] Polling too often at lower packet rates?
> 
> Hi,
> 
> I have a machine with 6 DPDK ports (4 igb, 2 ixgbe), with 1.23Mpps traffic offered to only one of the 10G ports (the other 5 are
> unused).  I also have a program with a pretty standard looking DPDK receive loop, where it calls rte_eth_rx_burst() for each
> configured port.  If I configure the loop to read from all 6 ports, it can read the 1.23Mpps rate with no drops.  If I configure the loop to
> poll only 1 port (the ixgbe receiving the traffic), I lose about 1/3rd of the packets (i.e., the NIC drops ~400Kpps).

That seems a bit strange to see packet drops with such low rate.
Tried it with latest DPDK over 1 ixgbe port (X520-4) , 2.8 GHz IVB cpu:

./dpdk.org/x86_64-native-linuxapp-gcc/app/testpmd -c ff -n 4 --socket-mem=1024,0 -w 0000:04:00.1 -- -i --burst=32
testpmd> start

Tried with 1/1.5/2/3 Mpps  - no packet loss. 

Wonder to what port do you forward your packets?
What stats tells?
Does number of RX packets equal to the number of TX packets?

Konstantin

> 
> Another data point is that if I configure the loop to read from 3 out of the 6 ports, the drop rate is reduced to less than half (i.e., the
> NIC is only dropping ~190Kpps now).  So it seems that in this test, throughput improves by adding NICs, not removing them, which is
> counter-intuitive.  Again, I get no drops when polling all 6 ports.  Note, the burst size is 32.
> 
> I did find a reference to a similar issue in a recent paper
> (http://www.net.in.tum.de/fileadmin/bibtex/publications/papers/ICN2015.pdf), Section III, which states:
> 
> "The DPDK L2FWD application initially only managed to forward 13.8 Mpps in the single direction test at the maximum CPU frequency,
> a similar result can be found in [11]. Reducing the CPU frequency increased the throughput to the expected value of 14.88 Mpps. Our
> investigation of this anomaly revealed that the lack of any processing combined with the fast CPU caused DPDK to poll the NIC too
> often. DPDK does not use interrupts, it utilizes a busy wait loop that polls the NIC until at least one packet is returned. This resulted in
> a high poll rate which affected the throughput. We limited the poll rate to 500,000 poll operations per second (i.e., a batch size of
> about 30 packets) and achieved line rate in the unidirectional test with all frequencies. This effect was only observed with the X520
> NIC, tests with X540 NICs did not show this anomaly.”
> 
> Another reference, from this mailing list last year (http://wiki.dpdk.org/ml/archives/dev/2014-January/001169.html):
> 
> "I suggest you to check average burst sizes on receive queues. Looks like I stumbled upon a similar issue several times. If you are
> calling rte_eth_rx_burst too frequently, NIC begins losing packets no matter how many CPU horse power you have (more you have,
> more it loses, actually). In my case this situation occured when average burst size is less than 20 packets or so. I'm not sure what's the
> reason for this behavior, but I observed it on several applications on Intel 82599 10Gb cards.”
> 
> So I’m wondering if anyone can explain at a lower level what happens when you poll “too often”, and if there are any practical
> workarounds.  We’re using this same program and DPDK version to process 10G line-rate in other scenarios, so I’m confident that the
> overall packet capture architecture is sound.
> 
> -Aaron