[dpdk-users] DPDK weird forwarding loss

[Zhihan Jiang]

Hello,

My hardware is HP Proliant DL360 Gen9 with Intel 82599 NIC, 40 cores.
Traffic generator is IXIA Optixia XM2, using IXNetwork.

I use CentOS 6.5 on the host machine, creating 15 SR-IOVs (virtual
functions) on each of the 4 ports and attach them to the guest machine
(CentOS 6.7) I created.

Other settings: intel_iommu=on iommu=pt / blacklist ixgbevf on the host
machine / pause frame off / pin all the ports to the same NUMA node &
socket / VM uses CPU on the same NUMA node & socket.

I attached the vfs on the guest machines to the igb_uio and tried to run
l3fwd. Also sending 64 byte packets from IXIA to do the forwarding job. The
expected throughput should be 14.88 Mpps but I am experiencing 6% loss. I
also tried lower rate like 1~11 Mpps and various frame size (128/512/1518),
but there is always ~0.5%- 1% packet loss

The command line for l3fwd is:
./build/l3fwd -c6 -n4 -w [whitelist devices] --socket-mem=1024,0 -- -p3
--config "(0,0,1), (1,0,2)"

To look into the issue I tried to add the statistic interface on l3fwd. I
found that for 64 byte packet, when <11 Mpps I have no forwarding loss,
which means all the packets received by DPDK are forwarded back to IXIA,
also means all the loss occurs between IXIA and the DPDK. For
128/256/512/1518 byte packet, no forwarding loss shows on the statistics.
(also 0 rx loss/ 0 rx error from rte_eth_stats_get function)

After that I tried following tests but none of them worked:
1. change burst size from 32 to 16, 64, 128, 512.
2. change the rx descriptor size from 128 to 512.
3. use DPDK KNI to create virtual NIC and try to use ethtool to listen to
the incoming packet. Still the same percentage of packet loss (~0.5%)
occured.
4. also tried testpmd but still, the same percentage of packet loss.

I am wondering what is causing this weird transmitting loss between IXIA
and DPDK. It looks like the packet loss occurs in the NIC or before DPDK
can even poll it. Does anyone have any suggestion of checking why this loss
occurs and how I can fix it?

P.S. I have read the threads about the similar issue but they don't seems
to help.
http://dpdk.info/ml/archives/dev/2014-January/001098.html
http://dpdk.org/ml/archives/dev/2015-September/023480.html
http://dpdk.org/ml/archives/dev/2013-May/000123.html

Thanks so much for any help.
Jack