[dpdk-dev] DPDK delaying individual packets infinitely

[Dmitry Vyal]

Hello list,

For some time I've been writing a custom packet generator coupled with a 
packet receiver using DPDK.

It works greatly when used for generating  millions of packets, but 
unexpectedly It has troubles generating a several dozens of packets or so.

The application consists of two threads, first one sends the packets 
from one port and second receives them on another. These are 2 ports of 
four-ports 1Gb NIC. It's detected as Intel Corporation 82576 Gigabit 
Network Connection (rev 01). Ports are connected with a patch cord.

My experiment runs like the following:
After initializing NICs both generator and receiver wait for 1 second.
Generator sends N packets by calling rte_eth_tx_burst for each 
individual packet. It waits for 100000000 cpu ticks between bursts. 
rte_eth_tx_burst reports all packets are sent.
Receiver repeats calling  rte_eth_rx_burst and waits for 50000000 ticks 
if the function returns zero. After generators sends all the packets, 
the receiver continues polling for several seconds.

I'm observing the following behavior:

If N is small, say 20, than no packets a received. See the logs. 
Receiver prints Z when it got zero packets and R if it got at least one 
packet.

Starting experiment
receiver 0 started main loop
generator 0 sitting on socket 0 is waiting for experiment start
generator 0 started main loop
Zgenerating mbuf for file 0, port 0, addr 0 on socket 0
free_count on pool 0 = 100000000
sent     1 in queue 0 freed     0
ZZgenerating mbuf for file 0, port 0, addr 1 on socket 0
sent     1 in queue 0 freed     0
ZZgenerating mbuf for file 0, port 0, addr 2 on socket 0
sent     1 in queue 0 freed     0
ZZgenerating mbuf for file 0, port 0, addr 3 on socket 0
sent     1 in queue 0 freed     0
ZZgenerating mbuf for file 0, port 0, addr 4 on socket 0
sent     1 in queue 0 freed     0
ZZgenerating mbuf for file 0, port 0, addr 5 on socket 0
sent     1 in queue 0 freed     0
ZZgenerating mbuf for file 0, port 0, addr 6 on socket 0
sent     1 in queue 0 freed     0
ZZgenerating mbuf for file 0, port 0, addr 7 on socket 0
sent     1 in queue 0 freed     0
ZZgenerating mbuf for file 0, port 0, addr 8 on socket 0
sent     1 in queue 0 freed     0
ZZgenerating mbuf for file 0, port 0, addr 9 on socket 0
sent     1 in queue 0 freed     0
ZZgenerating mbuf for file 0, port 0, addr 10 on socket 0
sent     1 in queue 0 freed     0
ZZgenerating mbuf for file 0, port 0, addr 11 on socket 0
sent     1 in queue 0 freed     0
ZZgenerating mbuf for file 0, port 0, addr 12 on socket 0
sent     1 in queue 0 freed     0
ZZgenerating mbuf for file 0, port 0, addr 13 on socket 0
sent     1 in queue 0 freed     0
ZZgenerating mbuf for file 0, port 0, addr 14 on socket 0
sent     1 in queue 0 freed     0
ZZgenerating mbuf for file 0, port 0, addr 15 on socket 0
sent     1 in queue 0 freed     0
ZZgenerating mbuf for file 0, port 0, addr 16 on socket 0
sent     1 in queue 0 freed     0
ZZgenerating mbuf for file 0, port 0, addr 17 on socket 0
sent     1 in queue 0 freed     0
ZZgenerating mbuf for file 0, port 0, addr 18 on socket 0
sent     1 in queue 0 freed     0
ZZgenerating mbuf for file 0, port 0, addr 19 on socket 0
sent     1 in queue 0 freed     0
Stopping after reaching 20 packets limit
ZZwaiting for receivers to 
stopZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ********* 
Statistics:
Seconds elapsed: 6.123671


If I made N bigger, say 25, than all of a sudden some packets a 
received, like so:


Starting experiment
receiver 0 started main loop
generator 0 sitting on socket 0 is waiting for experiment start
generator 0 started main loop
generating mbuf for file 0, port 0, addr 0 on socket 0
Zfree_count on pool 0 = 100000000
sent     1 in queue 0 freed     0
ZZgenerating mbuf for file 0, port 0, addr 1 on socket 0
sent     1 in queue 0 freed     0
ZZgenerating mbuf for file 0, port 0, addr 2 on socket 0
sent     1 in queue 0 freed     0
ZZgenerating mbuf for file 0, port 0, addr 3 on socket 0
sent     1 in queue 0 freed     0
ZZgenerating mbuf for file 0, port 0, addr 4 on socket 0
sent     1 in queue 0 freed     0
ZZgenerating mbuf for file 0, port 0, addr 5 on socket 0
sent     1 in queue 0 freed     0
ZZgenerating mbuf for file 0, port 0, addr 6 on socket 0
sent     1 in queue 0 freed     0
ZZgenerating mbuf for file 0, port 0, addr 7 on socket 0
sent     1 in queue 0 freed     0
ZZgenerating mbuf for file 0, port 0, addr 8 on socket 0
sent     1 in queue 0 freed     0
ZZgenerating mbuf for file 0, port 0, addr 9 on socket 0
sent     1 in queue 0 freed     0
ZZgenerating mbuf for file 0, port 0, addr 10 on socket 0
sent     1 in queue 0 freed     0
ZZgenerating mbuf for file 0, port 0, addr 11 on socket 0
sent     1 in queue 0 freed     0
ZZgenerating mbuf for file 0, port 0, addr 12 on socket 0
sent     1 in queue 0 freed     0
ZZgenerating mbuf for file 0, port 0, addr 13 on socket 0
sent     1 in queue 0 freed     0
ZZgenerating mbuf for file 0, port 0, addr 14 on socket 0
sent     1 in queue 0 freed     0
ZZgenerating mbuf for file 0, port 0, addr 15 on socket 0
sent     1 in queue 0 freed     0
ZZgenerating mbuf for file 0, port 0, addr 16 on socket 0
sent     1 in queue 0 freed     0
ZZgenerating mbuf for file 0, port 0, addr 17 on socket 0
sent     1 in queue 0 freed     0
ZZgenerating mbuf for file 0, port 0, addr 18 on socket 0
sent     1 in queue 0 freed     0
ZZgenerating mbuf for file 0, port 0, addr 19 on socket 0
sent     1 in queue 0 freed     0
ZZgenerating mbuf for file 0, port 0, addr 20 on socket 0
sent     1 in queue 0 freed     0
Rtotal received packets on queue 0: 21; received: 21; zero_iters: 41
Zgenerating mbuf for file 0, port 0, addr 21 on socket 0
free_count on pool 0 = 6113
sent     1 in queue 0 freed     0
Rtotal received packets on queue 0: 22; received: 1; zero_iters: 1
Zgenerating mbuf for file 0, port 0, addr 22 on socket 0
sent     1 in queue 0 freed     0
Rtotal received packets on queue 0: 23; received: 1; zero_iters: 1
Zgenerating mbuf for file 0, port 0, addr 23 on socket 0
sent     1 in queue 0 freed     0
Rtotal received packets on queue 0: 24; received: 1; zero_iters: 1
Zgenerating mbuf for file 0, port 0, addr 24 on socket 0
sent     1 in queue 0 freed     0
Stopping after reaching 25 packets limit
Rtotal received packets on queue 0: 25; received: 1; zero_iters: 1
Zwaiting for receivers to 
stopZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ********* 
Statistics:
Seconds elapsed: 6.293315


So looks like DPDK is buffering first 21 packets and doesn't flush the 
buffer in the first run even if I wait for several seconds after last 
packet is sent. In second run we see it forwards packets one by one 
after first burst of size 21.

I tried on DPDK-1.3.1 and DPDK-1.5.1 observing similar results. Is it 
expected behavior?