[dpdk-dev] Calling rte_eth_rx_burst() multiple times

[Zoltan Kiss]

On 15/10/15 11:43, Younghwan Go wrote:
> Hi Zoltan,
>
> Thanks for the email.
>
> 2015-10-15 오후 7:23에 Zoltan Kiss 이(가) 쓴 글:
>>
>>
>> On 15/10/15 09:32, Younghwan Go wrote:
>>> Hi,
>>>
>>> I'm pretty new to playing with DPDK. I was trying to see if I can always
>>> receive MAX_BURST packets by calling rte_eth_rx_burst() multiple times
>>> on same <port, queue> pair (code shown below). I'm using DPDK-2.1.0 on 2
>>> dual-port Intel 82599ES 10Gbps NICs with Ubuntu 14.04.3 (kernel
>>> 3.13.0-63-generic).
>>>
>>> Since packet processing is slower (~10 Gbps) than pure RX speed (~40
>>> Gbps), I assumed rte_eth_rx_burst() would always receive some number of
>>> packets, eventually filling up MAX_BURST. But for multi-core case (4
>>> CPUs, 4 ports), rte_eth_rx_burst() starts to always return 0 after some
>>> time, causing all cores to be blocked forever. Analyzing the DPDK code
>>> (drivers/net/ixgbe/ixgbe_rxtx.c), I'm seeing that inside
>>> ixgbe_rx_scan_hw_ring() function, "rxdp->wb.upper.status.error" always
>>> returns 0 (where is this value set by the way?).
>>
>> I think it is set by the hardware.
>>
>>>
>>> I didn't see this problem for single-core case, in which it returned
>>> MAX_BURST packets at every rte_eth_rx_burst() call. Also, if I break out
>>> of while loop when I receive 0, I keep receiving packets in next <port,
>>> queue> pairs. Does anyone know why this block might happen? Or am I not
>>> allowed to call rte_eth_rx_burst() multiple times on same <port, queue>
>>> pair if I get 0? Any help will be great! Thank you!
>>
>> Although not mentioned in the documentation itself, as far as I know
>> rte_eth_rx_burst() is not thread-safe. If you look in to receive
>> functions, there are no locking anywhere. You should call it on
>> separate queues from different threads, and configure e.g RSS to
>> distribute the traffic by the hardware.
>
> I'm calling rte_eth_rx_burst() on separate queue ids for each thread.
> I'm actually using lcore_id (= 0, 1, 2, 3 for 4 threads pinned to each
> separate CPU core) as queue_id. I also made sure that this problem is
> not caused by threads conflicting by locking before calling
> rte_eth_rx_burst().
>
> For RSS, I configured with ETH_RSS_IP for load balancing traffic to each
> port and queue. But even if RSS wasn't set, shouldn't at least one core
> be receiving packets? What I'm seeing is all threads getting stuck at
> rte_eth_rx_burst() with return value of 0s indefinitely.
>
>>>
>>> ------------------------------------------------------------------------
>>> int cnt = MAX_BURST; // MAX_BURST = 32
>>> int off = 0;
>>> do {
>>>      ret = rte_eth_rx_burst(port_id, queue_id, &m_table[off], cnt);

Another thing which might cause your problem is that I don't see where 
do you release the buffers after received into m_table. You need to call 
rte_pktmbuf_free() on them at some point, otherwise your pool can get 
depleted, and the receive function can't refill the descriptor rings.


>>>      if (ret == 0) {
>>>          // don't break out but continue
>>>      } else if (ret > 0) {
>>>          off += ret;
>>>          cnt -= ret;
>>>      }
>>> } while (cnt > 0);
>>> ------------------------------------------------------------------------
>>>
>>> Best,
>>> Younghwan
>
> Thanks,
> Younghwan