[dpdk-dev] packet loss: multi-queue (RSS enabled)

[Daniel Kaminsky]

Hi Hamid,

I didn't see any attachment but I think there is a solution. My first
question is, when you created the mempool, how much did you defined the
memory cache per CPU? In the example program this number is fairly small
(32) and I think that increasing it to something much bigger (e.g. 512)
will significantly improve the CPU scalability.

Regards,
Daniel Kaminsky


On Wed, Apr 30, 2014 at 7:56 AM, Jayakumar, Muthurajan <
muthurajan.jayakumar at intel.com> wrote:

> Hi,
> Please find the attached paper http://kfall.net/ucbpage/papers/snc.pdf
> Figures 4 and 5 refers about the degradation when the # of queues are
> increased.
> It refers sweet spot as 2 to 4 queues.
>
> Have you please verified with smaller # of queues please?
>
> Thanks,
>
>
> -----Original Message-----
> From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of Hamid Ramazani
> Sent: Tuesday, April 29, 2014 9:42 PM
> To: Thomas Monjalon; dev at dpdk.org
> Subject: [dpdk-dev] packet loss: multi-queue (RSS enabled)
>
> Hi,
> I tried a lot (more than a week) to solve the problem myself and not to
> bother the list, but I didn't succeed. Maybe other people have the same
> problem.
>
> I have a simple program attached. It is intended for simple packet
> capturing; captures from interface and writes to memory, and frees the
> memory in the next loop.
>
> I have a 10G 82599EB Intel SFI/SFP+ network interface for capturing
> packets.
> As you may know, this network card supports up to 128 RSS queues.
>
> This is just a test, so the packets being sent at 820Kpps (kilo packet per
> second). Each packet is 1500B (fixed size); it is 9.16 Gbit per second.
> Of course when the packet per seconds goes up and packet size goes down
> (e.g. 400B per packet), it gets much worse.
>
> When using one queue to receive, I receive all the packets, with no loss.
> When I use more than one queue (e.g. 8 queues), with each thread running
> on a dedicated core, I have a considerable amount of loss.
>
> Please note that:
> 1. The computer has 12 * 2.67GHz cores, and it does nothing else but
> capturing packets. The CPU is Intel Xeon X5650.
> 2. The operating system is Ubuntu 12.04.3 LTS
>
> Attached file includes:
> main.h
> main.c
> Makefile
> ./run.sh
>
> It is configured to be run with 8 queues.
>
> If you want to change the number of receive queues, please:
> 1. in main.c, change the value assigned to nb_rx_q_of_dev to the desired
> value.
> 2. change core mask in run.sh file (since there is SKIP_MASTER, you should
> give a core containing one more CPU than given number of queues).
>
> I think there might be following problems:
> 1. the port configuration is not fine.
> 2. freeing memory has a considerable amount of overhead, and may be I
> shouldn't do that. But If I don't the pool will be full, won't be? Is there
> any other way?
>
> Please help.
> Thanks a lot in advance for your help and comments.
>
> All the Best,
> Hamid
>