[dpdk-users] Low Rx throughput when using Mellanox ConnectX-3 card with DPDK

[Dave Wallace]

I have encountered a similar issue in the past on a system configuration 
where the PCI interface to the NIC was on the other NUMA node.

Something else to check...
-daw-


On 04/12/2017 08:06 PM, Shihabur Rahman Chowdhury wrote:
> We've disabled the pause frames. That also disables flow control I assume.
> Correct me if I am wrong.
>
> On the pktgen side, the dropped field and overrun field for Rx keeps
> increasing for ifconfig. Btw, both overrun and dropped fields have the
> exact same value always.
>
> Our ConnectX3 NIC has just a single 10G port. So there is no way to create
> such loopback connection.
>
> Thanks
>
> Shihabur Rahman Chowdhury
> David R. Cheriton School of Computer Science
> University of Waterloo
>
>
>
> On Wed, Apr 12, 2017 at 6:41 PM, Wiles, Keith <keith.wiles at intel.com> wrote:
>
>>> On Apr 12, 2017, at 4:00 PM, Shihabur Rahman Chowdhury <
>> shihab.buet at gmail.com> wrote:
>>> Hello,
>>>
>>> We are running a simple DPDK application and observing quite low
>>> throughput. We are currently testing a DPDK application with the
>> following
>>> setup
>>>
>>> - 2 machines with 2xIntel Xeon E5-2620 CPUs
>>> - Each machine with a Mellanox single port 10G ConnectX3 card
>>> - Mellanox DPDK version 16.11
>>> - Mellanox OFED 4.0-2.0.0.1 and latest firmware for ConnectX3
>>>
>>> The application is doing almost nothing. It is reading a batch of 64
>>> packets from a single rxq, swapping the mac of each packet and writing it
>>> back to a single txq. The rx and tx is being handled by separate lcores
>> on
>>> the same NUMA socket. We are running pktgen on another machine. With 64B
>>> sized packets we are seeing ~14.8Mpps Tx rate and ~7.3Mpps Rx rate in
>>> pktgen. We checked the NIC on the machine running the DPDK application
>>> (with ifconfig) and it looks like there is a large number of packets
>> being
>>> dropped by the interface. Our connectx3 card should be theoretically be
>>> able to handle 10Gbps Rx + 10Gbps Tx throughput (with channel width 4,
>> the
>>> theoretical max on PCIe 3.0 should be ~31.2Gbps). Interestingly, when Tx
>>> rate is reduced in pktgent (to ~9Mpps), the Rx rate increases to ~9Mpps.
>> Not sure what is going on here, when you drop the rate to 9Mpps I assume
>> you stop getting missed frames.
>> Do you have flow control enabled?
>>
>> On the pktgen side are you seeing missed RX packets?
>> Did you loopback the cable from pktgen machine to the other port on the
>> pktgen machine and did you get the same Rx/Tx performance in that
>> configuration?
>>
>>> We would highly appriciate if we could get some pointers as to what can
>> be
>>> possibly causing this mismatch in Rx and Tx. Ideally, we should be able
>> to
>>> see ~14Mpps Rx well. Is it because we are using a single port? Or
>> something
>>> else?
>>>
>>> FYI, we also ran the sample l2fwd application and test-pmd and got
>>> comparable results in the same setup.
>>>
>>> Thanks
>>> Shihab
>> Regards,
>> Keith
>>
>>