[dpdk-users] Low Rx throughput when using Mellanox ConnectX-3 card with DPDK

[Wiles, Keith]

> On Apr 12, 2017, at 7:06 PM, Shihabur Rahman Chowdhury <shihab.buet at gmail.com> wrote:
> 
> We've disabled the pause frames. That also disables flow control I assume. Correct me if I am wrong.
> 
> On the pktgen side, the dropped field and overrun field for Rx keeps increasing for ifconfig. Btw, both overrun and dropped fields have the exact same value always.

Are you using the Linux kernel pktgen or the DPDK Pktgen?

http://dpdk.org/browse/apps/pktgen-dpdk/


> 
> Our ConnectX3 NIC has just a single 10G port. So there is no way to create such loopback connection.
> 
> Thanks
> 
> Shihabur Rahman Chowdhury
> David R. Cheriton School of Computer Science
> University of Waterloo
> 
> 
> 
> On Wed, Apr 12, 2017 at 6:41 PM, Wiles, Keith <keith.wiles at intel.com> wrote:
> 
> > On Apr 12, 2017, at 4:00 PM, Shihabur Rahman Chowdhury <shihab.buet at gmail.com> wrote:
> >
> > Hello,
> >
> > We are running a simple DPDK application and observing quite low
> > throughput. We are currently testing a DPDK application with the following
> > setup
> >
> > - 2 machines with 2xIntel Xeon E5-2620 CPUs
> > - Each machine with a Mellanox single port 10G ConnectX3 card
> > - Mellanox DPDK version 16.11
> > - Mellanox OFED 4.0-2.0.0.1 and latest firmware for ConnectX3
> >
> > The application is doing almost nothing. It is reading a batch of 64
> > packets from a single rxq, swapping the mac of each packet and writing it
> > back to a single txq. The rx and tx is being handled by separate lcores on
> > the same NUMA socket. We are running pktgen on another machine. With 64B
> > sized packets we are seeing ~14.8Mpps Tx rate and ~7.3Mpps Rx rate in
> > pktgen. We checked the NIC on the machine running the DPDK application
> > (with ifconfig) and it looks like there is a large number of packets being
> > dropped by the interface. Our connectx3 card should be theoretically be
> > able to handle 10Gbps Rx + 10Gbps Tx throughput (with channel width 4, the
> > theoretical max on PCIe 3.0 should be ~31.2Gbps). Interestingly, when Tx
> > rate is reduced in pktgent (to ~9Mpps), the Rx rate increases to ~9Mpps.
> 
> Not sure what is going on here, when you drop the rate to 9Mpps I assume you stop getting missed frames.
> Do you have flow control enabled?
> 
> On the pktgen side are you seeing missed RX packets?
> Did you loopback the cable from pktgen machine to the other port on the pktgen machine and did you get the same Rx/Tx performance in that configuration?
> 
> >
> > We would highly appriciate if we could get some pointers as to what can be
> > possibly causing this mismatch in Rx and Tx. Ideally, we should be able to
> > see ~14Mpps Rx well. Is it because we are using a single port? Or something
> > else?
> >
> > FYI, we also ran the sample l2fwd application and test-pmd and got
> > comparable results in the same setup.
> >
> > Thanks
> > Shihab
> 
> Regards,
> Keith
> 
> 

Regards,
Keith