[dpdk-dev] Could not achieve wire speed for 40GE with any DPDK version on XL710 NIC's

[Vladimir Medvedkin]

Hi Pavel,

Looks like you ran into pcie bottleneck. So let's calculate xl710 rx only
case.
Assume we have 32byte descriptors (if we want more offload).
DMA makes one pcie transaction with packet payload, one descriptor
writeback and one memory request for free descriptors for every 4 packets.
For Transaction Layer Packet (TLP) there is 30 bytes overhead (4 PHY + 6
DLL + 16 header + 4 ECRC). So for 1 rx packet dma sends 30 + 64(packet
itself) + 30 + 32 (writeback descriptor) + (16 / 4) (read request for new
descriptors). Note that we do not take into account PCIe ACK/NACK/FC Update
DLLP. So we have 160 bytes per packet. One lane PCIe 3.0 transmits 1 byte
in 1 ns, so x8 transmits 8 bytes  in 1 ns. 1 packet transmits in 20 ns.
Thus in theory pcie 3.0 x8 may transfer not more than 50mpps.
Correct me if I'm wrong.

Regards,
Vladimir


2015-06-29 18:41 GMT+03:00 Pavel Odintsov <pavel.odintsov at gmail.com>:

> Hello, Andrew!
>
> What NIC have you used? Is it XL710?
>
> On Mon, Jun 29, 2015 at 6:38 PM, Andrew Theurer <atheurer at redhat.com>
> wrote:
> >
> >
> > On Mon, Jun 29, 2015 at 10:06 AM, Keunhong Lee <dlrmsghd at gmail.com>
> wrote:
> >>
> >> I have not used XL710 or i40e.
> >> I have no opinion for those NICs.
> >>
> >> Keunhong.
> >>
> >> 2015-06-29 15:59 GMT+09:00 Pavel Odintsov <pavel.odintsov at gmail.com>:
> >>
> >> > Hello!
> >> >
> >> > Lee, thank you so much for sharing your experience! What do you think
> >> > about 40GE version of 82599?
> >> >
> >> > On Mon, Jun 29, 2015 at 2:35 AM, Keunhong Lee <dlrmsghd at gmail.com>
> >> > wrote:
> >> > > DISCLAIMER: This information is not verified. This is truly my
> >> > > personal
> >> > > opinion.
> >> > >
> >> > > As I know, intel 82599 is the only 10G NIC which supports line rate
> >> > > with
> >> > > minimum sized packets (64 byte).
> >> > > According to our internal tests, Mellanox's 40G NICs even support
> less
> >> > than
> >> > > 30Mpps.
> >> > > I think 40 Mpps is the hardware capacity.
> >
> >
> > This is approximately what I see as well.
> >
> >>
> >> > >
> >> > > Keunhong.
> >> > >
> >> > >
> >> > >
> >> > > 2015-06-28 19:34 GMT+09:00 Pavel Odintsov <pavel.odintsov at gmail.com
> >:
> >> > >>
> >> > >> Hello, folks!
> >> > >>
> >> > >> We have execute bunch of tests for receive data with Intel XL710
> 40GE
> >> > >> NIC. We want to achieve wire speed on this platform for traffic
> >> > >> capture.
> >> > >>
> >> > >> But we definitely can't do it. We tried with different versions of
> >> > >> DPDK: 1.4, 1.6, 1.8, 2.0. And have not success.
> >> > >>
> >> > >> We achieved only 40Mpps and could do more.
> >> > >>
> >> > >> Could anybody help us with this issue? Looks like this NIC's could
> >> > >> not
> >> > >> work on wire speed :(
> >> > >>
> >> > >> Platform: Intel Xeon E5 e5 2670 + XL 710.
> >> > >>
> >> > >> --
> >> > >> Sincerely yours, Pavel Odintsov
> >> > >
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Sincerely yours, Pavel Odintsov
> >> >
> >
> > -Andrew
> >
> >
>
>
>
> --
> Sincerely yours, Pavel Odintsov
>