[dpdk-dev] How to approach packet TX lockups

[Ananyev, Konstantin]

Hi Matt,

As I said, at least  try to upgrade contents of shared code to the latest one.
In previous releases: lib/librte_pmd_ixgbe/ixgbe, now located at: drivers/net/ixgbe/.

> For reference, my transmit function is  rte_eth_tx_burst().
I meant what ixgbe TX function it points to: ixgbe_xmit_pkts or ixgbe_xmit_pkts_simple()?
For ixgbe_xmit_pkts_simple() don’t set tx_rs_thresh > 32,
for ixgbe_xmit_pkts() the safest way is to set  tx_rs_thresh=1.
Though as I understand from your previous mails, you already did that, and it didn’t help.
Konstantin


From: Matt Laswell [mailto:laswell at infiniteio.com]
Sent: Tuesday, November 17, 2015 3:05 PM
To: Ananyev, Konstantin
Cc: Stephen Hemminger; dev at dpdk.org
Subject: Re: [dpdk-dev] How to approach packet TX lockups

Hey Konstantin,

Moving from 1.6r2 to 2.2 is going to be a pretty significant change due to things like changes in the MBuf format, API differences, etc.  Even as an experiment, that's an awfully large change to absorb.  Is there a subset that you're referring to that could be more readily included without modifying so many touch points into DPDK?

For reference, my transmit function is  rte_eth_tx_burst().  It seems to reliably tell me that it has enqueued all of the packets that I gave it, however the stats from rte_eth_stats_get() indicate that no packets are actually being sent.

Thanks,

- Matt

On Tue, Nov 17, 2015 at 8:44 AM, Ananyev, Konstantin <konstantin.ananyev at intel.com<mailto:konstantin.ananyev at intel.com>> wrote:


> -----Original Message-----
> From: dev [mailto:dev-bounces at dpdk.org<mailto:dev-bounces at dpdk.org>] On Behalf Of Matt Laswell
> Sent: Tuesday, November 17, 2015 2:24 PM
> To: Stephen Hemminger
> Cc: dev at dpdk.org<mailto:dev at dpdk.org>
> Subject: Re: [dpdk-dev] How to approach packet TX lockups
>
> Yes, we're on 1.6r2.  That said, I've tried a number of different values
> for the thresholds without a lot of luck.  Setting wthresh/hthresh/pthresh
> to 0/0/32 or 0/0/0 doesn't appear to fix things.  And, as Matthew
> suggested, I'm pretty sure using 0 for the thresholds leads to auto-config
> by the driver.  I also tried 1/1/32, which required that I also change the
> rs_thresh value from 0 to 1 to work around a panic in PMD initialization
> ("TX WTHRESH must be set to 0 if tx_rs_thresh is greater than 1").
>
> Any other suggestions?

That's not only DPDK code changed since 1.6.
I am pretty sure that we also have a new update of shared code since then
(and as I remember probably more than one).
One suggestion would be at least try to upgrade the shared code up to the latest.
Another one - even if you can't upgrade to 2.2 in you production environment,
it probably worth to do that in some test environment and then check does the problem persist.
If yes,  then we'll need some guidance how to reproduce it.

Another question it is not clear what TX function do you use?
Konstantin

>
> On Mon, Nov 16, 2015 at 7:31 PM, Stephen Hemminger <
> stephen at networkplumber.org<mailto:stephen at networkplumber.org>> wrote:
>
> > On Mon, 16 Nov 2015 18:49:15 -0600
> > Matt Laswell <laswell at infiniteio.com<mailto:laswell at infiniteio.com>> wrote:
> >
> > > Hey Stephen,
> > >
> > > Thanks a lot; that's really useful information.  Unfortunately, I'm at a
> > > stage in our release cycle where upgrading to a new version of DPDK isn't
> > > feasible.  Any chance you (or others reading this) has a pointer to the
> > > relevant changes?  While I can't afford to upgrade DPDK entirely,
> > > backporting targeted fixes is more doable.
> > >
> > > Again, thanks.
> > >
> > > - Matt
> > >
> > >
> > > On Mon, Nov 16, 2015 at 6:12 PM, Stephen Hemminger <
> > > stephen at networkplumber.org<mailto:stephen at networkplumber.org>> wrote:
> > >
> > > > On Mon, 16 Nov 2015 17:48:35 -0600
> > > > Matt Laswell <laswell at infiniteio.com<mailto:laswell at infiniteio.com>> wrote:
> > > >
> > > > > Hey Folks,
> > > > >
> > > > > I sent this to the users email list, but I'm not sure how many
> > people are
> > > > > actively reading that list at this point.  I'm dealing with a
> > situation
> > > > in
> > > > > which my application loses the ability to transmit packets out of a
> > port
> > > > > during times of moderate stress.  I'd love to hear suggestions for
> > how to
> > > > > approach this problem, as I'm a bit at a loss at the moment.
> > > > >
> > > > > Specifically, I'm using DPDK 1.6r2 running on Ubuntu 14.04LTS on
> > Haswell
> > > > > processors.  I'm using the 82599 controller, configured to spread
> > packets
> > > > > across multiple queues.  Each queue is accessed by a different lcore
> > in
> > > > my
> > > > > application; there is therefore concurrent access to the controller,
> > but
> > > > > not to any of the queues.  We're binding the ports to the igb_uio
> > driver.
> > > > > The symptoms I see are these:
> > > > >
> > > > >
> > > > >    - All transmit out of a particular port stops
> > > > >    - rte_eth_tx_burst() indicates that it is sending all of the
> > packets
> > > > >    that I give to it
> > > > >    - rte_eth_stats_get() gives me stats indicating that no packets
> > are
> > > > >    being sent on the affected port.  Also, no tx errors, and no pause
> > > > frames
> > > > >    sent or received (opackets = 0, obytes = 0, oerrors = 0, etc.)
> > > > >    - All other ports continue to work normally
> > > > >    - The affected port continues to receive packets without problems;
> > > > only
> > > > >    TX is affected
> > > > >    - Resetting the port via rte_eth_dev_stop() and
> > rte_eth_dev_start()
> > > > >    restores things and packets can flow again
> > > > >    - The problem is replicable on multiple devices, and doesn't
> > follow
> > > > one
> > > > >    particular port
> > > > >
> > > > > I've tried calling rte_mbuf_sanity_check() on all packets before
> > sending
> > > > > them.  I've also instrumented my code to look for packets that have
> > > > already
> > > > > been sent or freed, as well as cycles in chained packets being
> > sent.  I
> > > > > also put a lock around all accesses to rte_eth* calls to synchronize
> > > > access
> > > > > to the NIC.  Given some recent discussion here, I also tried
> > changing the
> > > > > TX RS threshold from 0 to 32, 16, and 1.  None of these strategies
> > proved
> > > > > effective.
> > > > >
> > > > > Like I said at the top, I'm a little at a loss at this point.  If you
> > > > were
> > > > > dealing with this set of symptoms, how would you proceed?
> > > > >
> > > >
> > > > I remember some issues with old DPDK 1.6 with some of the prefetch
> > > > thresholds on 82599. You would be better off going to a later DPDK
> > > > version.
> > > >
> >
> > I hope you are on 1.6.0r2 at least??
> >
> > With older DPDK there was no way to get driver to tell you what the
> > preferred settings were for pthresh/hthresh/wthresh. And the values
> > in Intel sample applications were broken on some hardware.
> >
> > I remember reverse engineering the safe values from reading the Linux
> > driver.
> >
> > The Linux driver is much better tested than the DPDK one...
> > In the Linux driver, the Transmit Descriptor Controller (txdctl)
> > is fixed at (for transmit)
> >    wthresh = 1
> >    hthresh = 1
> >    pthresh = 32
> >
> > The DPDK 2.2 driver uses:
> >     wthresh = 0
> >     hthresh = 0
> >     pthresh = 32
> >
> >
> >
> >
> >
> >
> >