[dpdk-dev] How to approach packet TX lockups

[Matt Laswell]

Yes, we're on 1.6r2.  That said, I've tried a number of different values
for the thresholds without a lot of luck.  Setting wthresh/hthresh/pthresh
to 0/0/32 or 0/0/0 doesn't appear to fix things.  And, as Matthew
suggested, I'm pretty sure using 0 for the thresholds leads to auto-config
by the driver.  I also tried 1/1/32, which required that I also change the
rs_thresh value from 0 to 1 to work around a panic in PMD initialization
("TX WTHRESH must be set to 0 if tx_rs_thresh is greater than 1").

Any other suggestions?

On Mon, Nov 16, 2015 at 7:31 PM, Stephen Hemminger <
stephen at networkplumber.org> wrote:

> On Mon, 16 Nov 2015 18:49:15 -0600
> Matt Laswell <laswell at infiniteio.com> wrote:
>
> > Hey Stephen,
> >
> > Thanks a lot; that's really useful information.  Unfortunately, I'm at a
> > stage in our release cycle where upgrading to a new version of DPDK isn't
> > feasible.  Any chance you (or others reading this) has a pointer to the
> > relevant changes?  While I can't afford to upgrade DPDK entirely,
> > backporting targeted fixes is more doable.
> >
> > Again, thanks.
> >
> > - Matt
> >
> >
> > On Mon, Nov 16, 2015 at 6:12 PM, Stephen Hemminger <
> > stephen at networkplumber.org> wrote:
> >
> > > On Mon, 16 Nov 2015 17:48:35 -0600
> > > Matt Laswell <laswell at infiniteio.com> wrote:
> > >
> > > > Hey Folks,
> > > >
> > > > I sent this to the users email list, but I'm not sure how many
> people are
> > > > actively reading that list at this point.  I'm dealing with a
> situation
> > > in
> > > > which my application loses the ability to transmit packets out of a
> port
> > > > during times of moderate stress.  I'd love to hear suggestions for
> how to
> > > > approach this problem, as I'm a bit at a loss at the moment.
> > > >
> > > > Specifically, I'm using DPDK 1.6r2 running on Ubuntu 14.04LTS on
> Haswell
> > > > processors.  I'm using the 82599 controller, configured to spread
> packets
> > > > across multiple queues.  Each queue is accessed by a different lcore
> in
> > > my
> > > > application; there is therefore concurrent access to the controller,
> but
> > > > not to any of the queues.  We're binding the ports to the igb_uio
> driver.
> > > > The symptoms I see are these:
> > > >
> > > >
> > > >    - All transmit out of a particular port stops
> > > >    - rte_eth_tx_burst() indicates that it is sending all of the
> packets
> > > >    that I give to it
> > > >    - rte_eth_stats_get() gives me stats indicating that no packets
> are
> > > >    being sent on the affected port.  Also, no tx errors, and no pause
> > > frames
> > > >    sent or received (opackets = 0, obytes = 0, oerrors = 0, etc.)
> > > >    - All other ports continue to work normally
> > > >    - The affected port continues to receive packets without problems;
> > > only
> > > >    TX is affected
> > > >    - Resetting the port via rte_eth_dev_stop() and
> rte_eth_dev_start()
> > > >    restores things and packets can flow again
> > > >    - The problem is replicable on multiple devices, and doesn't
> follow
> > > one
> > > >    particular port
> > > >
> > > > I've tried calling rte_mbuf_sanity_check() on all packets before
> sending
> > > > them.  I've also instrumented my code to look for packets that have
> > > already
> > > > been sent or freed, as well as cycles in chained packets being
> sent.  I
> > > > also put a lock around all accesses to rte_eth* calls to synchronize
> > > access
> > > > to the NIC.  Given some recent discussion here, I also tried
> changing the
> > > > TX RS threshold from 0 to 32, 16, and 1.  None of these strategies
> proved
> > > > effective.
> > > >
> > > > Like I said at the top, I'm a little at a loss at this point.  If you
> > > were
> > > > dealing with this set of symptoms, how would you proceed?
> > > >
> > >
> > > I remember some issues with old DPDK 1.6 with some of the prefetch
> > > thresholds on 82599. You would be better off going to a later DPDK
> > > version.
> > >
>
> I hope you are on 1.6.0r2 at least??
>
> With older DPDK there was no way to get driver to tell you what the
> preferred settings were for pthresh/hthresh/wthresh. And the values
> in Intel sample applications were broken on some hardware.
>
> I remember reverse engineering the safe values from reading the Linux
> driver.
>
> The Linux driver is much better tested than the DPDK one...
> In the Linux driver, the Transmit Descriptor Controller (txdctl)
> is fixed at (for transmit)
>    wthresh = 1
>    hthresh = 1
>    pthresh = 32
>
> The DPDK 2.2 driver uses:
>     wthresh = 0
>     hthresh = 0
>     pthresh = 32
>
>
>
>
>
>
>