Testpmd/l3fwd port shutdown failure on Arm Altra systems

[Juraj Linkeš]

Hello Qiming, Beilei,

Could you please help us debug this issue? Anything that would help with
getting to the bottom of anything that could go wrong during port
init/cleanup would be appreciated - extra eal/testpmd options or even code
changes (such as where could add extra debug messages).

Thanks,
Juraj

On Wed, Mar 8, 2023 at 7:25 AM Juraj Linkeš <juraj.linkes at pantheon.tech>
wrote:

> Hello Qiming, Beilei,
>
> Another reminder - are you looking at this by any chance?
>
> The high level short description is that testpmd/l3fwd breaks a link
> between two servers while VPP (using DPDK) doesn't. This leads us to
> believe there's a problem with testpmd/l3fwd/i40e driver in DPDK.
>
> Thanks,
> Juraj
>
> On Tue, Feb 21, 2023 at 12:18 PM Juraj Linkeš
> <juraj.linkes at pantheon.tech> wrote:
> >
> > Hi Qiming,
> >
> > Just a friendly reminder, would you please take a look?
> >
> > Thanks,
> > Juraj
> >
> >
> > On Tue, Feb 7, 2023 at 3:10 AM Xing, Beilei <beilei.xing at intel.com>
> wrote:
> > >
> > > Hi Qiming,
> > >
> > > Could you please help on this? Thanks.
> > >
> > > BR,
> > > Beilei
> > >
> > > > -----Original Message-----
> > > > From: Juraj Linkeš <juraj.linkes at pantheon.tech>
> > > > Sent: Monday, February 6, 2023 4:53 PM
> > > > To: Singh, Aman Deep <aman.deep.singh at intel.com>; Zhang, Yuying
> > > > <yuying.zhang at intel.com>; Xing, Beilei <beilei.xing at intel.com>
> > > > Cc: dev at dpdk.org; Ruifeng Wang <Ruifeng.Wang at arm.com>; Zhang, Lijian
> > > > <Lijian.Zhang at arm.com>; Honnappa Nagarahalli
> > > > <Honnappa.Nagarahalli at arm.com>
> > > > Subject: Re: Testpmd/l3fwd port shutdown failure on Arm Altra systems
> > > >
> > > > Hello i40e and testpmd maintainers,
> > > >
> > > > A gentle reminder - would you please advise how to debug the issue
> described
> > > > below?
> > > >
> > > > Thanks,
> > > > Juraj
> > > >
> > > > On Fri, Jan 20, 2023 at 1:07 PM Juraj Linkeš
> <juraj.linkes at pantheon.tech>
> > > > wrote:
> > > > >
> > > > > Adding the logfile.
> > > > >
> > > > >
> > > > >
> > > > > One thing that's in the logs but didn't explicitly mention is the
> DPDK version
> > > > we've tried this with:
> > > > >
> > > > > EAL: RTE Version: 'DPDK 22.07.0'
> > > > >
> > > > >
> > > > >
> > > > > We also tried earlier versions going back to 21.08, with no luck.
> I also did a
> > > > quick check on 22.11, also with no luck.
> > > > >
> > > > >
> > > > >
> > > > > Juraj
> > > > >
> > > > >
> > > > >
> > > > > From: Juraj Linkeš
> > > > > Sent: Friday, January 20, 2023 12:56 PM
> > > > > To: 'aman.deep.singh at intel.com' <aman.deep.singh at intel.com>;
> > > > > 'yuying.zhang at intel.com' <yuying.zhang at intel.com>; Xing, Beilei
> > > > > <beilei.xing at intel.com>
> > > > > Cc: dev at dpdk.org; Ruifeng Wang <Ruifeng.Wang at arm.com>; 'Lijian
> Zhang'
> > > > > <Lijian.Zhang at arm.com>; 'Honnappa Nagarahalli'
> > > > > <Honnappa.Nagarahalli at arm.com>
> > > > > Subject: Testpmd/l3fwd port shutdown failure on Arm Altra systems
> > > > >
> > > > >
> > > > >
> > > > > Hello i40e and testpmd maintainers,
> > > > >
> > > > >
> > > > >
> > > > > We're hitting an issue with DPDK testpmd on Ampere Altra servers
> in FD.io
> > > > lab.
> > > > >
> > > > >
> > > > >
> > > > > A bit of background: along with VPP performance tests (which uses
> DPDK),
> > > > we're running a small number of basic DPDK testpmd and l3fwd tests
> in FD.io
> > > > as well. This is to catch any performance differences due to VPP
> updating its
> > > > DPDK version.
> > > > >
> > > > >
> > > > >
> > > > > We're running both l3fwd tests and testpmd tests. The Altra
> servers are two
> > > > socket and the topology is TG -> DUT1 -> DUT2 -> TG, traffic flows
> in both
> > > > directions, but nothing gets forwarded (with a slight caveat - put a
> pin in this).
> > > > There's nothing special in the tests, just forwarding traffic. The
> NIC we're
> > > > testing is xl710-QDA2.
> > > > >
> > > > >
> > > > >
> > > > > The same tests are passing on all other testbeds - we have various
> two node
> > > > (1 DUT, 1 TG) and three node (2 DUT, 1 TG) Intel and Arm testbeds
> and with
> > > > various NICs (Intel 700 and 800 series and the Intel testbeds use
> some
> > > > Mellanox NICs as well). We don't have quite the same combination of
> another
> > > > three node topology with the same NIC though, so it looks like
> something with
> > > > testpmd/l3fwd and xl710-QDA2 on Altra servers.
> > > > >
> > > > >
> > > > >
> > > > > VPP performance tests are passing, but l3fwd and testpmd fail.
> This leads us
> > > > to believe to it's a software issue, but there could something wrong
> with the
> > > > hardware. I'll talk about testpmd from now on, but as far we can
> tell, the
> > > > behavior is the same for testpmd and l3fwd.
> > > > >
> > > > >
> > > > >
> > > > > Getting back to the caveat mentioned earlier, there seems to be
> something
> > > > wrong with port shutdown. When running testpmd on a testbed that
> hasn't
> > > > been used for a while it seems that all ports are up right away (we
> don't see
> > > > any "Port 0|1: link state change event") and the setup works fine
> (forwarding
> > > > works). After restarting testpmd (restarting on one server is
> sufficient), the
> > > > ports between DUT1 and DUT2 (but not between DUTs and TG) go down and
> > > > are not usable in DPDK, VPP or in Linux (with i40e kernel driver)
> for a while
> > > > (measured in minutes, sometimes dozens of minutes; the duration is
> seemingly
> > > > random). The ports eventually recover and can be used again, but
> there's
> > > > nothing in syslog suggesting what happened.
> > > > >
> > > > >
> > > > >
> > > > > What seems to be happening is testpmd put the ports into some
> faulty state.
> > > > This only happens on the DUT1 -> DUT2 link though (the ports between
> the
> > > > two testpmds), not on TG -> DUT1 link (the TG port is left alone).
> > > > >
> > > > >
> > > > >
> > > > > Some more info:
> > > > >
> > > > > We've come across the issue with this configuration:
> > > > >
> > > > > OS: Ubuntu20.04 with kernel 5.4.0-65-generic.
> > > > >
> > > > > Old NIC firmware, never upgraded: 6.01 0x800035da 1.1747.0.
> > > > >
> > > > > Drivers versions: i40e 2.17.15 and iavf 4.3.19.
> > > > >
> > > > >
> > > > >
> > > > > As well as with this configuration:
> > > > >
> > > > > OS: Ubuntu22.04 with kernel 5.15.0-46-generic.
> > > > >
> > > > > Updated firmware: 8.30 0x8000a4ae 1.2926.0.
> > > > >
> > > > > Drivers: i40e 2.19.3 and iavf 4.5.3.
> > > > >
> > > > >
> > > > >
> > > > > Unsafe noiommu mode is disabled:
> > > > >
> > > > > cat /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
> > > > >
> > > > > N
> > > > >
> > > > >
> > > > >
> > > > > We used DPDK 22.07 in manual testing and built it on DUTs, using
> generic
> > > > build:
> > > > >
> > > > > meson -Dexamples=l3fwd -Dc_args=-DRTE_LIBRTE_I40E_16BYTE_RX_DESC=y
> > > > > -Dplatform=generic build
> > > > >
> > > > >
> > > > >
> > > > > We're running testpmd with this command:
> > > > >
> > > > > sudo build/app/dpdk-testpmd -v -l 1,2 -a 0004:04:00.1 -a
> 0004:04:00.0
> > > > > --in-memory -- -i --forward-mode=io --burst=64 --txq=1 --rxq=1
> > > > > --tx-offloads=0x0 --numa --auto-start --total-num-mbufs=32768
> > > > > --nb-ports=2 --portmask=0x3 --max-pkt-len=1518 --mbuf-size=16384
> > > > > --nb-cores=1
> > > > >
> > > > >
> > > > >
> > > > > And l3fwd (with different macs on the other server):
> > > > >
> > > > > sudo /tmp/openvpp-testing/dpdk/build/examples/dpdk-l3fwd -v -l 1,2
> -a
> > > > > 0004:04:00.0 -a 0004:04:00.1 --in-memory -- --parse-ptype
> > > > > --eth-dest="0,40:a6:b7:85:e7:79" --eth-dest="1,3c:fd:fe:c3:e7:a1"
> > > > > --config="(0, 0, 2),(1, 0, 2)" -P -L -p 0x3
> > > > >
> > > > >
> > > > >
> > > > > We tried adding logs with  --log-level=pmd,debug and
> --no-lsc-interrupt, but
> > > > that didn't reveal anything helpful, as far as we can tell - please
> have a look at
> > > > the attached log. The faulty port is port0 (starts out as down, then
> we waited
> > > > for around 25 minutes for it to go up and then we shut down testpmd).
> > > > >
> > > > >
> > > > >
> > > > > We'd like to ask for pointers on what could be the cause or how to
> debug
> > > > this issue further.
> > > > >
> > > > >
> > > > >
> > > > > Thanks,
> > > > > Juraj
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mails.dpdk.org/archives/dev/attachments/20230403/7e0a3c6c/attachment-0001.htm>