[dpdk-dev] Occasional instability in RSS Hashes/Queues from X540 NIC

Yang, Qiming qiming.yang at intel.com
Tue Jul 18 10:58:37 CEST 2017


Hi Matt,

We can reproduce this RSS issue on 16.04 but can't on 17.02, so this issue was fixed on 17.02.
We suggest using the new version.

Qiming
> -----Original Message-----
> From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of Matt Laswell
> Sent: Friday, May 5, 2017 9:05 PM
> To: dev at dpdk.org
> Subject: Re: [dpdk-dev] Occasional instability in RSS Hashes/Queues from X540
> NIC
> 
> On Thu, May 4, 2017 at 1:15 PM, Matt Laswell <laswell at infinite.io> wrote:
> 
> > Hey Keith,
> >
> > Here is a hexdump of a subset of one of my packet captures.  In this
> > capture, all of the packets are part of the same TCP connection, which
> > happens to be NFSv3 traffic. All of them except packet number 6 get
> > the correct RSS hash and go to the right queue.  Packet number 6 (an NFS
> rename
> > reply with an NFS error) gets RSS hash 0 and goes to queue 0.   Whenever I
> > repeat this test, the reply to this particular rename attempt always
> > goes to the wrong core, though it seemingly differs from the rest of
> > the flow only in layers 4-7.
> >
> >  I'll also attach a pcap to this email, in case that's a more
> > convenient way to interact with the packets.
> >
> > --
> > Matt Laswell
> > laswell at infinite.io
> >
> >
> > 16:08:37.093306 IP 10.151.3.81.disclose > 10.151.3.161.nfsd: Flags
> > [P.], seq 3173509264:3173509380, ack 3244259549, win 580, options
> > [nop,nop,TS val
> > 23060466 ecr 490971270], length 116: NFS request xid 2690728524 112
> > access fh
> >
> Unknown/8B6BFEBB04000000CFABD10301000000FFFFFFFF00000000DABC0502
> 010000
> > 00 NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_
> > ACCESS_EXTEND|NFS_ACCESS_DELETE
> > 0x0000:  4500 00a8 6d0f 4000 4006 b121 0a97 0351  E...m. at .@..!...Q
> > 0x0010:  0a97 03a1 029b 0801 bd27 e890 c15f 78dd  .........'..._x.
> > 0x0020:  8018 0244 1cba 0000 0101 080a 015f dff2  ...D........._..
> > 0x0030:  1d43 a086 8000 0070 a061 424c 0000 0000  .C.....p.aBL....
> > 0x0040:  0000 0002 0001 86a3 0000 0003 0000 0004  ................
> > 0x0050:  0000 0001 0000 0020 0107 8d2f 0000 0007  .........../....
> > 0x0060:  6573 7869 3275 3100 0000 0000 0000 0000  esxi2u1.........
> > 0x0070:  0000 0001 0000 0000 0000 0000 0000 0000  ................
> > 0x0080:  0000 0020 8b6b febb 0400 0000 cfab d103  .....k..........
> > 0x0090:  0100 0000 ffff ffff 0000 0000 dabc 0502  ................
> > 0x00a0:  0100 0000 0000 001f                      ........
> > 16:08:37.095837 IP 10.151.3.161.nfsd > 10.151.3.81.disclose: Flags
> > [P.], seq 1:125, ack 116, win 28688, options [nop,nop,TS val 490971270
> > ecr 23060466], length 124: NFS reply xid 2690728524 reply ok 120
> > access c 001f
> > 0x0000:  4500 00b0 1b80 4000 4006 02a9 0a97 03a1  E..... at .@.......
> > 0x0010:  0a97 0351 0801 029b c15f 78dd bd27 e904  ...Q....._x..'..
> > 0x0020:  8018 7010 a61a 0000 0101 080a 1d43 a086  ..p..........C..
> > 0x0030:  015f dff2 8000 0078 a061 424c 0000 0001  ._.....x.aBL....
> > 0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
> > 0x0050:  0000 0000 0000 0001 0000 0002 0000 01ed  ................
> > 0x0060:  0000 0003 0000 0000 0000 0000 0000 0000  ................
> > 0x0070:  0000 0029 0000 0000 0000 0800 0000 00ff  ...)............
> > 0x0080:  ffff 00ff 0000 0000 bbfe 6b8b 0000 0001  ..........k.....
> > 0x0090:  03d1 abcf 5908 f554 3272 e4e6 5908 f554  ....Y..T2r..Y..T
> > 0x00a0:  3272 e4e6 5908 f554 3365 2612 0000 001f  2r..Y..T3e&.....
> > 16:08:37.096235 IP 10.151.3.81.disclose > 10.151.3.161.nfsd: Flags
> > [P.], seq 256:372, ack 285, win 589, options [nop,nop,TS val 23060467
> > ecr 490971270], length 116: NFS request xid 2724282956 112 access fh
> > Unknown/
> >
> 8B6BFEBB04000000D0ABD10301000000FFFFFFFF00000000DABC050201000000
> > NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_
> > ACCESS_EXTEND|NFS_ACCESS_DELETE
> > 0x0000:  4500 00a8 6d11 4000 4006 b11f 0a97 0351  E...m. at .@......Q
> > 0x0010:  0a97 03a1 029b 0801 bd27 e990 c15f 79f9  .........'..._y.
> > 0x0020:  8018 024d 1cba 0000 0101 080a 015f dff3  ...M........._..
> > 0x0030:  1d43 a086 8000 0070 a261 424c 0000 0000  .C.....p.aBL....
> > 0x0040:  0000 0002 0001 86a3 0000 0003 0000 0004  ................
> > 0x0050:  0000 0001 0000 0020 0107 8d2f 0000 0007  .........../....
> > 0x0060:  6573 7869 3275 3100 0000 0000 0000 0000  esxi2u1.........
> > 0x0070:  0000 0001 0000 0000 0000 0000 0000 0000  ................
> > 0x0080:  0000 0020 8b6b febb 0400 0000 d0ab d103  .....k..........
> > 0x0090:  0100 0000 ffff ffff 0000 0000 dabc 0502  ................
> > 0x00a0:  0100 0000 0000 001f                      ........
> > 16:08:37.098361 IP 10.151.3.161.nfsd > 10.151.3.81.disclose: Flags
> > [P.], seq 285:409, ack 372, win 28688, options [nop,nop,TS val
> > 490971270 ecr 23060467], length 124: NFS reply xid 2724282956 reply ok
> > 120 access c 001f
> > 0x0000:  4500 00b0 1b81 4000 4006 02a8 0a97 03a1  E..... at .@.......
> > 0x0010:  0a97 0351 0801 029b c15f 79f9 bd27 ea04  ...Q....._y..'..
> > 0x0020:  8018 7010 ec45 0000 0101 080a 1d43 a086  ..p..E.......C..
> > 0x0030:  015f dff3 8000 0078 a261 424c 0000 0001  ._.....x.aBL....
> > 0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
> > 0x0050:  0000 0000 0000 0001 0000 0002 0000 01ed  ................
> > 0x0060:  0000 0004 0000 0000 0000 0000 0000 0000  ................
> > 0x0070:  0000 0050 0000 0000 0000 0800 0000 00ff  ...P............
> > 0x0080:  ffff 00ff 0000 0000 bbfe 6b8b 0000 0001  ..........k.....
> > 0x0090:  03d1 abd0 5908 f554 3536 88ea 5908 f554  ....Y..T56..Y..T
> > 0x00a0:  3536 88ea 5908 f555 01ff bf76 0000 001f  56..Y..U...v....
> > 16:08:37.099013 IP 10.151.3.81.disclose > 10.151.3.161.nfsd: Flags
> > [P.], seq 652:856, ack 813, win 605, options [nop,nop,TS val 23060467
> > ecr 490971270], length 204: NFS request xid 2774614604 200 rename fh
> > Unknown/
> >
> 8B6BFEBB04000000D0ABD10301000000FFFFFFFF00000000DABC050201000000
> > "DirReplaceNotEmpty_ovr" -> fh
> Unknown/8B6BFEBB04000000D0ABD103010000
> > 00FFFFFFFF00000000DABC050201000000 "DirReplaceNotEmpty_src"
> > 0x0000:  4500 0100 6d14 4000 4006 b0c4 0a97 0351  E...m. at .@......Q
> > 0x0010:  0a97 03a1 029b 0801 bd27 eb1c c15f 7c09  .........'..._|.
> > 0x0020:  8018 025d 1d12 0000 0101 080a 015f dff3  ...]........._..
> > 0x0030:  1d43 a086 8000 00c8 a561 424c 0000 0000  .C.......aBL....
> > 0x0040:  0000 0002 0001 86a3 0000 0003 0000 000e  ................
> > 0x0050:  0000 0001 0000 0020 0107 8d2f 0000 0007  .........../....
> > 0x0060:  6573 7869 3275 3100 0000 0000 0000 0000  esxi2u1.........
> > 0x0070:  0000 0001 0000 0000 0000 0000 0000 0000  ................
> > 0x0080:  0000 0020 8b6b febb 0400 0000 d0ab d103  .....k..........
> > 0x0090:  0100 0000 ffff ffff 0000 0000 dabc 0502  ................
> > 0x00a0:  0100 0000 0000 0016 4469 7252 6570 6c61  ........DirRepla
> > 0x00b0:  6365 4e6f 7445 6d70 7479 5f6f 7672 0000  ceNotEmpty_ovr..
> > 0x00c0:  0000 0020 8b6b febb 0400 0000 d0ab d103  .....k..........
> > 0x00d0:  0100 0000 ffff ffff 0000 0000 dabc 0502  ................
> > 0x00e0:  0100 0000 0000 0016 4469 7252 6570 6c61  ........DirRepla
> > 0x00f0:  6365 4e6f 7445 6d70 7479 5f73 7263 0000  ceNotEmpty_src..
> > 16:08:37.101770 IP 10.151.3.161.nfsd > 10.151.3.81.disclose: Flags
> > [P.], seq 4294966865:4294966961, ack 4294967244, win 28688, options
> > [nop,nop,TS val 490971270 ecr 23060467], length 96: NFS reply xid
> > 2774614604 reply ok
> > 92 rename ERROR: File exists
> > 0x0000:  4500 0094 1b82 4000 4006 02c3 0a97 03a1  E..... at .@.......
> > 0x0010:  0a97 0351 0801 029b c15f 772d bd27 e85c  ...Q....._w-.'.\
> > 0x0020:  8018 7010 c0f8 0000 0101 080a 1d43 a086  ..p..........C..
> > 0x0030:  015f dff3 8000 005c a561 424c 0000 0001  ._.....\.aBL....
> > 0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
> > 0x0050:  0000 0011 0000 0001 0000 0000 0000 0050  ...............P
> > 0x0060:  5908 f554 3536 88ea 5908 f555 01ff bf76  Y..T56..Y..U...v
> > 0x0070:  0000 0000 0000 0001 0000 0000 0000 0050  ...............P
> > 0x0080:  5908 f554 3536 88ea 5908 f555 01ff bf76  Y..T56..Y..U...v
> > 0x0090:  0000 0000                                ....
> > 16:08:37.101774 IP 10.151.3.81.disclose > 10.151.3.161.nfsd: Flags
> > [.], ack 813, win 605, options [nop,nop,TS val 23060468 ecr
> > 490971270,nop,nop,sack 1 {4294966865:4294966961}], length 0
> > 0x0000:  4500 0040 6d15 4000 4006 b183 0a97 0351  E.. at m.@. at ......Q
> > 0x0010:  0a97 03a1 029b 0801 bd27 ebe8 c15f 7c09  .........'..._|.
> > 0x0020:  b010 025d 1c52 0000 0101 080a 015f dff4  ...].R......._..
> > 0x0030:  1d43 a086 0101 050a c15f 772d c15f 778d  .C......._w-._w.
> >
> > On Thu, May 4, 2017 at 11:34 AM, Wiles, Keith <keith.wiles at intel.com>
> > wrote:
> >
> >>
> >> > On May 4, 2017, at 8:04 AM, Matt Laswell <laswell at infinite.io> wrote:
> >> >
> >> > Hey Folks,
> >> >
> >> > I'm seeing some strange behavior with regard to the RSS hash values
> >> > in
> >> my
> >> > applications and was hoping somebody might have some pointers on
> >> > where
> >> to
> >> > look.  In my application, I'm using RSS to divide work among
> >> > multiple cores, each of which services a single RX queue.  When
> >> > dealing with a single long-lived TCP connection, I occasionally see
> >> > packets going to
> >> the
> >> > wrong core.   That is, almost all of the packets in the connection go to
> >> > core 5 in this case, but every once in a while, one goes to core 0
> >> instead.
> >> >
> >> > Upon further investigation, I find two problems are occurring.  The
> >> first
> >> > is that problem packets have the RSS hash value in the mbuf
> >> > incorrectly
> >> set
> >> > to zero.  They are therefore put in queue zero, where they are read
> >> > by
> >> core
> >> > zero.  Other packets from the same connection that occur
> >> > immediately
> >> before
> >> > and after the packet in question have the correct hash value and
> >> therefore
> >> > go to a different core.   The second problem is that we sometimes see
> >> > packets in which the RSS hash in the mbuf appears correct, but the
> >> packets
> >> > are incorrectly put into queue zero.  As with the first, this
> >> > results in the wrong core getting the packet.  Either one of these
> >> > confuses the
> >> state
> >> > tracking we're doing per-core.
> >> >
> >> > A few details:
> >> >
> >> >   - Using an Intel X540-AT2 NIC and the igb_uio driver
> >> >   - DPDK 16.04
> >> >   - A particular packet in our workflow always encounters this problem.
> >> >   - Retransmissions of the packet in question also encounter the problem
> >> >   - The packet is IPv4, with header length of 20 (so no options), no
> >> >   fragmentation.
> >> >   - The only differences I can see in the IP header between packets that
> >> >   get the right hash value and those that get the wrong one are in
> >> > the
> >> IP ID,
> >> >   total length, and checksum fields.
> >> >   - Using ETH_RSS_IPV4
> >> >   - The packet is TCP with about 100 bytes of payload - it's not a jumbo
> >> >   or a runt
> >> >   - We fill the key in with 0x6d5a to get symmetric hashing of both
> >> sides
> >> >   of the connection
> >> >   - We only configure RSS information at boot; things like the key or
> >> >   header fields are not being changed dynamically
> >> >   - Traffic load is light when the problem occurs
> >> >
> >> > Is anybody aware of an errata, either in the NIC or the PMD's
> >> configuration
> >> > of it that might explain something like this?   Failing that, if you ran
> >> > into this sort of behavior, how would you approach finding the
> >> > reason
> >> for
> >> > the error?  Every failure mode I can think of would tend to affect
> >> > all
> >> of
> >> > the packets in the connection consistently, even if incorrectly.
> >>
> >> Just to add more information to this email, can you provide hexdumps
> >> of the packets to help someone maybe spot the problem?
> >>
> >> Need the previous OK packet plus the one after it and the failing
> >> packets you are seeing.
> >>
> >> I do not know why this is happening as I do not know of any errata to
> >> explain this issue.
> >>
> >> >
> >> > Thanks in advance for any ideas.
> >> >
> >> > --
> >> > Matt Laswell
> >> > laswell at infinite.io
> >>
> >> Regards,
> >> Keith
> >>
> >>
> >
> It appears that the list stripped off the pcap attachment that I included in the
> above email.  If anybody is interested in looking at it, please let me know and I'd
> be happy to send it off-list.
> 
> --
> Matt Laswell
> laswell at infinite.io


More information about the dev mailing list