[dpdk-dev] [dpdk-users] RSS Hash not working for XL710/X710 NICs for some RX mbuf sizes

Take Ceara dumitru.ceara at gmail.com
Thu Jul 21 12:58:29 CEST 2016
Previous message: [dpdk-dev] [dpdk-users] RSS Hash not working for XL710/X710 NICs for some RX mbuf sizes
Next message: [dpdk-dev] [dpdk-users] RSS Hash not working for XL710/X710 NICs for some RX mbuf sizes
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Hi Beilei,

On Wed, Jul 20, 2016 at 3:59 AM, Xing, Beilei <beilei.xing at intel.com> wrote:
> Hi Ceara,
>
>> -----Original Message-----
>> From: Take Ceara [mailto:dumitru.ceara at gmail.com]
>> Sent: Tuesday, July 19, 2016 10:59 PM
>> To: Xing, Beilei <beilei.xing at intel.com>
>> Cc: Zhang, Helin <helin.zhang at intel.com>; Wu, Jingjing
>> <jingjing.wu at intel.com>; dev at dpdk.org
>> Subject: Re: [dpdk-dev] [dpdk-users] RSS Hash not working for XL710/X710
>> NICs for some RX mbuf sizes
>>
>> Hi Beilei,
>>
>> I changed the way I run testmpd to:
>>
>> testpmd -c 0x331 -w 0000:82:00.0 -w 0000:83:00.0 -- --mbuf-size 1152 --rss-ip -
>> -rxq=2 --txpkts 1024 -i
>>
>> As far as I understand this will allocate mbufs with the same size I was using
>> in my test (--mbuf-size seems to include the mbuf headroom therefore 1152
>> = 1024 + 128 headroom).
>>
>> testpmd> start tx_first
>>   io packet forwarding - CRC stripping disabled - packets/burst=32
>>   nb forwarding cores=1 - nb forwarding ports=2
>>   RX queues=2 - RX desc=128 - RX free threshold=32
>>   RX threshold registers: pthresh=8 hthresh=8 wthresh=0
>>   TX queues=1 - TX desc=512 - TX free threshold=32
>>   TX threshold registers: pthresh=32 hthresh=0 wthresh=0
>>   TX RS bit threshold=32 - TXQ flags=0xf01
>> testpmd> show port stats all
>>
>>   ######################## NIC statistics for port 0
>> ########################
>>   RX-packets: 18817613   RX-missed: 5          RX-bytes:  19269115888
>>   RX-errors: 0
>>   RX-nombuf:  0
>>   TX-packets: 18818064   TX-errors: 0          TX-bytes:  19269567464
>>
>> ##########################################################
>> ##################
>>
>>   ######################## NIC statistics for port 1
>> ########################
>>   RX-packets: 18818392   RX-missed: 5          RX-bytes:  19269903360
>>   RX-errors: 0
>>   RX-nombuf:  0
>>   TX-packets: 18817979   TX-errors: 0          TX-bytes:  19269479424
>>
>> ##########################################################
>> ##################
>>
>> Ttraffic is sent/received. However, I couldn't find any way to verify that the
>> incoming mbufs actually have the mbuf->hash.rss field set except for starting
>> test-pmd with gdb and setting a breakpoint in the io fwd engine. After doing
>> that I noticed that none of the incoming packets has the PKT_RX_RSS_HASH
>> flag set in ol_flags... I guess for some reason test-pmd doesn't actually
>> configure RSS in this case but I fail to see where.
>>
>
> Actually there's a way to check mbuf->hash.rss, you need set forward mode to "rxonly", and set verbose to 1.
> I run testpmd with the configuration you used, and found i40e RSS works well.
> With the following steps, you can see RSS hash value and receive queue, and PKT_RX_RSS_HASH is set too.
> I think you can use the same way to check what you want.
>
> ./testpmd -c fffff -n 4 -- -i --coremask=0xffffe --rxq=16 --txq=16 --mbuf-size 1152 --rss-ip --txpkts 1024
> testpmd> set verbose 1
> testpmd> set fwd rxonly
> testpmd> start
> testpmd> port 0/queue 1: received 1 packets
>   src=00:00:01:00:0F:00 - dst=68:05:CA:32:03:4C - type=0x0800 - length=1020 - nb
>  - Receive queue=0x1
>   PKT_RX_RSS_HASH
> port 0/queue 0: received 1 packets
>   src=00:00:01:00:0F:00 - dst=68:05:CA:32:03:4C - type=0x0800 - length=1020 - nb_segs=1 - RSS hash=0x4e949f23 - RSS queue=0x0Unknown packet type
>  - Receive queue=0x0
>   PKT_RX_RSS_HASH
> port 0/queue 8: received 1 packets
>   src=00:00:01:00:0F:00 - dst=68:05:CA:32:03:4C - type=0x0800 - length=1020 - nb_segs=1 - RSS hash=0xa3c78b2b - RSS queue=0x8Unknown packet type
>  - Receive queue=0x8
>   PKT_RX_RSS_HASH
> port 0/queue 5: received 1 packets
>   src=00:00:01:00:0F:00 - dst=68:05:CA:32:03:4C - type=0x0800 - length=1020 - nb_segs=1 - RSS hash=0xe29b3d36 - RSS queue=0x5Unknown packet type
>  - Receive queue=0x5
>   PKT_RX_RSS_HASH
>

Following your testpmd example run I managed to replicate the problem
on my dpdk 16.04 setup like this:

I have two X710 adapters connected back to back:
$ ./tools/dpdk_nic_bind.py -s

Network devices using DPDK-compatible driver
============================================
0000:01:00.3 'Ethernet Controller X710 for 10GbE SFP+' drv=igb_uio unused=
0000:81:00.3 'Ethernet Controller X710 for 10GbE SFP+' drv=igb_uio unused=

The firmware of the two adapters is up to date with the latest
version: 5.04 (f5.0.40043 a1.5 n5.04 e24cd)

I run testpmd with mbuf-size 1152 and txpktsize 1024 such that upon
receival the whole mbuf (except headroom) is filled.
I enabled RX IP checksum in hw and RX RSS hashing for UDP.
With test-pmd forward mode "rxonly" and verbose 1 I see that incoming
packets have PKT_RX_RSS_HASH set but the hash value is 0.

./testpmd -c ffff1 -n 4 -w 0000:01:00.3 -w 0000:81:00.3 -- -i
--coremask=0xffff0 --rxq=16 --txq=16 --mbuf-size 1152 --txpkts 1024
--enable-rx-cksum --rss-udp
[...]
testpmd> set verbose 1
Change verbose level from 0 to 1
testpmd> set fwd rxonly
Set rxonly packet forwarding mode
testpmd> start tx_first
  rxonly packet forwarding - CRC stripping disabled - packets/burst=32
  nb forwarding cores=16 - nb forwarding ports=2
  RX queues=16 - RX desc=128 - RX free threshold=32
  RX threshold registers: pthresh=8 hthresh=8 wthresh=0
  TX queues=16 - TX desc=512 - TX free threshold=32
  TX threshold registers: pthresh=32 hthresh=0 wthresh=0
  TX RS bit threshold=32 - TXQ flags=0xf01
port 0/queue 1: received 32 packets
  src=68:05:CA:38:6D:63 - dst=02:00:00:00:00:01 - type=0x0800 -
length=1024 - nb_segs=1 - RSS hash=0x0 - RSS queue=0x1 - (outer) L2
type: ETHER - (outer) L3 type: IPV4_EXT_UNKNOWN - (outer) L4 type: UDP
- Tunnel type: Unknown - Inner L2 type: Unknown - Inner L3 type:
Unknown - Inner L4 type: Unknown
 - Receive queue=0x1
  PKT_RX_RSS_HASH
  src=68:05:CA:38:6D:63 - dst=02:00:00:00:00:01 - type=0x0800 -
length=1024 - nb_segs=1 - RSS hash=0x0 - RSS queue=0x1 - (outer) L2
type: ETHER - (outer) L3 type: IPV4_EXT_UNKNOWN - (outer) L4 type: UDP
- Tunnel type: Unknown - Inner L2 type: Unknown - Inner L3 type:
Unknown - Inner L4 type: Unknown
 - Receive queue=0x1
  PKT_RX_RSS_HASH

If I use a different mbuf-size, for example 2048, everything is fine:

./testpmd -c ffff1 -n 4 -w 0000:01:00.3 -w 0000:81:00.3 -- -i
--coremask=0xffff0 --rxq=16 --txq=16 --mbuf-size 2048 --txpkts 1024
--enable-rx-cksum --rss-udp
[...]
testpmd> set verbose 1
Change verbose level from 0 to 1
testpmd> set fwd rxonly
Set rxonly packet forwarding mode
testpmd> start tx_first
  rxonly packet forwarding - CRC stripping disabled - packets/burst=32
  nb forwarding cores=16 - nb forwarding ports=2
  RX queues=16 - RX desc=128 - RX free threshold=32
  RX threshold registers: pthresh=8 hthresh=8 wthresh=0
  TX queues=16 - TX desc=512 - TX free threshold=32
  TX threshold registers: pthresh=32 hthresh=0 wthresh=0
  TX RS bit threshold=32 - TXQ flags=0xf01
port 0/queue 1: received 32 packets
  src=68:05:CA:38:6D:63 - dst=02:00:00:00:00:01 - type=0x0800 -
length=1024 - nb_segs=1 - RSS hash=0x5263c3f2 - RSS queue=0x1 -
(outer) L2 type: ETHER - (outer) L3 type: IPV4_EXT_UNKNOWN - (outer)
L4 type: UDP - Tunnel type: Unknown - Inner L2 type: Unknown - Inner
L3 type: Unknown - Inner L4 type: Unknown
 - Receive queue=0x1
  PKT_RX_RSS_HASH

Another weird thing I noticed is when I run test-pmd without
--enable-rx-cksum (which is the default mode) then the RSS flag doesn
get set at all. Instead the PKT_RX_FDIR flag gets set. This happens
even though fdir_mode is set to RTE_FDIR_MODE_NONE in the device
configuration:

./testpmd -c ffff1 -n 4 -w 0000:01:00.3 -w 0000:81:00.3 -- -i
--coremask=0xffff0 --rxq=16 --txq=16 --mbuf-size 1152 --txpkts 1024
--rss-udp
[...]
testpmd> set verbose 1
Change verbose level from 0 to 1
testpmd> set fwd rxonly
Set rxonly packet forwarding mode
testpmd> start tx_first
  rxonly packet forwarding - CRC stripping disabled - packets/burst=32
  nb forwarding cores=16 - nb forwarding ports=2
  RX queues=16 - RX desc=128 - RX free threshold=32
  RX threshold registers: pthresh=8 hthresh=8 wthresh=0
  TX queues=16 - TX desc=512 - TX free threshold=32
  TX threshold registers: pthresh=32 hthresh=0 wthresh=0
  TX RS bit threshold=32 - TXQ flags=0xf01
port 0/queue 1: received 16 packets
  src=68:05:CA:38:6D:63 - dst=02:00:00:00:00:01 - type=0x0800 -
length=1024 - nb_segs=2 - FDIR matched hash=0xc3f2 ID=0x5263 Unknown
packet type
 - Receive queue=0x1
  PKT_RX_FDIR
  src=68:05:CA:38:6D:63 - dst=02:00:00:00:00:01 - type=0x0800 -
length=1024 - nb_segs=2 - FDIR matched hash=0xc3f2 ID=0x5263 Unknown
packet type
 - Receive queue=0x1
  PKT_RX_FDIR

Please let me know if there's more debug info that might be of
interest in troubleshooting the RSS=0 issue.

Thanks,
Dumitru

> /Beilei
>
>> Thanks,
>> Dumitru
>>
Previous message: [dpdk-dev] [dpdk-users] RSS Hash not working for XL710/X710 NICs for some RX mbuf sizes
Next message: [dpdk-dev] [dpdk-users] RSS Hash not working for XL710/X710 NICs for some RX mbuf sizes
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
More information about the dev mailing list