1399 – Intel X710 (i40e): cannot configure RSS with testpmd and rte_flow API

Bug 1399 - Intel X710 (i40e): cannot configure RSS with testpmd and rte_flow API

Summary: Intel X710 (i40e): cannot configure RSS with testpmd and rte_flow API

Status:	UNCONFIRMED

Alias:	None

Product:	DPDK
Classification:	Unclassified
Component:	testpmd (show other bugs)
Version:	23.11
Hardware:	x86 Linux

Importance:	Normal normal
Target Milestone:	---
Assignee:	dev

URL:

Depends on:
Blocks:

Reported:	2024-03-12 11:14 CET by Anton Vanda
Modified:	2024-03-12 11:14 CET (History)
CC List:	0 users

Attachments
sample gre traffic (814.94 KB, application/vnd.tcpdump.pcap) 2024-03-12 11:14 CET, Anton Vanda	Details
Add an attachment (proposed patch, testcase, etc.)

Description Anton Vanda 2024-03-12 11:14:51 CET

Created attachment 277 [details]
sample gre traffic

Environment:
- DPDK 21.11 and DPDK 23.11
- NIC: 'Ethernet Controller X710 for 10GbE SFP+ 1572'
- NIC firmware: 9.40 0x8000ed0e 1.3429.0

I've got problem with Intel X710 NIC (we have other NICs, and they work just fine).
RSS don't work for ETH-IPV4-GRE-ERSPAN packets.
We need to setup 2-tuple RSS using IPV4 src and dst addresses only. 
It's OK for all NICs except X710. With x710 we have all packets (with unique IP addresses) placed in the same RX queue.

I can reproduce it with testpmd (tried 21.11 and 23.11).

# ./build/app/dpdk-testpmd -l 0,2,4,6,8 -n 4 -- -i --portmask=0x1 --nb-cores=4 --rxq=4 --forward-mode=rxonly

testpmd> set verbose 1
Change verbose level from 0 to 1

testpmd> flow create 0 ingress pattern eth / ipv4 / end actions rss types ipv4 end queues end / end
Flow rule #0 created

testpmd> flow list 0
ID      Group   Prio    Attr    Rule
0       0       0       i--     ETH IPV4 => RSS

testpmd> start
rxonly packet forwarding - ports=1 - cores=4 - streams=4 - NUMA support enabled, MP allocation mode: native
Logical Core 2 (socket 0) forwards packets on 1 streams:
  RX P=0/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00
Logical Core 4 (socket 0) forwards packets on 1 streams:
  RX P=0/Q=1 (socket 0) -> TX P=0/Q=1 (socket 0) peer=02:00:00:00:00:00
Logical Core 6 (socket 0) forwards packets on 1 streams:
  RX P=0/Q=2 (socket 0) -> TX P=0/Q=2 (socket 0) peer=02:00:00:00:00:00
Logical Core 8 (socket 0) forwards packets on 1 streams:
  RX P=0/Q=3 (socket 0) -> TX P=0/Q=3 (socket 0) peer=02:00:00:00:00:00

Then I send simple eth-ipv4-udp traffic from another server:

$ tcpdump -v -r /home/pcap/syslog.pcap
14:19:00.572282 IP (tos 0x0, ttl 64, id 51158, offset 0, flags [DF], proto UDP (17), length 237)
    10.0.49.143.53346 > 10.10.192.162.10514: UDP, length 209
14:19:00.572956 IP (tos 0x0, ttl 64, id 51159, offset 0, flags [DF], proto UDP (17), length 136)
    10.0.49.143.53346 > 10.10.192.162.10514: UDP, length 108
...

$ tcpreplay -K -i ens3f1 --pps 1 -l 0 --unique-ip /home/pcap/syslog.pcap
...

In testpmd I see following:

testpmd> port 0/queue 0: received 1 packets
  src=60:22:32:14:09:DD - dst=00:1C:7F:6C:8E:20 - type=0x0800 - length=251 - nb_segs=1 - hw ptype: L2_ETHER L3_IPV4_EXT_UNKNOWN L4_UDP  - sw ptype: L2_ETHER L3_IPV4 L4_UDP  - l2_len=14 - l3_len=20 - l4_len=8 - Receive queue=0x0
  ol_flags: RTE_MBUF_F_RX_L4_CKSUM_GOOD RTE_MBUF_F_RX_IP_CKSUM_GOOD RTE_MBUF_F_RX_OUTER_L4_CKSUM_UNKNOWN 
port 0/queue 0: received 1 packets

!!! RSS wasn't computed at all, and all packets are placed in queue 0.

Speedup TX, and check stats:

testpmd> show fwd stats all

  ------- Forward Stats for RX Port= 0/Queue= 0 -> TX Port= 0/Queue= 0 -------
  RX-packets: 111974         TX-packets: 0              TX-dropped: 0             

  ---------------------- Forward statistics for port 0  ----------------------
  RX-packets: 111974         RX-dropped: 0             RX-total: 111974
  TX-packets: 0              TX-dropped: 0             TX-total: 0
  ----------------------------------------------------------------------------


Now if I add flow rule for eth-ipv4-udp:

testpmd> flow create 0 ingress pattern eth / ipv4 / udp / end actions rss types ipv4 end queues end / end
i40e_check_write_global_reg(): i40e device 0000:4c:00.2 changed global register [0x002676fc]. original: 0x0001801e, new: 0x00018018
Flow rule #1 created

testpmd> show port 0 rss-hash 
RSS functions:
 ipv4-frag ipv4-udp ipv4-other

!!! Then RSS will work for UDP packets (we can see RSS hash=0x2110f405, so hash was calculated):

testpmd> port 0/queue 1: received 1 packets
  src=60:22:32:14:09:DD - dst=00:1C:7F:6C:8E:20 - type=0x0800 - length=251 - nb_segs=1 - RSS hash=0x2110f405 - RSS queue=0x1 - hw ptype: L2_ETHER L3_IPV4_EXT_UNKNOWN L4_UDP  - sw ptype: L2_ETHER L3_IPV4 L4_UDP  - l2_len=14 - l3_len=20 - l4_len=8 - Receive queue=0x1

testpmd> show fwd stats all

  ------- Forward Stats for RX Port= 0/Queue= 0 -> TX Port= 0/Queue= 0 -------
  RX-packets: 8898           TX-packets: 0              TX-dropped: 0             

  ------- Forward Stats for RX Port= 0/Queue= 1 -> TX Port= 0/Queue= 1 -------
  RX-packets: 17796          TX-packets: 0              TX-dropped: 0             

  ------- Forward Stats for RX Port= 0/Queue= 2 -> TX Port= 0/Queue= 2 -------
  RX-packets: 30156          TX-packets: 0              TX-dropped: 0             

  ------- Forward Stats for RX Port= 0/Queue= 3 -> TX Port= 0/Queue= 3 -------
  RX-packets: 17796          TX-packets: 0              TX-dropped: 0             

  ---------------------- Forward statistics for port 0  ----------------------
  RX-packets: 74647          RX-dropped: 0             RX-total: 74647
  TX-packets: 0              TX-dropped: 0             TX-total: 0
  ----------------------------------------------------------------------------

Now if I take another pcap with GRE-ERSPAN traffic

$ tcpdump -v -r /home/pcap/eth-ip-gre-erspan.pcap 
23:41:12.550628 IP (tos 0x0, ttl 255, id 1517, offset 0, flags [DF], proto GRE (47), length 110)
    1.1.1.2 > 198.18.1.2: GREv0, Flags [sequence# present], seq 2029, length 90
        gre-proto-0x88be
...
23:48:56.055419 IP (tos 0x0, ttl 255, id 1899, offset 0, flags [DF], proto GRE (47), length 110)
    2.2.2.2 > 198.18.1.2: GREv0, Flags [sequence# present], seq 5368, length 90
        gre-proto-0x88be
...

$ tcpreplay -K -i ens3f1 --pps 1 -l 0 --unique-ip /home/eth-ip-gre-erspan.pcap

Testpmd reports:

testpmd> port 0/queue 0: received 1 packets
  src=54:7F:EE:CE:25:BC - dst=6C:2B:59:7B:E5:92 - type=0x0800 - length=124 - nb_segs=1 - hw ptype: L2_ETHER L3_IPV4_EXT_UNKNOWN TUNNEL_GRENAT  - sw ptype: L2_ETHER L3_IPV4 TUNNEL_GRE  - l2_len=14 - l3_len=20 - tunnel_len=8 - Receive queue=0x0
  ol_flags: RTE_MBUF_F_RX_L4_CKSUM_GOOD RTE_MBUF_F_RX_IP_CKSUM_GOOD RTE_MBUF_F_RX_OUTER_L4_CKSUM_UNKNOWN 
port 0/queue 0: received 1 packets

!!! Again, RSS wasn't computed at all, and all packets are placed in queue 0.

Let's try to make a flow rule for GRE:

testpmd> flow create 0 ingress pattern eth / ipv4 / gre / end actions rss types ipv4 end queues end / end
port_flow_complain(): Caught PMD error type 13 (specific pattern item): cause: 0x7ffcdca24938, Pattern not supported: Operation not supported

Driver failed to make it. If we try simple drop rule instead of RSS:

testpmd> flow create 0 ingress pattern eth / ipv4 / gre / end actions drop / end
port_flow_complain(): Caught PMD error type 13 (specific pattern item): cause: 0x7ffcdca248f8, Unsupported pattern: Invalid argument

Failed again.

GDB can highlight some details. Let's see what happened inside and which item has the problem address.

Restart testpmd under the debugger:

(gdb) bt
#0  i40e_flow_validate (dev=0x56298c45fec0 <rte_eth_devices>, attr=0x7ffeb453a658, pattern=0x7ffeb453a6e8, actions=0x7ffeb453a768, error=0x7ffeb4538570) at ../drivers/net/i40e/i40e_flow.c:4610
#1  0x00005629882eb059 in i40e_flow_create (dev=0x56298c45fec0 <rte_eth_devices>, attr=0x7ffeb453a658, pattern=0x7ffeb453a6e8, actions=0x7ffeb453a768, error=0x7ffeb4538570) at ../drivers/net/i40e/i40e_flow.c:4638
#2  0x0000562987677d5d in rte_flow_create (port_id=0, attr=0x7ffeb453a658, pattern=0x7ffeb453a6e8, actions=0x7ffeb453a768, error=0x7ffeb4538570) at ../lib/ethdev/rte_flow.c:390
#3  0x00005629871ca71b in port_flow_create (port_id=0, attr=0x7ffeb453a658, pattern=0x7ffeb453a6e8, actions=0x7ffeb453a768, tunnel_ops=0x7ffeb453a664) at ../app/test-pmd/config.c:2057
#4  0x00005629871a92ad in cmd_flow_parsed (in=0x7ffeb453a650) at ../app/test-pmd/cmdline_flow.c:8722
#5  0x00005629871a94c0 in cmd_flow_cb (arg0=0x7ffeb453a650, cl=0x56298df99190, arg2=0x0) at ../app/test-pmd/cmdline_flow.c:8784
#6  0x000056298765d372 in cmdline_parse (cl=0x56298df99190, buf=0x56298df991d8 "flow create 0 ingress pattern eth / ipv4 / gre / end actions drop / end\n") at ../lib/cmdline/cmdline_parse.c:290
#7  0x000056298765b877 in cmdline_valid_buffer (rdl=0x56298df991a0, buf=0x56298df991d8 "flow create 0 ingress pattern eth / ipv4 / gre / end actions drop / end\n", size=73) at ../lib/cmdline/cmdline.c:26
#8  0x0000562987660287 in rdline_char_in (rdl=0x56298df991a0, c=10 '\n') at ../lib/cmdline/cmdline_rdline.c:446
#9  0x000056298765bc6c in cmdline_in (cl=0x56298df99190, buf=0x7ffeb453e7af "\n\320\347S\264\376\177", size=1) at ../lib/cmdline/cmdline.c:148
#10 0x000056298765beb3 in cmdline_interact (cl=0x56298df99190) at ../lib/cmdline/cmdline.c:222
#11 0x00005629871a10a3 in prompt () at ../app/test-pmd/cmdline.c:18001
#12 0x0000562987230d6b in main (argc=9, argv=0x7ffeb453e960) at ../app/test-pmd/testpmd.c:4263

(gdb) p *error
$22 = {type = RTE_FLOW_ERROR_TYPE_ITEM, cause = 0x7ffeb453a6e8, message = 0x56298ba09e1a "Unsupported pattern"}

(gdb) p *(struct rte_flow_item*)0x7ffeb453a6e8
$27 = {type = RTE_FLOW_ITEM_TYPE_ETH, spec = 0x0, last = 0x0, mask = 0x0}

Looks like parser cannot handle the whole set of patterns.

What should I do to make RSS work?

Note You need to log in before you can comment on or make changes to this bug.