As shown in [1],a geneve packet with TSO request is transmitted, and resulting packets mismatch because of the wrong checksum in the outer UDP header (it is the same as the one in the original superframe). However, since the test sees capability bit TX_OFFLOAD_OUTER_UDP_CKSUM in device info, the checksum is expected to be offloaded. Both X710 and E810 encounter the same issue. Upon reviewing the unloading portion of the code, it's evident that only adding capabilities without unloading part of the code was implemented. [1] https://github.com/openvswitch/ovs-issues/issues/321
Can you suggest a simple way to reproduce this issue and to validate any fix? Is it enough to try and send an mbuf with UDP checksum offloading and verify that, or is more setup for tunneling required?
I think vxlan or geneve overlay is required for this. I'm using ovs-dpdk's vhost-user-client for virtual machines, and the communication between two physical nodes is done via a geneve tunnel.On the ovs side, I can see that RTE_ETH_TX_OFFLOAD_OUTER_UDP_CKSUM is set, but when I actually capture packets on the receiving end, the outer checksum is incorrect. Disabling RTE_ETH_TX_OFFLOAD_OUTER_UDP_CKSUM on the ovs side seems to resolve the issue,As shown in [1]. Additionally, after inspecting the DPDK driver code, it seems that it only advertises the feature but doesn't actually implement it,As shown in [2]. [1]https://patchwork.ozlabs.org/project/openvswitch/patch/1711095597-5729-1-git-send-email-junwang01@cestc.cn/ [2]https://github.com/DPDK/dpdk/blob/main/drivers/net/i40e/i40e_rxtx.c#L301
According to Intel(R) Ethernet Controller X710/ XXV710/XL710 Datasheet, section 8.4.4.2, "Tunneling UDP headers and GRE header are not offloaded while the X710/XXV710/XL710 leaves their checksum field as is".So for the X710 series, DPDK should not advertise support for outer UDP offload, which appears to be a bug in DPDK. https://cdrdv2-public.intel.com/332464/332464_710_Series_Datasheet_v_4_1.pdf According to Intel(R) Ethernet Controller E810 Datasheet,section 10.5.8.3, The tunneling UDP checksum offload appears to be supported. https://www.intel.com/content/www/us/en/content-details/613875/intel-ethernet-controller-e810-datasheet.html
Indeed, I had read the datasheets and came to the same conclusions. I spent some time on this topic and I think we are missing the setup of the pseudo header checksum on the dpdk side: diff --git a/lib/net/rte_net.h b/lib/net/rte_net.h index ef3ff4c6fd..73a6618a66 100644 --- a/lib/net/rte_net.h +++ b/lib/net/rte_net.h @@ -121,7 +121,8 @@ rte_net_intel_cksum_flags_prepare(struct rte_mbuf *m, uint64_t ol_flags) * no offloads are requested. */ if (!(ol_flags & (RTE_MBUF_F_TX_IP_CKSUM | RTE_MBUF_F_TX_L4_MASK | RTE_MBUF_F_TX_TCP_SEG | - RTE_MBUF_F_TX_UDP_SEG | RTE_MBUF_F_TX_OUTER_IP_CKSUM))) + RTE_MBUF_F_TX_UDP_SEG | RTE_MBUF_F_TX_OUTER_IP_CKSUM | + RTE_MBUF_F_TX_OUTER_UDP_CKSUM))) return 0; if (ol_flags & (RTE_MBUF_F_TX_OUTER_IPV4 | RTE_MBUF_F_TX_OUTER_IPV6)) { @@ -135,6 +136,22 @@ rte_net_intel_cksum_flags_prepare(struct rte_mbuf *m, uint64_t ol_flags) struct rte_ipv4_hdr *, m->outer_l2_len); ipv4_hdr->hdr_checksum = 0; } + if (ol_flags & RTE_MBUF_F_TX_OUTER_UDP_CKSUM) { + if (ol_flags & RTE_MBUF_F_TX_OUTER_IPV4) { + ipv4_hdr = rte_pktmbuf_mtod_offset(m, struct rte_ipv4_hdr *, + m->outer_l2_len); + udp_hdr = (struct rte_udp_hdr *)((char *)ipv4_hdr + + m->outer_l3_len); + udp_hdr->dgram_cksum = rte_ipv4_phdr_cksum(ipv4_hdr, m->ol_flags); + } else { + ipv6_hdr = rte_pktmbuf_mtod_offset(m, struct rte_ipv6_hdr *, + m->outer_l2_len); + /* non-TSO udp */ + udp_hdr = rte_pktmbuf_mtod_offset(m, struct rte_udp_hdr *, + m->outer_l2_len + m->outer_l3_len); + udp_hdr->dgram_cksum = rte_ipv6_phdr_cksum(ipv6_hdr, m->ol_flags); + } + } } /* With a E810 + OVS (vxlan + tso): Before: 10:00:27.152204 50:7c:6f:3c:0c:26 > 50:7c:6f:3c:10:5a, ethertype 802.1Q (0x8100), length 148: vlan 200, p 0, ethertype IPv6, (hlim 64, next-header UDP (17) payload length: 90) 2001::2.36209 > 2001::1.vxlan: [bad udp cksum 0x35cc -> 0xf55b!] VXLAN, flags [I] (0x08), vni 1 52:54:00:00:11:01 > 36:44:f8:6f:0b:1a, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60) 172.31.22.2.targus-getdata1 > 172.31.22.1.41186: Flags [S.], cksum 0x0ee5 (correct), seq 3569139204, ack 2007533520, win 26960, options [mss 1360,sackOK,TS val 998328331 ecr 3390027474,nop,wscale 7], length 0 After: 10:09:51.088760 50:7c:6f:3c:0c:26 > 50:7c:6f:3c:10:5a, ethertype 802.1Q (0x8100), length 148: vlan 200, p 0, ethertype IPv6, (hlim 64, next-header UDP (17) payload length: 90) 2001::2.38344 > 2001::1.vxlan: [udp sum ok] VXLAN, flags [I] (0x08), vni 1 52:54:00:00:11:01 > 36:44:f8:6f:0b:1a, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60) 172.31.22.2.targus-getdata1 > 172.31.22.1.53988: Flags [S.], cksum 0xb25c (correct), seq 1558118886, ack 381507235, win 26960, options [mss 1360,sackOK,TS val 998892263 ecr 3390639231,nop,wscale 7], length 0
I posted a series trying to fix issues related to outer udp checksum. https://patchwork.dpdk.org/project/dpdk/list/?series=31687&state=%2A&archive=both
I've used your V2 version submitted to the DPDK community, and everything seems to be working fine now. It's great! https://patches.dpdk.org/project/dpdk/list/?series=31689