[dpdk-dev] Wrong TCP checksum of packets sent by Linux guest (virtIO/vhost)

[Franck Baudin]

Hi Huawei,

On 09/04/14 04:54, Xie, Huawei wrote:
> Hi Franck:
> I checked your original thread.
> root at linux-native:~# tcpdump -vnei eth0
> tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
> 17:27:09.262926 00:1b:21:b9:9b:2c > 52:54:00:51:51:12, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 47743, offset 0, flags [DF], proto TCP (6), length 60)
>     1.1.1.3.34272 > 1.1.1.2.5555: Flags [S], cksum 0x0435 (incorrect -> 0xb2dd), seq 1963818356, win 14600, options [mss 1460,sackOK,TS val 49530635 ecr 0,nop,wscale 7], length 0
> 17:27:09.263220 52:54:00:51:51:12 > 00:1b:21:b9:9b:2c, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 64, id 3367, offset 0, flags [DF], proto TCP (6), length 40)
>      1.1.1.2.5555 > 1.1.1.3.34272: Flags [R.], cksum 0x1db4 (correct), seq 0, ack 1963818357, win 0, length 0
>
> The packet from the guest, received on the native machine, has "correct" checksum 0x1db4. The TCP connection is refused is due to this packet has R flag. Could you check if it is the firewall that blocks the
> connection?
>
> Besides, I did ssh tcp test between two guest VMs,  the ssh connection could be established correctly with tx checksum off. I didn't use OVDK, but set the arp table and route table manually.
> The packet flow is
> 	Guest A(virtio)<->vhost example->Guest B(virtio)
I re-made a test between two identical VMs, UDP and ICMP are fine. Some 
observations:
on both client & server:

ethtool -k eth1 | grep ": on"
generic-receive-offload: on
highdma: on [fixed]
rx-vlan-filter: on [fixed]

ethtool -k eth1
Features for eth1:
rx-checksumming: off [fixed]
tx-checksumming: off
         tx-checksum-ipv4: off [fixed]
         tx-checksum-unneeded: off [fixed]
         tx-checksum-ip-generic: off [fixed]
         tx-checksum-ipv6: off [fixed]
         tx-checksum-fcoe-crc: off [fixed]
         tx-checksum-sctp: off [fixed]
scatter-gather: off
         tx-scatter-gather: off [fixed]
         tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
         tx-tcp-segmentation: off [fixed]
         tx-tcp-ecn-segmentation: off [fixed]
         tx-tcp6-segmentation: off [fixed]
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: off [requested on]
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]

On the client side:
11:00:23.296592 IP (tos 0x0, ttl 64, id 25239, offset 0, flags [DF], 
proto TCP
(6), length 60)
     1.1.1.2.47192 > 1.1.1.1.dsf: Flags [S], cksum 0x0433 (incorrect ->
0xe3bd), seq 3885781956, win 14600, options [mss 1460,sackOK,TS val 
4294914387
ecr 0,nop,wscale 5], length 0
         0x0000:  4500 003c 6297 4000 4006 d420 0101 0102 E..<b. at .@.......
         0x0010:  0101 0101 b858 022b e79c 53c4 0000 0000 .....X.+..S.....
         0x0020:  a002 3908 0433 0000 0204 05b4 0402 080a ..9..3..........
         0x0030:  ffff 3153 0000 0000 0103 0305 ..1S........
11:00:23.296881 IP (tos 0x0, ttl 64, id 6543, offset 0, flags [DF], 
proto TCP
(6), length 40)
     1.1.1.1.dsf > 1.1.1.2.47192: Flags [R.], cksum 0xb5e6 (correct), seq 0,
ack 3885781957, win 0, length 0
         0x0000:  4500 0028 198f 4000 4006 1d3d 0101 0101 E..(.. at .@..=....
         0x0010:  0101 0102 022b b858 0000 0000 e79c 53c5 .....+.X......S.
         0x0020:  5014 0000 b5e6 0000                      P.......

On the server side:
11:00:23.631991 IP (tos 0x0, ttl 64, id 25239, offset 0, flags [DF], 
proto TCP
(6), length 60)
     1.1.1.2.47192 > 1.1.1.1.dsf: Flags [S], cksum 0xe3bd (correct), seq
3885781956, win 14600, options [mss 1460,sackOK,TS val 4294914387 ecr
0,nop,wscale 5], length 0
         0x0000:  4500 003c 6297 4000 4006 d420 0101 0102 E..<b. at .@.......
         0x0010:  0101 0101 b858 022b e79c 53c4 0000 0000 .....X.+..S.....
         0x0020:  a002 3908 e3bd 0000 0204 05b4 0402 080a ..9.............
         0x0030:  ffff 3153 0000 0000 0103 0305 ..1S........
11:00:23.632034 IP (tos 0x0, ttl 64, id 6543, offset 0, flags [DF], 
proto TCP
(6), length 40)
     1.1.1.1.dsf > 1.1.1.2.47192: Flags [R.], cksum 0xb5e6 (correct), seq 0,
ack 3885781957, win 0, length 0
         0x0000:  4500 0028 198f 4000 4006 1d3d 0101 0101 E..(.. at .@..=....
         0x0010:  0101 0102 022b b858 0000 0000 e79c 53c5 .....+.X......S.
         0x0020:  5014 0000 b5e6 0000                      P.......


On the client, ethtool announce that TX checksum offload is disabled, 
but tcpdump show it's not. On the server side, all packets looks fine 
from a checksum perspective but the SYN is answered by a RST. Please 
note that the same images behave correctly with a regular openvswitch, 
as long as TX checksuming offloading is disabled.

If anyone has an equivalent setup, functioning properly, thanks for 
sharing its configurations parameters. if not, I might be able to wait 
for vhost backend PMD driver: I will then with testpmd instead of 
dpdk-ovs, to see where is the problem (maybe in my configuration...). 
When so you plan to release the vhost backend PMD driver?

Thanks!
Franck