[dpdk-users] .hw_strip_crc weird behavior with small UDP packets

[Julien Castets]

Hi,


I dump the packets received with rte_pktmbuf_dump(stderr, pkt,
pkt->data_len) and I have a strange behavior related to CRC stripping
with small UDP packets. By "small", I mean below the minimal ethernet
payload size, ie. 46 bytes.

* case 1: when my device is configured with hw_strip_crc=1 and the UDP
data is "aaa\n" (4 bytes), my function outputs:

    dump mbuf at 0x0x7fe93bdc47c0, phys=2dbc4840, buf_len=1920
      pkt_len=60, ol_flags=2, nb_segs=1, in_port=1
      segment at 0x0x7fe93bdc47c0, data=0x0x7fe93bdc48c0, data_len=60
      Dump data at [0x7fe93bdc48c0], len=60
    00000000: 38 EA A7 92 70 28 58 97 BD 4F 6F C7 08 00 45 00 | 8...p(X..Oo...E.
    00000010: 00 20 26 D5 00 00 39 11 3B FD D5 24 07 1F A3 AC | . &...9.;..$....
    00000020: 9F 0B EA 5D 10 92 00 0C 23 7F 61 61 61 0A 00 00 | ...]....#.aaa...
    00000030: 00 00 00 00 00 00 00 00 C6 63 C2 CF |  |  |  |  | .........c..

* case 2: when hw_strip_crc=1 and UDP data is
"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\n"
(66 bytes, ie. > to 46):

    dump mbuf at 0x0x7fe93bdd4fc0, phys=2dbd5040, buf_len=1920
      pkt_len=108, ol_flags=2, nb_segs=1, in_port=1
      segment at 0x0x7fe93bdd4fc0, data=0x0x7fe93bdd50c0, data_len=108
      Dump data at [0x7fe93bdd50c0], len=108
    00000000: 38 EA A7 92 70 28 58 97 BD 4F 6F C7 08 00 45 00 | 8...p(X..Oo...E.
    00000010: 00 5E F3 BA 00 00 39 11 6E D9 D5 24 07 1F A3 AC | .^....9.n..$....
    00000020: 9F 0B EA 5D 10 92 00 4A 58 38 61 61 61 61 61 61 | ...]...JX8aaaaaa
    00000030: 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 | aaaaaaaaaaaaaaaa
    00000040: 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 | aaaaaaaaaaaaaaaa
    00000050: 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 | aaaaaaaaaaaaaaaa
    00000060: 61 61 61 61 61 61 61 61 61 61 61 0A |  |  |  |  | aaaaaaaaaaa.

* case 3: when hw_strip_crc=0 and data is "aaa\n" (4 bytes):

    dump mbuf at 0x0x7effcc996140, phys=2db961c0, buf_len=1920
      pkt_len=60, ol_flags=2, nb_segs=1, in_port=1
      segment at 0x0x7effcc996140, data=0x0x7effcc996240, data_len=60
      Dump data at [0x7effcc996240], len=60
    00000000: 38 EA A7 92 70 28 58 97 BD 4F 6F C7 08 00 45 00 | 8...p(X..Oo...E.
    00000010: 00 20 A9 51 00 00 39 11 B9 80 D5 24 07 1F A3 AC | . .Q..9....$....
    00000020: 9F 0B EA 5D 10 92 00 0C 23 7F 61 61 61 0A 00 00 | ...]....#.aaa...
    00000030: 00 00 00 00 00 00 00 00 F5 8D CB 7C |  |  |  |  | ...........|

* case 4: when hw_strip_crc=1 and UDP data is
"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\n"
(66 bytes, ie. > to 46):

    dump mbuf at 0x0x7effcc99ed80, phys=2db9ee00, buf_len=1920
      pkt_len=108, ol_flags=2, nb_segs=1, in_port=1
      segment at 0x0x7effcc99ed80, data=0x0x7effcc99ee80, data_len=108
      Dump data at [0x7effcc99ee80], len=108
    00000000: 38 EA A7 92 70 28 58 97 BD 4F 6F C7 08 00 45 00 | 8...p(X..Oo...E.
    00000010: 00 5E 3B 4E 00 00 39 11 27 46 D5 24 07 1F A3 AC | .^;N..9.'F.$....
    00000020: 9F 0B EA 5D 10 92 00 4A 58 38 61 61 61 61 61 61 | ...]...JX8aaaaaa
    00000030: 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 | aaaaaaaaaaaaaaaa
    00000040: 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 | aaaaaaaaaaaaaaaa
    00000050: 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 | aaaaaaaaaaaaaaaa
    00000060: 61 61 61 61 61 61 61 61 61 61 61 0A |  |  |  |  | aaaaaaaaaaa.


There are a few things I don't understand:

* in case 1, why data_len = 60 and not 56, since hw_strip_crc=1? And
why the CRC is present at the end? (C6 63 C2 CF)
* case 2 seems totally normal: the CRC is not present
* for case 3, from what I understand, if hw_strip_crc is set to 0 the
ixgbe ddriver should remove the CRC
(http://dpdk.org/browse/dpdk/tree/drivers/net/ixgbe/ixgbe_rxtx.c#n1399).
Why pkt len is 60 here? And why is the CRC present at the end (F5 8D
CB 7C)?
* case 4 seems ok, the CRC is absent, as it has been removed by the driver


The context : I am working on a NAT software using DPDK and I want to
offload as much as possible, hence the following code when emitting a
packet:

        udp_hdr->dgram_cksum = 0;
        pkt->ol_flags |= PKT_TX_UDP_CKSUM;
        udp_hdr->dgram_cksum = rte_ipv4_phdr_cksum(ipv4_header(pkt),
                                                   pkt->ol_flags);

When I'm sending a small UDP packet (small = below the minimal size of
the ethernet payload, ie. 46 bytes), the UDP checksum is always wrong.
The output of tcpdump:

        # small packet
        15:02:01.536009 IP (tos 0x0, ttl 54, id 59115, offset 0, flags
[none], proto UDP (17), length 31)
            213.36.7.31.59997 > 10.1.13.48.4242: [bad udp cksum 0x2cf1
-> 0xa612!] UDP, length 3
        # big packet
        15:11:27.240870 IP (tos 0x0, ttl 54, id 5823, offset 0, flags
[none], proto UDP (17), length 112)
            213.36.7.31.59997 > 10.1.13.48.4242: [udp sum ok] UDP, length 84


This is because I only rewrite some fields (source and/or destination
IP) of the rte_mbuf structure received before transmitting it, and the
UDP checksum calculation uses the CRC that hasn't been stripped
correctly as explained above.


My device is configured by calling rte_eth_dev_configure() with the
following eth_conf structure:

    static const struct rte_eth_conf eth_conf = {
        .link_speed=0,
        .link_duplex=0,
        .rxmode = {
            .mq_mode=ETH_MQ_RX_RSS,
            .jumbo_frame=0,
            .max_rx_pkt_len=0,
            .header_split=1,
            .split_hdr_size=64,
            .hw_ip_checksum=1,
            .hw_vlan_filter=1,
            .hw_vlan_strip=1,
            .hw_vlan_extend=0,
            .hw_strip_crc=1,
            .enable_scatter=1,
            .enable_lro=0,
        },
        .txmode = {
            .mq_mode=ETH_MQ_TX_NONE,
            .pvid=0,
            .hw_vlan_reject_tagged=0,
            .hw_vlan_reject_untagged=0,
            .hw_vlan_insert_pvid=0,
        },
        .lpbk_mode=0,
        .rx_adv_conf = {
            .rss_conf = {
                .rss_key=NULL,
                .rss_key_len=0,
                .rss_hf = ETH_RSS_PROTO_MASK,
            },
        },
    };


And some important details:

# lshw -class network
[...]
  *-network:0
       description: Ethernet controller
       product: 82599ES 10-Gigabit SFI/SFP+ Network Connection
[...]


# ./dpdk_nic_bind.py --status

Network devices using DPDK-compatible driver
============================================
0000:08:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection'
drv=igb_uio unused=
[...]


Thanks,
-- 
Julien Castets