Bug 950 - Windows DPDK L2fwd- Receiving packets out of sequence
Summary: Windows DPDK L2fwd- Receiving packets out of sequence
Status: UNCONFIRMED
Alias: None
Product: DPDK
Classification: Unclassified
Component: examples (show other bugs)
Version: 21.11
Hardware: Other Windows
: Normal normal
Target Milestone: ---
Assignee: dev
URL:
Depends on:
Blocks:
 
Reported: 2022-03-04 15:21 CET by Burto
Modified: 2022-04-11 17:33 CEST (History)
2 users (show)



Attachments

Description Burto 2022-03-04 15:21:41 CET
I am validating DPDK receive functionality & for this I'm shooting a pcap externally & added code in l2fwd to dump received packets to pcap, the l2fwd dumped pcap have all the packets from shooter but some of them are not in sequence. Shooter is already validated.
Out of order packets are random. For the first run I saw 3 out of order packets and on the second run there were 6 packets out of order.

NIC used : Mellanox Connectx-6Dx l2fwd arguments : dpdk-l2fwd.exe -l 1,2 -n 1 -- -p 0x3 -q 1 -P.

Link of the pcap used : https://wiki.wireshark.org/uploads/__moin_import__/attachments/SampleCaptures/tcp-ecn-sample.pcap

Also opened a thread on this issue on stack overflow link:https://stackoverflow.com/questions/71307805/windows-dpdk-l2fwd-receiving-packets-out-of-sequence
Comment 1 Tal Shnaiderman 2022-03-07 08:12:25 CET
Hi Burto,

Thank you for the information, we will try to reproduce the scenario you've described and update you on the outcome.

Can you please also provide the WINOF2 driver and FW versions you're using on your side? are you using a native setup or a VM? and what is the Windows OS version?

Thanks,

Tal Shnaiderman.
Comment 2 Burto 2022-03-07 11:21:53 CET
(In reply to Tal Shnaiderman from comment #1)
> Hi Burto,
> 
> Thank you for the information, we will try to reproduce the scenario you've
> described and update you on the outcome.
> 
> Can you please also provide the WINOF2 driver and FW versions you're using
> on your side? are you using a native setup or a VM? and what is the Windows
> OS version?
> 
> Thanks,
> 
> Tal Shnaiderman.

Hi Tal Shnaiderman,

Thanks for taking interest in the matter.

WINOF2 Driver: 2.70.24708.0
FW version: 22.28.4512
Setup: Native
OS: Microsoft Windows Server 2019 Standard

Thanks,
Burto
Comment 3 Tal Shnaiderman 2022-03-09 13:45:03 CET
Hi Burto,

I cannot reproduce the issue, we used the tcp-ecn pcap and packets always seem to be in order.

Are the sender and receiver connected back-to-back or via switch?

Regards,

Tal.
Comment 4 Burto 2022-03-10 10:49:39 CET
(In reply to Tal Shnaiderman from comment #3)
> Hi Burto,
> 
> I cannot reproduce the issue, we used the tcp-ecn pcap and packets always
> seem to be in order.
> 
> Are the sender and receiver connected back-to-back or via switch?
> 
> Regards,
> 
> Tal.

Hi Tal,

I used Mellanox Management Tool "Mlx5Cmd.exe -Sniffer" to check received packets at card. I received all packets in the correct order here. 

While receiving it using DPDK "l2fwd" example, there were some packets out of order.

You can also refer this thread on this issue:
https://stackoverflow.com/questions/71307805/windows-dpdk-l2fwd-receiving-packets-out-of-sequence

Regards,
Burto
Comment 5 Burto 2022-03-10 10:50:24 CET
(In reply to Tal Shnaiderman from comment #3)
> Hi Burto,
> 
> I cannot reproduce the issue, we used the tcp-ecn pcap and packets always
> seem to be in order.
> 
> Are the sender and receiver connected back-to-back or via switch?
> 
> Regards,
> 
> Tal.

Hi Tal,

I used Mellanox Management Tool "Mlx5Cmd.exe -Sniffer" to check received packets at card. I received all packets in the correct order here. 

While receiving it using DPDK "l2fwd" example, there were some packets out of order.

You can also refer this thread on this issue:
https://stackoverflow.com/questions/71307805/windows-dpdk-l2fwd-receiving-packets-out-of-sequence

Regards,
Burto
Comment 6 Burto 2022-03-22 13:49:49 CET
Hi Tal,

Any updates on this issue? Were you able to replicate this issue?

I am using proprietary pcap shooter & I have validated shooter's packet sequencing.

Regards,
Burto
Comment 7 Idan Hackmon 2022-03-22 16:52:55 CET
Hey Burto,

Using TREX as a shooter I wasn't able to reproduce this issue.
Is it possible for you to share the shooter's code or any information that you think is relevant about the shooter?

Thanks,
Idan
Comment 8 Burto 2022-03-24 14:37:52 CET
Hi Idan,

While sending packets at 1 packet per milli-sec, the packets are received in order.

Increasing the speed causes more and more packets to be out of order. 
Try replicating it with speed greater than 15 packets per milli-sec.

I am using a proprietary shooter at the moment.
Comment 9 Idan Hackmon 2022-04-06 14:22:10 CEST
Thank you for the useful information,
Can you describe your network topology?
Are the machines connected back to back to via a switch?

Thanks,
Idan
Comment 10 Burto 2022-04-11 17:33:18 CEST
Hi Idan,

Checked with another shooter, "Tcpreplay" & observed same behaviour.

Network topology: tcpreplay --> Packet Broker (1G -> 100G conv) --> Mellanox Adapter

Thanks,
Burto

Note You need to log in before you can comment on or make changes to this bug.