Bug 866 - Huge Packet drops observed on vmxnet3 after updating DPDK version from 20.11 to 21.08
Summary: Huge Packet drops observed on vmxnet3 after updating DPDK version from 20.11 ...
Status: UNCONFIRMED
Alias: None
Product: DPDK
Classification: Unclassified
Component: ethdev (show other bugs)
Version: 21.08
Hardware: x86 Linux
: Normal major
Target Milestone: ---
Assignee: dev
URL:
Depends on:
Blocks:
 
Reported: 2021-10-29 13:01 CEST by Sahithi Singam
Modified: 2021-10-29 13:01 CEST (History)
0 users



Attachments

Description Sahithi Singam 2021-10-29 13:01:41 CEST
On the HOST, we are using VMWARE ESXi 6.7 and on one of the hosts, we are running DPDK based custom application over vmxnet3 devices. 
When we upgraded DPDK version in our application from 20.11 to 21.08, we saw a huge difference in the performance numbers. 
In our tests, we have observed that there is around packet drops of 1.5% with DPDK21.08 version and these packet drops were not visible either in the guest DPDK port stats or in the HOST vmxnet3 stats. 

On  further debugging, we have nailed out that this issue is due to recent commit "046f1161956777e3afb13504acbe8df2ec3a383c net/vmxnet3: support MSI-X interrupt" 

Following code change in above commit is resulting in this issue. 

=====================================================================
+		if (hw->intr.lsc_only)
+			tqd->conf.intrIdx = 1;
+		else
+			tqd->conf.intrIdx = intr_handle->intr_vec[i];
...
...
...
+ 		if (hw->intr.lsc_only)
+			rqd->conf.intrIdx = 1;
+		else
+			rqd->conf.intrIdx = intr_handle->intr_vec[i];
======================================================================

We are using igb_uio and link status interrupt for link status detection. Due to which our application is landing in the lsc_only if condition and as per code RX/TX queues are using interrupts at index 1. Though interrupt at index 1 is not enabled in the guest DPDK code, it is resulting in the packet drops. We are seeing following events at host stats. 

[root@vmw-node1:~] vsish -e cat  /net/portsets/switch1/ports/100663342/vmxnet3/intrs/0/stats
stats of the individual intr {
   actions posted:0
   actions posted with hint:0
   actions avoided:0
}
 
[root@vmw-node1:~] vsish -e cat  /net/portsets/switch1/ports/100663342/vmxnet3/intrs/1/stats
stats of the individual intr {
   actions posted:1
   actions posted with hint:1
   actions avoided:27207273
}

When I reverted above code changes to the changes as in DPDK version 20.11, i.e as below , we are not seeing the packet drops. 
=========================================================
tqd->conf.intrIdx      = txq->comp_ring.intr_idx;
rqd->conf.intrIdx         = rxq->comp_ring.intr_idx;
=========================================================

Note You need to log in before you can comment on or make changes to this bug.