[dpdk-users] vmxnet3 tx queue fills then never empties

[Paul Atkins]

Hi,

I am having an issue with the vmxnet3 driver that I have done a lot of 
investigation into (details follow) and have reached the stage where I 
feel I need help from people with access to the vmxnet3 virtual nic code.

The issue is with the dpdk vmxnet3 driver where a tx queue will get 
full, and then never empties.  The trigger for this is sending 1500byte 
packets out of the interface at ~60kpps and then marking the tx 
interface as 'not connected' in vmware.  At this stage the tx queues 
fill up, and when the interface is then marked as 'connected' again in 
vmware some of the tx queues are in a state where they never send any 
further packets.  In my setup that has 4 tx queues with the traffic 
being equally shared over them, I typically see this in 1 of the 4 
queues when the interface comes back up.

If we don't use dpdk and send the traffic via the linux kernel instead 
the problem is not seen, which would suggest a bug in the dpdk driver. 
However, if i modify the linux kernel driver to call 
vmxnet3_tq_tx_complete() inline with the tx code (in the same way that 
it is done for dpdk) then we start to see the bug with the kernel too.  
This suggests a timing issue with calling the tx_complete function. When 
it was called from the interrupt handler the issue was not seen.  
Further, adding debugs into the start of the tx_complete func (before 
any work done) caused the issue to no longer be seen, again suggesting 
some timing race.

I then proceeded to add debug that stores the index of next2fill and 
next2comp in the cmd ring, plus their gen bits, plus the gen bits of the 
3 indices above/below.  For the data ring I stored the index of the 
comp_ring, its gen bit and the gen bits for the 3 indices above/below.  
These values were stored in binary form (no string conversion until 
later) each time we entered/exited the tx_complete function, and each 
time we exited the xmit func.  Once the issue was seen, these were 
formatted, and the values all looked correct for both the working and 
the non working case.

I suspect that the cause of this is some quirk of the way the driver 
code is interacting with the virtual NIC, but I have no access to the 
code for the virtual NIC, so am struggling to make any progress 
identifying the root cause.

Is this a known issue, and do you have any suggestions as to how best to 
proceed with this?

I have seen this with the following versions:

ESXi 5.5 and later (VM version 10)
ESXi 5.0 and later (VM version 8)
Linux 4.4 kernel
dpdk 2.2

thanks,
Paul