Hi, I recently upgraded our DPDK to 22.11.1 and observed that the guest vmxnet3 interface is not receiving any unicast packets. It's able to receive the broadcast packets though. We enable promiscuous and all-multicast on the port. rte_eth_promiscuous_enable(port); rte_eth_allmulticast_enable(port); This happens when the vswitch has only 2 ports attached. On further triaging, - I could find out that on the same setup, dpdk-20.11 works fine. - Upon comparing the working scenario and non-working scenario on esx host, I could notice that there is some difference in port configuration of vswitch. Below is the snapshot of working case, where the ethFRP:frame routing { requested:filter flags is 0x1d. port { port index:365 vnic index:0x00000003 portCfg:build_1_15_2_vm-v3--p1e3-u2e3 dvPortId: clientName:sjc-cms-esx15-csm02-pan1 clientType: 5 -> VMM Virtual NIC clientSubType: 9 -> Vmxnet3 Client world leader:2120158 flags: 0x40013 -> IN_USE ENABLED WORLD_ASSOC CONNECTED Impl customized blocked flags:0x00000000 Passthru status: 0x8 -> DISABLED_BY_HOST fixed Hw Id:00:50:56:b8:20:aa: ethFRP:frame routing { requested:filter { flags:0x0000001d unicastAddr:00:50:56:b8:20:aa: numMulticastAddresses:0 multicastAddresses: LADRF:[0]: 0x0 [1]: 0x0 } accepted:filter { flags:0x0000001d unicastAddr:00:50:56:b8:20:aa: numMulticastAddresses:0 multicastAddresses: LADRF:[0]: 0x0 [1]: 0x0 } } For the non-working case the flag is set to 0x09. port { port index:365 vnic index:0x00000003 portCfg:build_1_15_2_vm-v3--p1e3-u2e3 dvPortId: clientName:sjc-cms-esx15-csm02-pan1 clientType: 5 -> VMM Virtual NIC clientSubType: 9 -> Vmxnet3 Client world leader:2120158 flags: 0x40013 -> IN_USE ENABLED WORLD_ASSOC CONNECTED Impl customized blocked flags:0x00000000 Passthru status: 0x8 -> DISABLED_BY_HOST fixed Hw Id:00:50:56:b8:20:aa: ethFRP:frame routing { requested:filter { flags:0x00000009 unicastAddr:00:50:56:b8:20:aa: numMulticastAddresses:0 multicastAddresses: LADRF:[0]: 0x0 [1]: 0x0 } accepted:filter { flags:0x00000009 unicastAddr:00:50:56:b8:20:aa: numMulticastAddresses:0 multicastAddresses: LADRF:[0]: 0x0 [1]: 0x0 } } filter supported features: 0 -> NONE filter properties: 0 -> NONE rx mode: 0 -> INLINE tune mode: 0 -> default fastpath switch ID:0xffffffff fastpath port ID:0xffffffff } I think these are some of the flags set by vmxnet3 device and in dpdk-22.11, its not updating the flags. Could you please help on this?? I would try to create a simplest test setup with testpmd and update here.
There were not many changes in this driver between v20.11 and v22.11.1. If you have a simple reproducer, would you be able to do a bisect?
Can we get something from the flags above, what they stand for? Looks like the filter flags has definitely something to do with this. I will try to repro with simple testpmd next.
From the esx level debugging, I see the below output. The Unicast Packets received is 39 and it does not increment. [root@sjc-engtools-cms-esx-15:~] esxcli network port stats get -p 704643437 Packet statistics for port 704643437 Packets received: 155 Packets sent: 9066 Bytes received: 13614 Bytes sent: 907800 Broadcast packets received: 110 Broadcast packets sent: 2399 Multicast packets received: 6 Multicast packets sent: 4019 Unicast packets received: 39 Unicast packets sent: 2648 Receive packets dropped: 2 Transmit packets dropped: 6358
Sorry, I am not familiar with this ESX stuff. I'll Cc: the vmxnet3 maintainers on the dev@ ml as they are not registered in bugzilla.
Thanks David, please add the vmxnet3 experts. I did some more debugging. Looks like it has to do with either the mac address of the vswitch port or the flags mentioned above when the vswitch has only one port group with 2 ports. From my observation, the mac address configured on the vswitch port is not same as the initial mac address of the virtual network adapter attached to the VM. but the flag is set to 0x1d which is when the port receives traffic. The other observation I made on another setup is, if the mac address is same as the initial mac address of the virtual network adapter attached to the VM, then still the port is able to receive unicast traffic even if the flag is set to 0x09. Also I tried with disabling/enabling all the below flags both in vswitch and port group, but it did not help. Allow MAC address Change, Allow promiscuous mode and Allow forged transmits I would need some expert advice to understand the flags on the vswitch port and why the mac address configured is different than the initial mac address. All these are getting exposed with dpdk-22.11, any additional configuration required for vmxnet3 to work with 22.11.
Hi David/All Could you please help here? I am kind of blocked because of this issue. Any suggestion/ideas would be highly appreciated. Regards Amiya
I have neither the time nor a setup to work on your issue. I can only advise you to find a good/simple reproducer. And if you do a bisection pointing at the faulty commit that introduced your issue, someone may find a workaround/fix for you.
I asked on of our ESX datapath engineers to take a look.
Thanks Jochen. Any breakthrough?
If you can help me understand, I see the mac address of vswitch port ( which is same as the mac address of virtual network adapter) is different than the mac address configured on the interface in the guest. Is it something valid? If so, how the mac addresses are mapped? name id speed/duplex/state mac address -------------------------------------------------------------------------------- ethernet1/1 16 10000/full/up 12:ab:11:65:68:10 ethernet1/2 17 10000/full/up 12:ab:11:65:68:11 ethernet1/3 18 10000/full/up 12:ab:11:65:68:12 ethernet1/4 19 10000/full/up 12:ab:11:65:68:13 ethernet1/5 20 10000/full/up 12:ab:11:65:68:14 ethernet1/6 21 10000/full/up 12:ab:11:65:68:15 ethernet1/7 22 10000/full/up 12:ab:11:65:68:16 ethernet1/8 23 10000/full/up 12:ab:11:65:68:17 vlan 1 [n/a]/[n/a]/up 12:ab:11:65:68:01 loopback 3 [n/a]/[n/a]/up 12:ab:11:65:68:03 tunnel 4 [n/a]/[n/a]/up 12:ab:11:65:68:04 The above is the list of mac addresses of the interfaces in the guest which looks random. If you take ethernet1/3, the mac address configured shows 12:ab:11:65:68:12 whereas the actual mac address of the network adapter is 00:50:56:b8:20:aa. Please help clarify this. I think if the actual mac address of the virtual network adapter gets configured on the guest interface, the interface would receive the unicast packets.
(In reply to Amiya from comment #10) > If you can help me understand, I see the mac address of vswitch port ( which > is same as the mac address of virtual network adapter) is different than the > mac address configured on the interface in the guest. Is it something valid? > If so, how the mac addresses are mapped? > > name id speed/duplex/state mac address > > ----------------------------------------------------------------------------- > --- > ethernet1/1 16 10000/full/up > 12:ab:11:65:68:10 > ethernet1/2 17 10000/full/up > 12:ab:11:65:68:11 > ethernet1/3 18 10000/full/up > 12:ab:11:65:68:12 > ethernet1/4 19 10000/full/up > 12:ab:11:65:68:13 > ethernet1/5 20 10000/full/up > 12:ab:11:65:68:14 > ethernet1/6 21 10000/full/up > 12:ab:11:65:68:15 > ethernet1/7 22 10000/full/up > 12:ab:11:65:68:16 > ethernet1/8 23 10000/full/up > 12:ab:11:65:68:17 > vlan 1 [n/a]/[n/a]/up > 12:ab:11:65:68:01 > loopback 3 [n/a]/[n/a]/up > 12:ab:11:65:68:03 > tunnel 4 [n/a]/[n/a]/up > 12:ab:11:65:68:04 > > > The above is the list of mac addresses of the interfaces in the guest which > looks random. If you take ethernet1/3, the mac address configured shows > 12:ab:11:65:68:12 whereas the actual mac address of the network adapter is > 00:50:56:b8:20:aa. > > Please help clarify this. I think if the actual mac address of the virtual > network adapter gets configured on the guest interface, the interface would > receive the unicast packets. I had a quick try on latest ToT dpdk using testpmd application. testpmd> show port info 0 ********************* Infos for port 0 ********************* MAC address: 00:0C:29:87:07:FF Device name: 0000:13:00.0 Driver name: net_vmxnet3 Firmware-version: v8 Connect to socket: 0 memory allocation on the socket: 0 Link status: up Link speed: 10 Gbps Link duplex: full-duplex Autoneg status: Off MTU: 1500 Promiscuous mode: enabled Allmulticast mode: enabled ... ethFRP:frame routing { requested:filter { flags:0x0000001d unicastAddr:00:0c:29:87:07:ff: numMulticastAddresses:0 multicastAddresses: LADRF:[0]: 0x0 [1]: 0x0 } accepted:filter { flags:0x0000001d unicastAddr:00:0c:29:87:07:ff: numMulticastAddresses:0 multicastAddresses: LADRF:[0]: 0x0 [1]: 0x0 } } I see "1d" flags are configured, and traffic works fine for me. ]I just ran below command and saw it pick up correct mac address. ./app/dpdk-testpmd -l 0-15 -n 15 -- -i --portmask=0x1 --nb-cores=15 Please check your setup. Thanks, Ronak
Hi Ronak, Thanks for sharing the above info. In my case, I am getting flag as 0x9. port { port index:365 vnic index:0x00000003 portCfg:build_1_15_2_vm-v3--p1e3-u2e3 dvPortId: clientName:sjc-cms-esx15-csm02-pan1 clientType: 5 -> VMM Virtual NIC clientSubType: 9 -> Vmxnet3 Client world leader:2120158 flags: 0x40013 -> IN_USE ENABLED WORLD_ASSOC CONNECTED Impl customized blocked flags:0x00000000 Passthru status: 0x8 -> DISABLED_BY_HOST fixed Hw Id:00:50:56:b8:20:aa: ethFRP:frame routing { requested:filter { flags:0x00000009 unicastAddr:00:50:56:b8:20:aa: numMulticastAddresses:0 multicastAddresses: LADRF:[0]: 0x0 [1]: 0x0 } accepted:filter { flags:0x00000009 unicastAddr:00:50:56:b8:20:aa: numMulticastAddresses:0 multicastAddresses: LADRF:[0]: 0x0 [1]: 0x0 } } filter supported features: 0 -> NONE filter properties: 0 -> NONE rx mode: 0 -> INLINE tune mode: 0 -> default fastpath switch ID:0xffffffff fastpath port ID:0xffffffff } What could be the reason? any idea. with 20.11, its gets set to 0x1d. Could you also share the esx version and vswitch/port group configuration.
from the flags, looks like promiscuos is not getting enabled. Did you try with 22.11.1 dpdk?
(In reply to Amiya from comment #13) > from the flags, looks like promiscuos is not getting enabled. Did you try > with 22.11.1 dpdk? I tried on ESX8.0 and ToT dpdk. Can you check if promiscuous is being enabled by the test? The requested filter is also 0x09 which means the application itself did not enable it the mode.
sorry, what does ToT stand for? Application enables it, not sure why it does not take effect. We see this issue with ESX6.7/ ESX7.0.3.
(In reply to Amiya from comment #15) > sorry, what does ToT stand for? > Application enables it, not sure why it does not take effect. We see this > issue with ESX6.7/ ESX7.0.3. ToT - Top of Tree dpdk I don't think ESX is the issue as requestFilter itself is not setting the mode. Can you provide the output of below when you run testpmd? show port info <port_id>
We are definitely calling the dpdk api to enable promiscuous mode. But not sure underneath why it's not taking effect. Need couple of clarifications: 1. Do you think the mac addresses being different than that of the virtual network adapter is valid? 2. Our tests is on ESX6.7 and ESX7.0.3Uf and we are using dpdk-22.11.1. And the vswitch has a single port group with just 2 ports. Not sure if this particular configuration can contribute to it.
(In reply to Amiya from comment #17) > We are definitely calling the dpdk api to enable promiscuous mode. But not > sure underneath why it's not taking effect. Can you provide the output of the command I requested? show port info <port_id> You said in above comments, you are using testpmd right? > > Need couple of clarifications: > > 1. Do you think the mac addresses being different than that of the virtual > network adapter is valid? > I don't know have information on what your tests are, so not sure if I can comment on this. But, if same scenario works on 20.11, it should be no different on 22.11 > 2. Our tests is on ESX6.7 and ESX7.0.3Uf and we are using dpdk-22.11.1. And > the vswitch has a single port group with just 2 ports. Not sure if this > particular configuration can contribute to it. This should be fine.