DPDK: Data Plane Development Kit

vmxnet3-usermap

DESCRIPTION

Here are two drivers for the third generation of paravirtualized NIC in VMware ESXi (vmxnet3): a Linux kernel driver and a Poll-Mode Driver (PMD) for DPDK. They work together.

The standard Linux kernel driver for this vNIC is simply called vmxnet3.ko. The "kmod" subdirectory provides a replacement called vmxnet3-usermap.ko. It essentially provides a new mode called usermap using the enable_shm=2 option.

The DPDK driver found in the "pmd" subdirectory is a shared object (librte_pmd_vmxnet3.so) that must be loaded as a DPDK extension. This driver gets hardware access through the usermap mode of the kernel driver.

HOW TO BUILD

There are two build dependencies: Linux kernel headers and DPDK >= 1.2.3r3.

The kernel driver is based on vmxnet3-8.6.10 from VMware ESXi-5.0.0u2. So the latest supported Linux version is 3.2.

DPDK must be built first in order to have all its header files in the same directory, that RTE_INCLUDE must point to.

Example:

dpdk$ make config T=x86_64-native-linuxapp-gcc
[...]
dpdk$ make
[...]
dpdk$ cd ../vmx3
vmx3$ make RTE_INCLUDE=../dpdk/build/include
[...]

Build options can be displayed using make help.

HOW TO RUN

Previous driver must be unloaded:

$ rmmod vmxnet3

The driver must be loaded in usermap mode:

$ insmod vmxnet3-usermap.ko enable_shm=2,2 num_rqs=1,1 num_rxds=512 num_txds=512

The above command will enable usermap-mode only for the first two vmxnet3 interfaces detected. Commas can be used to separate parameters values if several interfaces need to be managed from userland.

The following parameters take one value per interface:

Other parameters have a global effect:

Because the kernel module initializes interfaces, it's very important that DPDK parameters match kernel ones. It won't work otherwise.

For more information, use modinfo vmxnet3-usermap.ko.

In userland, librte_pmd_vmxnet3.so must be loaded by the DPDK using the -d option.

Another interesting option is --vmware-tsc-map. It allows VMware TSC mapping which provides an access to physical TSC. When this option is used, take care of setting this ESXi parameter:

monitor_control.pseudo_perfctr = TRUE

testpmd example for two ports, matching the insmod command above:

$ ../dpdk/build/app/testpmd -c 0xff -n 1 \
  -d librte_pmd_vmxnet3.so \
  -- \
  -i --rxq=1 --rxd=512 --txd=512
[...]
EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd
EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd
Interactive-mode selected
Initializing port 0... done:  Link Up - speed 10000 Mbps - full-duplex
Initializing port 1... done:  Link Up - speed 10000 Mbps - full-duplex
testpmd>

HOW TO RESTART APPLICATIONS

In order to relaunch a DPDK application using vmxnet3-usermap, the kernel driver must be reinitialized:

$ rmmod vmxnet3_usermap
$ insmod vmxnet3-usermap.ko enable_shm=2,2 num_rqs=1,1 num_rxds=512 num_txds=512

Then the DPDK application can be restarted.