[dpdk-dev] Poor SRIOV performance with ESXi Linux guest

[Ale Mansoor]

Thank you for your input, earlier on this same ESXi server another similar guest was able to run the Kernel mode ixgbevf driver and push several hundred kilo packets per second via the kernel mode driver. 
 
I am trying to get my hands on a second similar physical system for comparison, but likely the hardware and ESXi SRIOV VF mapping into the guest is not the issue.
 
Are there any Kernel options or DPDK options that may influence performance so significantly under ESXi ? 
 
I used the tools/setup.sh script under DPDK version 2.1 for building and setting up my DPDK environment.
 
My Kernel UIO settings are below:
----------------------------------------------
 
cat /boot/config-3.19.8-100.fc20.x86_64 |grep -i uio
CONFIG_HID_HUION=m
CONFIG_UIO=m
CONFIG_UIO_CIF=m
# CONFIG_UIO_PDRV_GENIRQ is not set
# CONFIG_UIO_DMEM_GENIRQ is not set
CONFIG_UIO_AEC=m
CONFIG_UIO_SERCOS3=m
CONFIG_UIO_PCI_GENERIC=m
# CONFIG_UIO_NETX is not set
# CONFIG_UIO_MF624 is not set
 
--

 
Regards
Ale
 

 
> Date: Wed, 2 Sep 2015 15:31:04 -0700
> From: stephen at networkplumber.org
> To: mansooraa at hotmail.com
> CC: dev at dpdk.org
> Subject: Re: [dpdk-dev] Poor SRIOV performance with ESXi Linux guest
> 
> On Wed, 2 Sep 2015 22:18:27 +0000
> Ale Mansoor <mansooraa at hotmail.com> wrote:
> 
> >  Getting less than 100 packets per second throughput between VF's under my Fedora FC20 VM running under ESXi 6.0 with DPDK l2fwd (Used as ./l2fwd -c 0xf -n 4 -- -p 0x3 -T 1)
> 
> That is many orders of magnitude less than expected.
> 
>  
> > Questions:
> > ---------------
> >  
> > Q1) Is DPDK + SRIOV under ESXi supposed to use the igb_uio driver or the vfio-pci driver inside Linux guest os ?
> 
> You have to use igb_uio, there is no emulated IOMMU in ESX
> 
> > Q2) What is the expected l2fwd performance when running DPDK under the Linux guest OS under ESXI with SRIOV ?
> 
> Depends on many things. With SRIOV you should reach 10Mpps or more.
> Did you try running Linux on baremetal on same hardware first?
> 
> > Q3) Any idea what may be preventing vfio-pci driver from binding to the VF's inside the guest instance ?
> 
> vfio-pci needs IOMMU which is not available in guest.
> 
> > Q4) Why is igb_uio performing so poorly ?
> 
> Don't blame igb_uio. It is probably something in system or vmware.
>