[dpdk-users] Cannot attach DPDK NIC port to OVS

[Vincenzo Maffione]

Hi,
  I eventually managed to get it working by using "uio_pci_generic" as a
DPDK-enabled driver for Intel 82599 ports. I had to disable IOMMU (e.g.
intel_iommu=off in the kernel boot parameters), otherwise uio_pci_generic
does not work. Is this documented somewhere? If not, I think it should be.

In any case, I confirm that DPDK with OVS 2.7.0 does not work when vfio-pci
is used as DPDK enabled driver; I keep getting "Error attaching device".
This is weird, because DPDK pktgen works fine with vfio-pci DPDK driver
(intel_iommu must be on in this case).

Thanks,
  Vincenzo

2017-04-20 18:21 GMT+02:00 Vincenzo Maffione <vincenzo.maffione at ing.unipi.it
>:

> vfio-pci is loaded and bound to the NIC, as shown by dpdk-devbind.
>
> $ sudo tools/dpdk-devbind.py --status
>
> Network devices using DPDK-compatible driver
> ============================================
> 0000:01:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=vfio-pci
> unused=ixgbe
>
> Network devices using kernel driver
> ===================================
> 0000:00:19.0 'Ethernet Connection (2) I218-V' if=eno1 drv=e1000e
> unused=vfio-pci
> 0000:01:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' if=enp1s0f1
> drv=ixgbe unused=vfio-pci
> 0000:06:00.0 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'
> if=enp6s0 drv=r8169 unused=vfio-pci
> 0000:07:00.0 '82574L Gigabit Network Connection' if=enp7s0 drv=e1000e
> unused=vfio-pci
>
> [...]
>
> And I get:
>
> Apr 20 18:19:22 arch-server ovs-vswitchd[650]: EAL: PCI device
> 0000:01:00.0 on NUMA socket -1
> Apr 20 18:19:22 arch-server ovs-vswitchd[650]: EAL:   probe driver:
> 8086:10fb net_ixgbe
> Apr 20 18:19:22 arch-server ovs-vswitchd[650]: EAL: Requested device
> 0000:01:00.0 cannot be used
> Apr 20 18:19:22 arch-server ovs-vswitchd[650]: EAL: Driver cannot attach
> the device (0000:01:00.0)
> Apr 20 18:19:22 arch-server ovs-vswitchd[650]: 2017-04-20T16:19:22Z|00027|netdev_dpdk|WARN|Error
> attaching device '0000:01:00.0' to DPDK
> Apr 20 18:19:22 arch-server ovs-vswitchd[650]: 2017-04-20T16:19:22Z|00028|netdev|WARN|dpdk-p1:
> could not set configuration (Invalid argumen
> Apr 20 18:19:22 arch-server ovs-vswitchd[650]: EAL: PCI device
> 0000:01:00.0 on NUMA socket -1
> Apr 20 18:19:22 arch-server ovs-vswitchd[650]: EAL:   probe driver:
> 8086:10fb net_ixgbe
> Apr 20 18:19:22 arch-server ovs-vswitchd[650]: EAL: Requested device
> 0000:01:00.0 cannot be used
> Apr 20 18:19:22 arch-server ovs-vswitchd[650]: EAL: Driver cannot attach
> the device (0000:01:00.0)
> Apr 20 18:19:22 arch-server ovs-vswitchd[650]: ovs|00027|netdev_dpdk|WARN|Error
> attaching device '0000:01:00.0' to DPDK
> Apr 20 18:19:22 arch-server ovs-vswitchd[650]:
> ovs|00028|netdev|WARN|dpdk-p1: could not set configuration (Invalid
> argument)
>
>
> Thanks,
>   Vincenzo
>
>
> 2017-04-20 18:13 GMT+02:00 Shivaram Mysore <shivaram.mysore at gmail.com>:
>
>> I have a feeling that vfio-pci is not installed.  Note that you will have
>> to edit /etc/modules to load on boot.
>>
>> You can also check by
>> # lsmod | grep vfio
>>
>> To load on demand - use
>> # modprobe vfio-pci
>>
>>
>>
>> On Thu, Apr 20, 2017 at 9:03 AM, Vincenzo Maffione <v.maffione at gmail.com>
>> wrote:
>>
>>> Hi,
>>>   I installed OVS 2.7.0 and use uio_pci_generic in place of vfio-pci,
>>> and the error disappeared.
>>>
>>> Thanks,
>>>   Vincenzo
>>>
>>> 2017-04-15 17:56 GMT+02:00 Shivaram Mysore <shivaram.mysore at gmail.com>:
>>>
>>>> You should look at emails from -(Hint:  search DPDK in subject)
>>>>
>>>> https://mail.openvswitch.org/pipermail/ovs-discuss/2017-Marc
>>>> h/thread.html
>>>> https://mail.openvswitch.org/pipermail/ovs-discuss/2017-Apri
>>>> l/thread.html
>>>> https://mail.openvswitch.org/pipermail/ovs-discuss/2017-Apri
>>>> l/044155.html
>>>>
>>>> If possible, I would recommend using Ubuntu 17.04 with OVS 2.7 with
>>>> DPDK packages from http://packages.wand.net.nz/ which can simplify
>>>> your deployment.
>>>>
>>>> /Shivaram
>>>>
>>>> On Fri, Apr 14, 2017 at 4:04 AM, Vincenzo Maffione <
>>>> vincenzo.maffione at ing.unipi.it> wrote:
>>>>
>>>>> Hi,
>>>>>   I would like to attach one of my ixgbe NIC ports to an OpenVSwitch
>>>>> instance, but I keep getting an unexpected error, see below.
>>>>>
>>>>> I've successfully bound both my ixgbe NIC ports to DPDK, using VFIO:
>>>>>
>>>>> # tools/dpdk-devbind.py --status
>>>>>
>>>>> Network devices using DPDK-compatible driver
>>>>> ============================================
>>>>> 0000:01:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection'
>>>>> drv=vfio-pci
>>>>> unused=ixgbe
>>>>> 0000:01:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection'
>>>>> drv=vfio-pci
>>>>> unused=ixgbe
>>>>>
>>>>> Network devices using kernel driver
>>>>> ===================================
>>>>> 0000:00:19.0 'Ethernet Connection (2) I218-V' if=eno1 drv=e1000e
>>>>> unused=vfio-pci
>>>>> 0000:06:00.0 'RTL8111/8168/8411 PCI Express Gigabit Ethernet
>>>>> Controller'
>>>>> if=enp6s0 drv=r8169 unused=vfio-pci
>>>>> 0000:07:00.0 '82574L Gigabit Network Connection' if=enp7s0 drv=e1000e
>>>>> unused=vfio-pci
>>>>>
>>>>> Other network devices
>>>>> =====================
>>>>> <none>
>>>>>
>>>>> Crypto devices using DPDK-compatible driver
>>>>> ===========================================
>>>>> <none>
>>>>>
>>>>> Crypto devices using kernel driver
>>>>> ==================================
>>>>> <none>
>>>>>
>>>>> Other crypto devices
>>>>> ====================
>>>>> <none>
>>>>>
>>>>>
>>>>> Then I created a bridge
>>>>>
>>>>> # ovs-vsctl add-br obr1 -- set bridge obr1 datapath_type=netdev
>>>>>
>>>>> But then when trying to add the dpdk NIC:
>>>>>
>>>>> # ovs-vsctl add-port obr1 dpdk-p0 -- set Interface dpdk-p0 type=dpdk
>>>>> options:dpdk-devargs=0000:01:00.0
>>>>>
>>>>> the command fails, and I get this in the log:
>>>>>
>>>>> ovs-vswitchd[353]: 2017-04-14T10:54:06Z|00026|bridge|WARN|could not
>>>>> open
>>>>> network device dpdk-p0 (No such device)
>>>>> ovs-vswitchd[353]: ovs|00026|bridge|WARN|could not open network device
>>>>> dpdk-p0 (No such device)
>>>>>
>>>>> Note that I've successfully managed to setup an OVS instance with two
>>>>> DPDK
>>>>> vhost-user ports (each one assigned to a QEMU VM): this is why I'm
>>>>> confident my OVS is built and configured correctly to work with DPDK.
>>>>> Also,
>>>>> DPDK by itself is working as expected, I can use dpdk pktgen.
>>>>> Moreover, OVS log shows that DPDK is recognized
>>>>>
>>>>> [...]
>>>>> ovs|00005|dpdk|INFO|DPDK Enabled, initializing
>>>>> [...]
>>>>>
>>>>> Note that I'm carefully following the documentation here
>>>>> http://docs.openvswitch.org/en/latest/intro/install/dpdk/ and here
>>>>> http://docs.openvswitch.org/en/latest/howto/dpdk/
>>>>>
>>>>> More info:
>>>>>
>>>>> DPDK version: 16.11
>>>>> OVS version: 2.6.1, configured with --with-dpdk
>>>>> Host: Archlinux
>>>>> /proc/cmdline: BOOT_IMAGE=../vmlinuz-linux-vfio
>>>>> root=UUID=1c777330-3f00-4ab8-8d5d-d4b2b5d860b0 rw intel_iommu=on
>>>>> hugepages=2048 pci=assign-busses pcie_acs_override=downstream
>>>>> initrd=../initramfs-linux-vfio.img
>>>>>
>>>>> Can anybody help? I'm at a loss here.
>>>>>
>>>>> Thanks,
>>>>>   Vincenzo
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Vincenzo Maffione
>>>
>>
>>
>