[dpdk-dev] [PATCH 0/3] add vDPA sample driver

[Xiao Wang]

This patch set has dependency on the vhost lib patch:
http://dpdk.org/dev/patchwork/patch/34872/

This patch set shows a reference sample of making vDPA device driver 
, this driver uses a QEMU-emulated virtio-net PCI device as vDPA device,
and make it serve as a backend for a virtio-net pci device in nested VM.

The key driver ops implemented are:

* vdpa_virtio_eng_init
Mapping virtio pci device with VFIO into userspace, and read device
capability and intialize internal data.

* vdpa_virtio_eng_uninit
Release the mapped device.

* vdpa_virtio_info_query
Device capability reporting, e.g. queue number, features.

* vdpa_virtio_dev_config
With the guest virtio information provideed by vhost lib, this
function configures device and IOMMU to set up vhost datapath,
which includes: Rx/Tx vring, VFIO interrupt, kick relay.

* vdpa_virtio_dev_close
Unset the stuff that are configured previously by dev_conf.

This driver requires the virtio device supports VIRTIO_F_IOMMU_PLATFORM
, because the buffer address written in desc is IOVA.

Because vDPA driver needs to set up MSI-X vector to interrupt the guest,
only vfio-pci is supported currently.

Below are setup steps for your reference:

1. Make sure your kernnel vhost module and QEMU support vIOMMU.
   - OS: CentOS 7.4
   - QEMU: 2.10.1
   - Guest OS: CentOS 7.2
   - Nested VM OS: CentOS 7.2

2. enable VT-x feature for vCPU in VM.
   modprobe kvm_intel nested=1

3. Start a VM with a virtio-net-pci device.
   ./qemu-2.10.1/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -cpu host \
   <snip>
   -machine q35 \
   -device intel-iommu \
   -netdev tap,id=mytap,ifname=vdpa,vhostforce=on \
   -device virtio-net-pci,netdev=mytap,mac=00:aa:bb:cc:dd:ee,\
   disable-modern=off,disable-legacy=on,iommu_platform=on \

4. Bind VFIO-pci to virtio_net_pci device
   a) login to VM;
   b) modprobe vfio-pci
   c) rmmod vfio_iommu_type1
   d) modprobe vfio_iommu_type1 allow_unsafe_interrupts=1
   e) ./usertools/dpdk-devbind.py -b vfio-pci 00:03.0

5. Start vdpa sample
   ./examples/vdpa/build/vdpa -c 0x2 -n 4 --socket-mem 1024 --no-pci \
    --vdev "net_vdpa_virtio_pci0,bdf=0000:00:03.0" -- --bdf 0000:00:03.0 \
    --iface /tmp/vhost-user- --devcnt 1  --queue 1

6. Start nested VM
   ./qemu-2.10.1/x86_64-softmmu/qemu-system-x86_64 -cpu host -enable-kvm \
   <snip>
   -mem-prealloc \
   -chardev socket,id=char0,path=/tmp/vhost-user-0 \
   -netdev type=vhost-user,id=vdpa,chardev=char0,vhostforce \
   -device virtio-net-pci,netdev=vdpa,mac=00:aa:bb:cc:dd:ee \

7. Login the nested VM, and verify the virtio in nested VM can communicate
   with tap device on host.

Xiao Wang (3):
  bus/pci: expose API for vDPA
  net/vdpa_virtio_pci: introduce vdpa sample driver
  examples/vdpa: add a new sample for vdpa

 config/common_base                                 |    6 +
 config/common_linuxapp                             |    1 +
 drivers/bus/pci/Makefile                           |    1 +
 drivers/bus/pci/linux/pci.c                        |   10 +-
 drivers/bus/pci/linux/pci_init.h                   |   22 +-
 drivers/bus/pci/linux/pci_vfio.c                   |    5 +-
 drivers/bus/pci/rte_bus_pci_version.map            |   13 +
 drivers/net/Makefile                               |    1 +
 drivers/net/vdpa_virtio_pci/Makefile               |   31 +
 .../net/vdpa_virtio_pci/rte_eth_vdpa_virtio_pci.c  | 1527 ++++++++++++++++++++
 .../rte_vdpa_virtio_pci_version.map                |    4 +
 examples/vdpa/Makefile                             |   32 +
 examples/vdpa/main.c                               |  387 +++++
 mk/rte.app.mk                                      |    1 +
 14 files changed, 2030 insertions(+), 11 deletions(-)
 create mode 100644 drivers/net/vdpa_virtio_pci/Makefile
 create mode 100644 drivers/net/vdpa_virtio_pci/rte_eth_vdpa_virtio_pci.c
 create mode 100644 drivers/net/vdpa_virtio_pci/rte_vdpa_virtio_pci_version.map
 create mode 100644 examples/vdpa/Makefile
 create mode 100644 examples/vdpa/main.c

-- 
2.15.1