[dpdk-dev] [PATCH v3 00/10] rxq interrupt mode for virtio PMD

Jianfeng Tan jianfeng.tan at intel.com
Mon Jan 16 15:46:52 CET 2017

  - Update documents:
    * doc/guides/nics/features/virtio.ini
    * doc/guides/nics/features/virtio_vec.ini
    * doc/guides/nics/virtio.rst
  - Use hw->max_queue_pairs instead of dev->data->nb_rx_queues to
    allocate intr_vec array.
  - Fix v2 not working on legacy virtio devices by moving msix enabling
    before queue/irq binding.
  - Reword cover letter to give an overview of this series.
  - Remove wrapper to call vtpci->set_config_irq and vtpci->set_queue_irq.
  - Rebase on the new code, and fix a bug after changes by the commit
    bb30369dc10("eal: allow passing const interrupt handle"). Basically,
    it changes the way to get max interrupts. And we need to re-register
    callback to update intr_handle->max_intr.
  - In l3fwd-power ptype fix, use rte_eth_dev_get_supported_ptypes() to
    query if PMD provides needed ptypes.

  - Add PCI queue/irq config ops.
  - Move rxq interrupt settings before sending DRIVER OK.

Historically, virtio PMD can only be binded to igb_uio or
uio_pci_generic, and not for vfio-pci (iommu group cannot be created as
vIOMMU is not enabled in QEMU yet).  Besides, quote from
  "Per queue RX interrupt events are only allowed in VFIO
   which supports multiple MSI-X vectors."

Linux starts to support VFIO NO-IOMMU mode since 4.8.0. It cannot put
devices into groups for separation as normal VFIO does. So it does not
require QEMU to support vIOMMU. But it does inherit other benefits from
VFIO framework, like better interrupts supports (than UIO). It gives a
good chance to enable rxq interrupt for virtio PMD.

To implement it,
a. Firstly, we need to enable msix. This should be done before DRIVER_OK
setting and also before queue/irq binding in step b.
b. Bind queue/irq through portio (legacy devices) or mmio (modern devices).
   So far, we hard-code 1:1 queue/irq mapping (each rx queue has one
   exclusive interrupt), like this:
      vec 0 -> config irq
      vec 1 -> rxq0
      vec 2 -> rxq1
    which means, the "vectors" option of QEMU should be configured with
    a value >= N+1 (N is the number of the queue pairs).
c. To enable/disable interrupt notification, flags on virtqueues are used
   to control devices either sending interrupt or not.
d. Encap above behaviors into callbacks in ether_dev_ops, like
   rx_queue_intr_enable/rx_queue_intr_disable/rx_descriptor_done etc.

How to test:

Step 1, prepare a VM image with kernel version >= 4.8.0, and make sure
the kernel is compiled with CONFIG_VFIO_NOIOMMU=y.

Step 2, on the host, start a testpmd with a vhost port:
  $ testpmd -c 0x7 -m 1024 --vdev 'eth_vhost0,iface=/tmp/sock0,queues=2' \
	--no-pci -- -i --rxq=2 --txq=2 --nb-cores=2

Step 3, boot the VM:
  $ qemu ... -chardev socket,id=chr1,path=/tmp/sock0 \
	-netdev vhost-user,id=net1,chardev=chr1,vhostforce,queues=2 \
	-device virtio-net-pci,netdev=net1,mq=on,vectors=5 ...

Step 4, insert kernel modules
  $ modprobe vfio enable_unsafe_noiommu_mode=1
  $ modprobe vfio-pci

Step 5, start l3fwd-power in VM:
  $ l3fwd-power -c 0x3 -n 4 -- -p 1 -P --config="(0,0,1),(0,1,1)" \
						 --no-numa --parse-ptype

Step 6, send packets from testpmd on the host:
  $ start tx_first

Then l3fwd-power outputs:
  L3FWD_POWER: lcore 1 is waked up from rx interrupt on port 0 queue 0
  L3FWD_POWER: lcore 1 is waked up from rx interrupt on port 0 queue 1

Signed-off-by: Jianfeng Tan <jianfeng.tan at intel.com>

Jianfeng Tan (10):
  net/virtio: fix rewriting LSC flag
  net/virtio: clean up wrapper of set_config_irq
  net/virtio: add Rx descriptor check
  net/virtio: add PCI ops for queue/irq binding
  net/virtio: add Rx queue intr enable/disable functions
  net/virtio: setup rxq interrupts
  net/virtio: unbind intr/eventfd when stop device
  net/virtio: unmapping queue/irq when close device
  examples/l3fwd-power: add parse-ptype option
  examples/l3fwd-power: fix not stop and close device

 drivers/net/virtio/virtio_ethdev.c | 161 +++++++++++++++++++++++++++++++++++--
 drivers/net/virtio/virtio_ethdev.h |   3 +
 drivers/net/virtio/virtio_pci.c    |  31 +++++--
 drivers/net/virtio/virtio_pci.h    |   5 +-
 drivers/net/virtio/virtio_rxtx.c   |   9 +++
 drivers/net/virtio/virtqueue.c     |  11 ---
 drivers/net/virtio/virtqueue.h     |  16 +++-
 examples/l3fwd-power/main.c        | 120 ++++++++++++++++++++++++++-
 8 files changed, 325 insertions(+), 31 deletions(-)


More information about the dev mailing list