[dpdk-dev] [PATCH 00/10] Infrastructure to detect iova mapping on the bus

[Santosh Shukla]

Q) Why do we need such infrastructure?

A) Some NPU hardware like OCTEONTX follows push model to get the packet
from the pktio device. Where packet allocation and freeing done
by the HW. Since HW can operate only on IOVA with help of SMMU/IOMMU,
when packet receives from the Ethernet device, it is the IOVA address
(which is PA in existing scheme).

Mapping IOVA as PA is expensive on those HW, where every packet
needs to be converted to VA from PA/IOVA.

This patchset proposes the method to autodetect the preferred
IOVA mode for a device. Summary of IOVA scheme:
- If all the devices are iommu capable and support IOMMU
  capable driver then selects IOVA_VA.
- If any of the devices are non-iommu then use default IOVA
  scheme ie. IOVA_PA.
- If no device found then IOVA scheme would be
  IOVA_DC (Don't care).

To achieve that, two global APIs introduced:
- rte_bus_get_iommu_class
- rte_pci_get_iommu_class

Return values for those APIs are:
enum rte_iova_mod {
        RTE_IOVA_DC, /* Don't care */
        RTE_IOVA_PA,
        RTE_IOVA_VA
}

Those are the bus policy for selecting IOVA mode. In case user
want to override bus IOVA mapping then added an EAL option
"--iova-mode=<string>". User to pass string format 'pa' --> IOVA_PA,
'va' --> IOVA_VA.

To support new eal option, adding global API:
- rte_eal_iova_mode

Patch Summary:
2) 1st - 2th patch: Adds infrastructure in linuxapp and bsdapp
layer.
1) 3rd patch: Introduces global bus api named rte_bus_get_iommu_class.
3) 4th patch: Add new eal option called --iova-mode=<mode-string>.
4) 5th - 6th patch: Logic to detect iova scheme.
5) 9th patch: Check IOVA mode before programing vfio dma_map.iova.
Default scheme is IOVA_PA.
6) 10th-12th patch: Check for IOVA_VA mode in below APIs
        - rte_mem_virt2phy
        - rte_mempool_virt2phy
        - rte_malloc_virt2phy
If set then return paddr=vaddr, else return value from default
implementation.

Test History:
- Tested for x86/XL710 40G NIC card for both modes (iova_va/pa).
- Tested for arm64/thunderx vNIC Integrated NIC for both modes
- Tested for arm64/Octeontx integrated NICs for only
  Iova_va mode(It supports only one mode.)
- Ran standalone tests like mempool_autotest, mbuf_autotest.
- Verified for Doxygen.

Work History:
Refer prev RFC proposal[1].

Noticed false positive checkpatch error:
- WARNING: quoted string split across lines
#60: FILE: lib/librte_eal/common/eal_common_bus.c:164:
+				RTE_LOG(INFO, EAL, "Bus (%s) iommu class of"
+					" devices not found.\n", bus->name);

- WARNING: LINUX_VERSION_CODE should be avoided, code should be for the version to which it is merged
#86: FILE: lib/librte_eal/linuxapp/eal/eal_vfio.c:822:
+#if LINUX_VERSION_CODE < KERNEL_VERSION(4, 2, 0)

Thanks.

[1] http://dpdk.org/dev/patchwork/patch/24549/


Santosh Shukla (10):
  bsdapp/eal_pci: get iommu class
  linuxapp/eal_pci: get iommu class
  bus: get iommu class
  eal: add eal option to configure iova mode
  linuxapp/eal: detect iova mode
  bsdapp/eal: detect iova mapping mode
  linuxapp/eal_vfio: honor iova mode before mapping
  linuxapp/eal_memory: honor iova mode in virt2phy
  mempool: honor iova mode in virt2phy
  eal/rte_malloc: honor iova mode in virt2phy

 lib/librte_eal/bsdapp/eal/eal.c                 | 24 ++++++++++++----
 lib/librte_eal/bsdapp/eal/eal_pci.c             | 10 +++++++
 lib/librte_eal/bsdapp/eal/rte_eal_version.map   |  9 ++++++
 lib/librte_eal/common/eal_common_bus.c          | 23 +++++++++++++++
 lib/librte_eal/common/eal_common_options.c      | 31 ++++++++++++++++++++
 lib/librte_eal/common/eal_common_pci.c          |  1 +
 lib/librte_eal/common/eal_internal_cfg.h        |  1 +
 lib/librte_eal/common/eal_options.h             |  2 ++
 lib/librte_eal/common/include/rte_bus.h         | 31 ++++++++++++++++++++
 lib/librte_eal/common/include/rte_eal.h         | 10 +++++++
 lib/librte_eal/common/include/rte_pci.h         | 11 +++++++
 lib/librte_eal/common/rte_malloc.c              |  9 +++++-
 lib/librte_eal/linuxapp/eal/eal.c               | 24 ++++++++++++----
 lib/librte_eal/linuxapp/eal/eal_memory.c        |  3 ++
 lib/librte_eal/linuxapp/eal/eal_pci.c           | 38 +++++++++++++++++++++++++
 lib/librte_eal/linuxapp/eal/eal_vfio.c          | 33 +++++++++++++++++++--
 lib/librte_eal/linuxapp/eal/eal_vfio.h          |  4 +++
 lib/librte_eal/linuxapp/eal/rte_eal_version.map |  9 ++++++
 lib/librte_mempool/rte_mempool.h                | 10 +++++--
 19 files changed, 266 insertions(+), 17 deletions(-)

-- 
2.11.0