[PATCH v4 1/4] bus/cdx: introduce cdx bus

[Xia, Chenbo]

> -----Original Message-----
> From: Nipun Gupta <nipun.gupta at amd.com>
> Sent: Monday, May 8, 2023 7:18 PM
> To: dev at dpdk.org; thomas at monjalon.net; david.marchand at redhat.com
> Cc: ferruh.yigit at amd.com; harpreet.anand at amd.com; nikhil.agarwal at amd.com;
> Nipun Gupta <nipun.gupta at amd.com>
> Subject: [PATCH v4 1/4] bus/cdx: introduce cdx bus
> 
> CDX bus supports multiple type of devices, which can be
> exposed to user-space via vfio-cdx.
> 
> vfio-cdx provides the MMIO IO_MEMORY regions as well as the
> DMA interface for the device (IOMMU).
> 
> This support aims to enable the DPDK to support the cdx
> devices in user-space using VFIO interface.
> 
> Signed-off-by: Nipun Gupta <nipun.gupta at amd.com>
> ---
>  MAINTAINERS                            |   5 +
>  doc/guides/rel_notes/release_23_07.rst |   6 +
>  drivers/bus/cdx/bus_cdx_driver.h       | 201 ++++++++++
>  drivers/bus/cdx/cdx.c                  | 520 +++++++++++++++++++++++++
>  drivers/bus/cdx/cdx_logs.h             |  37 ++
>  drivers/bus/cdx/cdx_vfio.c             | 437 +++++++++++++++++++++
>  drivers/bus/cdx/meson.build            |  13 +
>  drivers/bus/cdx/private.h              |  49 +++
>  drivers/bus/cdx/version.map            |  11 +
>  drivers/bus/meson.build                |   1 +
>  10 files changed, 1280 insertions(+)
>  create mode 100644 drivers/bus/cdx/bus_cdx_driver.h
>  create mode 100644 drivers/bus/cdx/cdx.c
>  create mode 100644 drivers/bus/cdx/cdx_logs.h
>  create mode 100644 drivers/bus/cdx/cdx_vfio.c
>  create mode 100644 drivers/bus/cdx/meson.build
>  create mode 100644 drivers/bus/cdx/private.h
>  create mode 100644 drivers/bus/cdx/version.map
> 

...

> --- /dev/null
> +++ b/drivers/bus/cdx/cdx.c
> @@ -0,0 +1,520 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
> + */
> +
> +/*
> + * Architecture Overview
> + * =====================
> + * CDX is a Hardware Architecture designed for AMD FPGA devices. It
> + * consists of sophisticated mechanism for interaction between FPGA,
> + * Firmware and the APUs (Application CPUs).
> + *
> + * Firmware resides on RPU (Realtime CPUs) which interacts with
> + * the FPGA program manager and the APUs. The RPU provides memory-mapped
> + * interface (RPU if) which is used to communicate with APUs.
> + *
> + * The diagram below shows an overview of the CDX architecture:
> + *
> + *          +--------------------------------------+
> + *          |   DPDK                               |
> + *          |                    DPDK CDX drivers  |
> + *          |                             |        |
> + *          |                      DPDK CDX bus    |
> + *          |                             |        |
> + *          +-----------------------------|--------+
> + *                                        |
> + *          +-----------------------------|--------+
> + *          |    Application CPUs (APU)   |        |
> + *          |                             |        |
> + *          |                     VFIO CDX driver  |
> + *          |     Linux OS                |        |
> + *          |                      Linux CDX bus   |
> + *          |                             |        |
> + *          +-----------------------------|--------+
> + *                                        |
> + *                                        |
> + *          +------------------------| RPU if |----+
> + *          |                             |        |
> + *          |                             V        |
> + *          |          Realtime CPUs (RPU)         |
> + *          |                                      |
> + *          +--------------------------------------+
> + *                                |
> + *          +---------------------|----------------+
> + *          |  FPGA               |                |
> + *          |      +-----------------------+       |
> + *          |      |           |           |       |
> + *          | +-------+    +-------+   +-------+   |
> + *          | | dev 1 |    | dev 2 |   | dev 3 |   |
> + *          | +-------+    +-------+   +-------+   |
> + *          +--------------------------------------+
> + *
> + * The RPU firmware extracts the device information from the loaded FPGA
> + * image and implements a mechanism that allows the APU drivers to
> + * enumerate such devices (device personality and resource details) via
> + * a dedicated communication channel.

What is APU? CPU resources that application uses? Then why cpu resources that DPDK
Uses are not part of it?

I also notice that there are more and more vfio-XXX devices in DPDK now: pci/platform/cdx.
Some vfio related code are very similar. In the future we may need to think about make them
a common lib or something.

Thanks,
Chenbo

> + *
> + * VFIO CDX driver provides the CDX device resources like MMIO and
> interrupts
> + * to map to user-space. DPDK CDX bus uses sysfs interface and the vfio-
> cdx
> + * driver to discover and initialize the CDX devices for user-space
> + * applications.
> + */
> +