[dpdk-dev,v10,1/3] bus/ifpga: Add Intel FPGA BUS Library
Checks
Commit Message
From: Rosen Xu <rosen.xu@intel.com>
Defined FPGA-BUS for Acceleration Drivers of AFUs
1. FPGA PCI Scan (1st Scan) follows DPDK UIO/VFIO PCI Scan Process,
probe Intel FPGA Rawdev Driver, it will be covered in following patches.
2. AFU Scan(2nd Scan) bind DPDK driver to FPGA Partial-Bitstream.
This scan is trigged by hotplug of IFPGA Rawdev probe, in this scan
the AFUs will be created and their drivers are also probed.
This patch will introduce rte_afu_device which describe the AFU device
listed in the FPGA-BUS.
Signed-off-by: Rosen Xu <rosen.xu@intel.com>
Signed-off-by: Tianfei Zhang <tianfei.zhang@intel.com>
---
MAINTAINERS | 4 +
config/common_base | 5 +
doc/guides/rel_notes/release_18_05.rst | 5 +
drivers/bus/Makefile | 1 +
drivers/bus/ifpga/Makefile | 32 ++
drivers/bus/ifpga/ifpga_bus.c | 501 ++++++++++++++++++++++++++++
drivers/bus/ifpga/ifpga_common.c | 88 +++++
drivers/bus/ifpga/ifpga_common.h | 18 +
drivers/bus/ifpga/ifpga_logs.h | 31 ++
drivers/bus/ifpga/meson.build | 8 +
drivers/bus/ifpga/rte_bus_ifpga.h | 160 +++++++++
drivers/bus/ifpga/rte_bus_ifpga_version.map | 10 +
drivers/bus/meson.build | 2 +-
mk/rte.app.mk | 1 +
14 files changed, 865 insertions(+), 1 deletion(-)
create mode 100644 drivers/bus/ifpga/Makefile
create mode 100644 drivers/bus/ifpga/ifpga_bus.c
create mode 100644 drivers/bus/ifpga/ifpga_common.c
create mode 100644 drivers/bus/ifpga/ifpga_common.h
create mode 100644 drivers/bus/ifpga/ifpga_logs.h
create mode 100644 drivers/bus/ifpga/meson.build
create mode 100644 drivers/bus/ifpga/rte_bus_ifpga.h
create mode 100644 drivers/bus/ifpga/rte_bus_ifpga_version.map
Comments
Hi, Rosen
Few comments below.
Thanks
Jingjing
[......]
> +static struct rte_ifpga_device *
> +ifpga_find_ifpga_dev(const struct rte_rawdev *rdev)
> +{
> + struct rte_ifpga_device *ifpga_dev = NULL;
> +
> + TAILQ_FOREACH(ifpga_dev, &ifpga_device_list, next) {
> + if (rdev &&
rdev -> ifpage_dev ??
> + ifpga_dev->rdev &&
> + ifpga_dev->rdev == rdev)
> + return ifpga_dev;
> + }
> + return NULL;
> +}
> +
> +static struct rte_afu_device *
> +ifpga_find_afu_dev(const struct rte_ifpga_device *ifpga_dev,
> + const struct rte_afu_id *afu_id)
> +{
> + struct rte_afu_device *afu_dev = NULL;
> +
> + TAILQ_FOREACH(afu_dev, &ifpga_dev->afu_list, next) {
> + if (!ifpga_afu_id_cmp(&afu_dev->id, afu_id))
Add checking afu_dev?
[...]
> +static int
> +ifpga_parse(const char *name, void *addr)
> +{
> + int *out = addr;
> + struct rte_rawdev *rawdev = NULL;
> + char rawdev_name[RTE_RAWDEV_NAME_MAX_LEN];
> + char *c1 = NULL, *c2 = NULL;
According to coding style, we need to two lines for the definition like:
char *c1 = NULL;
char *c2 = NULL;
> + int port = IFPGA_BUS_DEV_PORT_MAX;
> + char str_port[8];
> + int str_port_len = 0;
> + int ret;
> +
> + memset(str_port, 0, 8);
> + c1 = strchr(name, '|');
> + if (c1 != NULL) {
> + str_port_len = c1-name;
According to coding style, spaces are required around opreations.
> + c2 = c1+1;
> + }
Hi Jingjing,
> -----Original Message-----
> From: Wu, Jingjing
> Sent: Thursday, May 10, 2018 16:44
> To: Xu, Rosen <rosen.xu@intel.com>; dev@dpdk.org; thomas@monjalon.net
> Cc: Xu, Rosen <rosen.xu@intel.com>; Zhang, Roy Fan
> <roy.fan.zhang@intel.com>; Doherty, Declan <declan.doherty@intel.com>;
> Richardson, Bruce <bruce.richardson@intel.com>; shreyansh.jain@nxp.com;
> Yigit, Ferruh <ferruh.yigit@intel.com>; Ananyev, Konstantin
> <konstantin.ananyev@intel.com>; Zhang, Tianfei <tianfei.zhang@intel.com>;
> Liu, Song <song.liu@intel.com>; Wu, Hao <hao.wu@intel.com>;
> gaetan.rivet@6wind.com
> Subject: RE: [dpdk-dev] [PATCH v10 1/3] bus/ifpga: Add Intel FPGA BUS
> Library
>
> Hi, Rosen
>
> Few comments below.
Thanks a lot Jingjing.
>
> Thanks
> Jingjing
>
> [......]
> > +static struct rte_ifpga_device *
> > +ifpga_find_ifpga_dev(const struct rte_rawdev *rdev) {
> > + struct rte_ifpga_device *ifpga_dev = NULL;
> > +
> > + TAILQ_FOREACH(ifpga_dev, &ifpga_device_list, next) {
> > + if (rdev &&
> rdev -> ifpage_dev ??
Rdev doesn't has this variable.
> > + ifpga_dev->rdev &&
> > + ifpga_dev->rdev == rdev)
> > + return ifpga_dev;
> > + }
> > + return NULL;
> > +}
> > +
> > +static struct rte_afu_device *
> > +ifpga_find_afu_dev(const struct rte_ifpga_device *ifpga_dev,
> > + const struct rte_afu_id *afu_id)
> > +{
> > + struct rte_afu_device *afu_dev = NULL;
> > +
> > + TAILQ_FOREACH(afu_dev, &ifpga_dev->afu_list, next) {
> > + if (!ifpga_afu_id_cmp(&afu_dev->id, afu_id))
> Add checking afu_dev?
Fixed.
> [...]
>
> > +static int
> > +ifpga_parse(const char *name, void *addr) {
> > + int *out = addr;
> > + struct rte_rawdev *rawdev = NULL;
> > + char rawdev_name[RTE_RAWDEV_NAME_MAX_LEN];
> > + char *c1 = NULL, *c2 = NULL;
> According to coding style, we need to two lines for the definition like:
> char *c1 = NULL;
> char *c2 = NULL;
Fixed
> > + int port = IFPGA_BUS_DEV_PORT_MAX;
> > + char str_port[8];
> > + int str_port_len = 0;
> > + int ret;
> > +
> > + memset(str_port, 0, 8);
> > + c1 = strchr(name, '|');
> > + if (c1 != NULL) {
> > + str_port_len = c1-name;
> According to coding style, spaces are required around opreations.
Fixed.
> > + c2 = c1+1;
> > + }
Hi Rosen:
> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Xu, Rosen
> Sent: Wednesday, May 9, 2018 3:43 PM
> To: dev@dpdk.org; thomas@monjalon.net
> Cc: Xu, Rosen <rosen.xu@intel.com>; Zhang, Roy Fan
> <roy.fan.zhang@intel.com>; Doherty, Declan <declan.doherty@intel.com>;
> Richardson, Bruce <bruce.richardson@intel.com>; shreyansh.jain@nxp.com;
> Yigit, Ferruh <ferruh.yigit@intel.com>; Ananyev, Konstantin
> <konstantin.ananyev@intel.com>; Zhang, Tianfei <tianfei.zhang@intel.com>;
> Liu, Song <song.liu@intel.com>; Wu, Hao <hao.wu@intel.com>;
> gaetan.rivet@6wind.com
> Subject: [dpdk-dev] [PATCH v10 1/3] bus/ifpga: Add Intel FPGA BUS Library
>
> From: Rosen Xu <rosen.xu@intel.com>
>
> Defined FPGA-BUS for Acceleration Drivers of AFUs
>
> 1. FPGA PCI Scan (1st Scan) follows DPDK UIO/VFIO PCI Scan Process, probe
> Intel FPGA Rawdev Driver, it will be covered in following patches.
>
> 2. AFU Scan(2nd Scan) bind DPDK driver to FPGA Partial-Bitstream.
> This scan is trigged by hotplug of IFPGA Rawdev probe, in this scan the AFUs
> will be created and their drivers are also probed.
>
> This patch will introduce rte_afu_device which describe the AFU device listed in
> the FPGA-BUS.
>
> Signed-off-by: Rosen Xu <rosen.xu@intel.com>
> Signed-off-by: Tianfei Zhang <tianfei.zhang@intel.com>
> ---
> MAINTAINERS | 4 +
> config/common_base | 5 +
> doc/guides/rel_notes/release_18_05.rst | 5 +
> drivers/bus/Makefile | 1 +
> drivers/bus/ifpga/Makefile | 32 ++
> drivers/bus/ifpga/ifpga_bus.c | 501
> ++++++++++++++++++++++++++++
> drivers/bus/ifpga/ifpga_common.c | 88 +++++
> drivers/bus/ifpga/ifpga_common.h | 18 +
> drivers/bus/ifpga/ifpga_logs.h | 31 ++
> drivers/bus/ifpga/meson.build | 8 +
> drivers/bus/ifpga/rte_bus_ifpga.h | 160 +++++++++
> drivers/bus/ifpga/rte_bus_ifpga_version.map | 10 +
> drivers/bus/meson.build | 2 +-
> mk/rte.app.mk | 1 +
> 14 files changed, 865 insertions(+), 1 deletion(-) create mode 100644
> drivers/bus/ifpga/Makefile create mode 100644
> drivers/bus/ifpga/ifpga_bus.c create mode 100644
> drivers/bus/ifpga/ifpga_common.c create mode 100644
> drivers/bus/ifpga/ifpga_common.h create mode 100644
> drivers/bus/ifpga/ifpga_logs.h create mode 100644
> drivers/bus/ifpga/meson.build create mode 100644
> drivers/bus/ifpga/rte_bus_ifpga.h create mode 100644
> drivers/bus/ifpga/rte_bus_ifpga_version.map
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 7105920..fa0c5b1 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -383,6 +383,10 @@ F: drivers/mempool/bucket/ Bus Drivers
> -----------
>
> +Intel FPGA buses
> +M: Rosen Xu <rosen.xu@intel.com>
> +F: drivers/bus/ifpga/
> +
> NXP buses
> M: Hemant Agrawal <hemant.agrawal@nxp.com>
> M: Shreyansh Jain <shreyansh.jain@nxp.com> diff --git
> a/config/common_base b/config/common_base index 0d181ac..1440316
> 100644
> --- a/config/common_base
> +++ b/config/common_base
> @@ -139,6 +139,11 @@
> CONFIG_RTE_ETHDEV_PROFILE_ITT_WASTED_RX_ITERATIONS=n
> CONFIG_RTE_ETHDEV_TX_PREPARE_NOOP=n
>
> #
> +# Compile the Intel FPGA bus
> +#
> +CONFIG_RTE_LIBRTE_IFPGA_BUS=y
> +
> +#
> # Compile PCI bus driver
> #
> CONFIG_RTE_LIBRTE_PCI_BUS=y
> diff --git a/doc/guides/rel_notes/release_18_05.rst
> b/doc/guides/rel_notes/release_18_05.rst
> index 7187348..265950a 100644
> --- a/doc/guides/rel_notes/release_18_05.rst
> +++ b/doc/guides/rel_notes/release_18_05.rst
> @@ -183,6 +183,11 @@ New Features
> stats/xstats on shared memory from secondary process, and also pdump
> packets on
> those virtual devices.
>
> +* **Added Ifpga Bus, a generic Intel FPGA Bus library.**
> +
> + The Ifpga Bus library provides support for integrating any Intel FPGA
> + device with the DPDK framework. It provides Intel FPGA Partial Bit
> + Stream AFU(Accelerated Function Unit) scan and drivers prove.
>
> API Changes
> -----------
> diff --git a/drivers/bus/Makefile b/drivers/bus/Makefile index
> c251b65..ef7f247 100644
> --- a/drivers/bus/Makefile
> +++ b/drivers/bus/Makefile
> @@ -7,6 +7,7 @@ DIRS-$(CONFIG_RTE_LIBRTE_DPAA_BUS) += dpaa ifeq
> ($(CONFIG_RTE_EAL_VFIO),y)
> DIRS-$(CONFIG_RTE_LIBRTE_FSLMC_BUS) += fslmc endif
> +DIRS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += ifpga
> DIRS-$(CONFIG_RTE_LIBRTE_PCI_BUS) += pci
> DIRS-$(CONFIG_RTE_LIBRTE_VDEV_BUS) += vdev
>
> diff --git a/drivers/bus/ifpga/Makefile b/drivers/bus/ifpga/Makefile new file
> mode 100644 index 0000000..3ff3bdb
> --- /dev/null
> +++ b/drivers/bus/ifpga/Makefile
> @@ -0,0 +1,32 @@
> +# SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2018 Intel
> +Corporation
> +
> +include $(RTE_SDK)/mk/rte.vars.mk
> +
> +#
> +# library name
> +#
> +LIB = librte_bus_ifpga.a
> +
> +CFLAGS += -DALLOW_EXPERIMENTAL_API
> +CFLAGS += -O3
> +CFLAGS += $(WERROR_FLAGS)
> +LDLIBS += -lrte_eal
> +LDLIBS += -lrte_rawdev
> +LDLIBS += -lrte_kvargs
> +
> +# versioning export map
> +EXPORT_MAP := rte_bus_ifpga_version.map
> +
> +# library version
> +LIBABIVER := 1
> +
> +SRCS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += ifpga_bus.c
> +SRCS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += ifpga_common.c
> +
> +#
> +# Export include files
> +#
> +SYMLINK-$(CONFIG_RTE_LIBRTE_IFPGA_BUS)-include += rte_bus_ifpga.h
> +
> +include $(RTE_SDK)/mk/rte.lib.mk
> diff --git a/drivers/bus/ifpga/ifpga_bus.c b/drivers/bus/ifpga/ifpga_bus.c new
> file mode 100644 index 0000000..e144c01
> --- /dev/null
> +++ b/drivers/bus/ifpga/ifpga_bus.c
> @@ -0,0 +1,501 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2010-2018 Intel Corporation */
> +
> +#include <string.h>
> +#include <inttypes.h>
> +#include <stdint.h>
> +#include <stdlib.h>
> +#include <stdio.h>
> +#include <sys/queue.h>
> +#include <sys/mman.h>
> +#include <sys/types.h>
> +#include <unistd.h>
> +#include <fcntl.h>
> +
> +#include <rte_errno.h>
> +#include <rte_bus.h>
> +#include <rte_per_lcore.h>
> +#include <rte_memory.h>
> +#include <rte_memzone.h>
> +#include <rte_eal.h>
> +#include <rte_common.h>
> +
> +#include <rte_devargs.h>
> +#include <rte_kvargs.h>
> +#include <rte_alarm.h>
> +
> +#include "rte_rawdev.h"
> +#include "rte_rawdev_pmd.h"
> +#include "rte_bus_ifpga.h"
> +#include "ifpga_logs.h"
> +#include "ifpga_common.h"
> +
> +int ifpga_bus_logtype;
> +
> +/* Forward declaration to access Intel FPGA bus
> + * on which iFPGA devices are connected */ static struct rte_bus
> +rte_ifpga_bus;
> +
> +/** Double linked list of IFPGA device. */
> +TAILQ_HEAD(ifpga_device_list, rte_ifpga_device);
> +
> +static struct ifpga_device_list ifpga_device_list =
> + TAILQ_HEAD_INITIALIZER(ifpga_device_list);
> +static struct afu_driver_list afu_driver_list =
> + TAILQ_HEAD_INITIALIZER(afu_driver_list);
> +
> +
> +/* register a ifpga bus based driver */ void
> +rte_ifpga_driver_register(struct rte_afu_driver *driver) {
> + RTE_VERIFY(driver);
> +
> + TAILQ_INSERT_TAIL(&afu_driver_list, driver, next); }
> +
> +/* un-register a fpga bus based driver */ void
> +rte_ifpga_driver_unregister(struct rte_afu_driver *driver) {
> + TAILQ_REMOVE(&afu_driver_list, driver, next); }
> +
> +static struct rte_ifpga_device *
> +ifpga_find_ifpga_dev(const struct rte_rawdev *rdev) {
> + struct rte_ifpga_device *ifpga_dev = NULL;
> +
> + TAILQ_FOREACH(ifpga_dev, &ifpga_device_list, next) {
> + if (rdev &&
> + ifpga_dev->rdev &&
> + ifpga_dev->rdev == rdev)
> + return ifpga_dev;
> + }
> + return NULL;
> +}
> +
> +static struct rte_afu_device *
> +ifpga_find_afu_dev(const struct rte_ifpga_device *ifpga_dev,
> + const struct rte_afu_id *afu_id)
> +{
> + struct rte_afu_device *afu_dev = NULL;
> +
> + TAILQ_FOREACH(afu_dev, &ifpga_dev->afu_list, next) {
> + if (!ifpga_afu_id_cmp(&afu_dev->id, afu_id))
> + return afu_dev;
> + }
> + return NULL;
> +}
> +
> +static const char * const valid_args[] = {
> +#define IFPGA_ARG_NAME "ifpga"
> + IFPGA_ARG_NAME,
> +#define IFPGA_ARG_PORT "port"
> + IFPGA_ARG_PORT,
> +#define IFPGA_AFU_BTS "afu_bts"
> + IFPGA_AFU_BTS,
> + NULL
> +};
> +
> +/*
> + * Scan the content of the FPGA bus, and the devices in the devices
> + * list
> + */
> +static struct rte_afu_device *
> +ifpga_scan_one(struct rte_devargs *devargs,
> + struct rte_ifpga_device *ifpga_dev)
usually ifpag_dev should be the first parameter here, and devargs follows
> +{
> + struct rte_kvargs *kvlist = NULL;
> + struct rte_rawdev *rawdev = NULL;
> + struct rte_afu_device *afu_dev = NULL;
> + struct rte_afu_pr_conf afu_pr_conf;
> + int ret = 0;
> + char *path = NULL;
> +
> + memset(&afu_pr_conf, 0, sizeof(struct rte_afu_pr_conf));
> +
> + kvlist = rte_kvargs_parse(devargs->args, valid_args);
> + if (!kvlist) {
> + IFPGA_BUS_ERR("error when parsing param");
> + goto end;
> + }
> +
> + if (rte_kvargs_count(kvlist, IFPGA_ARG_PORT) == 1) {
> + if (rte_kvargs_process(kvlist, IFPGA_ARG_PORT,
> + &rte_ifpga_get_integer32_arg, &afu_pr_conf.afu_id.port) < 0) {
> + IFPGA_BUS_ERR("error to parse %s",
> + IFPGA_ARG_PORT);
> + goto end;
> + }
> + } else {
> + IFPGA_BUS_ERR("arg %s is mandatory for ifpga bus",
> + IFPGA_ARG_PORT);
> + goto end;
> + }
> +
> + if (rte_kvargs_count(kvlist, IFPGA_AFU_BTS) == 1) {
> + if (rte_kvargs_process(kvlist, IFPGA_AFU_BTS,
> + &rte_ifpga_get_string_arg, &path) < 0) {
> + IFPGA_BUS_ERR("Failed to parse %s",
> + IFPGA_AFU_BTS);
> + goto end;
> + }
> + } else {
> + IFPGA_BUS_ERR("arg %s is mandatory for ifpga bus",
> + IFPGA_AFU_BTS);
> + goto end;
> + }
> +
> + afu_pr_conf.afu_id.uuid.uuid_low = 0;
> + afu_pr_conf.afu_id.uuid.uuid_high = 0;
> + afu_pr_conf.pr_enable = path?1:0;
> +
> + rawdev = ifpga_dev->rdev;
> + if (ifpga_find_afu_dev(ifpga_dev, &afu_pr_conf.afu_id))
> + goto end;
> +
> + afu_dev = calloc(1, sizeof(*afu_dev));
> + if (!afu_dev)
> + goto end;
> +
> + afu_dev->device.devargs = devargs;
> + afu_dev->device.numa_node = SOCKET_ID_ANY;
> + afu_dev->device.name = devargs->name;
> + afu_dev->rawdev = rawdev;
> + afu_dev->id.uuid.uuid_low = 0;
> + afu_dev->id.uuid.uuid_high = 0;
> + afu_dev->id.port = afu_pr_conf.afu_id.port;
> + afu_dev->ifpga_dev = ifpga_dev;
> +
> + if (rawdev->dev_ops && rawdev->dev_ops->dev_info_get)
> + rawdev->dev_ops->dev_info_get(rawdev, afu_dev);
> +
> + if (rawdev->dev_ops &&
> + rawdev->dev_ops->dev_start &&
> + rawdev->dev_ops->dev_start(rawdev))
> + goto free_dev;
> +
> + strncpy(afu_pr_conf.bs_path, path, sizeof(afu_pr_conf.bs_path));
> + if (rawdev->dev_ops->firmware_load &&
> + rawdev->dev_ops->firmware_load(rawdev,
> + &afu_pr_conf)){
> + IFPGA_BUS_ERR("firmware load error %d\n", ret);
> + goto free_dev;
> + }
> + afu_dev->id.uuid.uuid_low = afu_pr_conf.afu_id.uuid.uuid_low;
> + afu_dev->id.uuid.uuid_high = afu_pr_conf.afu_id.uuid.uuid_high;
> +
> + return afu_dev;
> +
> +free_dev:
> + free(afu_dev);
> +end:
> + if (kvlist)
> + rte_kvargs_free(kvlist);
> + if (path)
> + free(path);
> +
> + return NULL;
> +}
> +
> +/*
> + * Scan the content of the FPGA bus, and the devices in the devices
> + * list
> + */
> +static int
> +ifpga_scan(void)
> +{
> + struct rte_ifpga_device *ifpga_dev;
> + struct rte_devargs *devargs;
> + struct rte_kvargs *kvlist = NULL;
> + struct rte_rawdev *rawdev = NULL;
> + char *name = NULL;
> + char name1[RTE_RAWDEV_NAME_MAX_LEN];
> + struct rte_afu_device *afu_dev = NULL;
> +
> + /* for FPGA devices we scan the devargs_list populated via cmdline */
I didn't see "--ifpga" is supported by cmdline, either you need to add corresponding parser or correct the comment here.
> + RTE_EAL_DEVARGS_FOREACH("ifpga", devargs) {
IFPGA_BUS_NAME to replace "ifpga" here.
> + if (devargs->bus != &rte_ifpga_bus)
> + continue;
> +
> + kvlist = rte_kvargs_parse(devargs->args, valid_args);
> + if (!kvlist) {
> + IFPGA_BUS_ERR("error when parsing param");
> + goto end;
> + }
> +
> + if (rte_kvargs_count(kvlist, IFPGA_ARG_NAME) == 1) {
> + if (rte_kvargs_process(kvlist, IFPGA_ARG_NAME,
> + &rte_ifpga_get_string_arg, &name) < 0) {
> + IFPGA_BUS_ERR("error to parse %s",
> + IFPGA_ARG_NAME);
> + goto end;
> + }
> + } else {
> + IFPGA_BUS_ERR("arg %s is mandatory for ifpga bus",
> + IFPGA_ARG_NAME);
> + goto end;
> + }
> +
> + memset(name1, 0, sizeof(name1));
> + snprintf(name1, RTE_RAWDEV_NAME_MAX_LEN, "IFPGA:%s",
> name);
> +
> + rawdev = rte_rawdev_pmd_get_named_dev(name1);
> + if (!rawdev)
> + goto end;
> +
> + if (ifpga_find_ifpga_dev(rawdev))
> + continue;
> +
> + ifpga_dev = calloc(1, sizeof(*ifpga_dev));
> + if (!ifpga_dev)
> + goto end;
> +
> + ifpga_dev->rdev = rawdev;
> + TAILQ_INIT(&ifpga_dev->afu_list);
> +
> + TAILQ_INSERT_TAIL(&ifpga_device_list, ifpga_dev, next);
> + afu_dev = ifpga_scan_one(devargs, ifpga_dev);
> + if (afu_dev != NULL)
> + TAILQ_INSERT_TAIL(&ifpga_dev->afu_list, afu_dev, next);
I don't understand why we need afu_list?, seems we only add a new afu_dev into the list after create a new ifpga_dev,
Is there any another place that we add one to list to make it reasonable? it looks like either a 1:1 or 1:0 for ifpga_dev:afu_dev?
> + }
> +
> +end:
> + if (kvlist)
> + rte_kvargs_free(kvlist);
> + if (name)
> + free(name);
> +
> + return 0;
> +}
> +
> +/*
> + * Match the AFU Driver and AFU Device using the ID Table */ static
> +int rte_afu_match(const struct rte_afu_driver *afu_drv,
> + const struct rte_afu_device *afu_dev) {
> + const struct rte_afu_uuid *id_table;
> +
> + for (id_table = afu_drv->id_table;
> + ((id_table->uuid_low != 0) && (id_table->uuid_high != 0));
> + id_table++) {
> + /* check if device's identifiers match the driver's ones */
> + if ((id_table->uuid_low != afu_dev->id.uuid.uuid_low) ||
> + id_table->uuid_high !=
> + afu_dev->id.uuid.uuid_high)
> + continue;
> +
> + return 1;
> + }
> +
> + return 0;
> +}
> +
> +static int
> +ifpga_probe_one_driver(struct rte_afu_driver *drv,
> + struct rte_afu_device *afu_dev)
> +{
> + int ret;
> +
> + if (!rte_afu_match(drv, afu_dev))
> + /* Match of device and driver failed */
> + return 1;
> +
> + /* reference driver structure */
> + afu_dev->driver = drv;
> + afu_dev->device.driver = &drv->driver;
> +
> + /* call the driver probe() function */
> + ret = drv->probe(afu_dev);
> + if (ret) {
> + afu_dev->driver = NULL;
> + afu_dev->device.driver = NULL;
> + }
> +
> + return ret;
> +}
> +
> +static int
> +ifpga_probe_all_drivers(struct rte_afu_device *afu_dev) {
> + struct rte_afu_driver *drv = NULL;
> + int ret = 0;
> +
> + if (afu_dev == NULL)
> + return -1;
> +
> + /* Check if a driver is already loaded */
> + if (afu_dev->driver != NULL)
> + return 0;
> +
> + TAILQ_FOREACH(drv, &afu_driver_list, next) {
> + if (ifpga_probe_one_driver(drv, afu_dev)) {
> + ret = -1;
> + break;
> + }
> + }
> + return ret;
> +}
> +
> +/*
> + * Scan the content of the Intel FPGA bus, and call the probe()
> +function for
> + * all registered drivers that have a matching entry in its id_table
> + * for discovered devices.
> + */
> +static int
> +ifpga_probe(void)
> +{
> + struct rte_ifpga_device *ifpga_dev;
> + struct rte_afu_device *afu_dev = NULL;
> + int ret = 0;
> +
> + TAILQ_FOREACH(ifpga_dev, &ifpga_device_list, next) {
> + TAILQ_FOREACH(afu_dev, &ifpga_dev->afu_list, next) {
> +
> + if (afu_dev->device.driver)
> + continue;
> +
> + ret = ifpga_probe_all_drivers(afu_dev);
> + if (ret < 0)
> + IFPGA_BUS_ERR("failed to initialize %s device\n",
> + rte_ifpga_device_name(afu_dev));
> + }
> + }
> +
> + return ret;
> +}
> +
> +static int
> +ifpga_plug(struct rte_device *dev)
> +{
> + return ifpga_probe_all_drivers(RTE_DEV_TO_AFU(dev));
> +}
> +
> +static int
> +ifpga_remove_driver(struct rte_afu_device *afu_dev) {
> + const char *name;
> + const struct rte_afu_driver *driver;
> +
> + name = rte_ifpga_device_name(afu_dev);
> + if (!afu_dev->device.driver) {
> + IFPGA_BUS_DEBUG("no driver attach to device %s\n", name);
> + return 1;
> + }
> +
> + driver = RTE_DRV_TO_AFU_CONST(afu_dev->device.driver);
> + return driver->remove(afu_dev);
> +}
> +
> +static int
> +ifpga_unplug(struct rte_device *dev)
> +{
> + struct rte_ifpga_device *ifpga_dev = NULL;
> + struct rte_afu_device *afu_dev = NULL;
> + struct rte_devargs *devargs = NULL;
> + int ret;
> +
> + if (dev == NULL)
> + return -EINVAL;
> +
> + afu_dev = RTE_DEV_TO_AFU(dev);
> + if (!dev)
> + return -ENOENT;
> +
> + ifpga_dev = afu_dev->ifpga_dev;
> + devargs = dev->devargs;
> +
> + ret = ifpga_remove_driver(afu_dev);
So what is the device type that be plugged into the ifgpa bus?
ifpga_dev or afu_dev? If its afu_dev, why we need ifpga_dev?
If it is ifpga_dev, we need remove ifpga_dev from ifpga_device_list here.
> + if (ret)
> + return ret;
> +
> + TAILQ_REMOVE(&ifpga_dev->afu_list, afu_dev, next);
> +
> + rte_devargs_remove(devargs->bus->name, devargs->name);
> + free(afu_dev);
> + return 0;
> +
> +}
> +
> +static struct rte_device *
> +ifpga_find_device(const struct rte_device *start,
> + rte_dev_cmp_t cmp, const void *data)
> +{
> + struct rte_ifpga_device *ifpga_dev;
> + struct rte_afu_device *afu_dev;
> +
> + TAILQ_FOREACH(ifpga_dev, &ifpga_device_list, next) {
> + TAILQ_FOREACH(afu_dev, &ifpga_dev->afu_list, next) {
> + if (start && &afu_dev->device == start) {
> + start = NULL;
> + continue;
> + }
> + if (cmp(&afu_dev->device, data) == 0)
> + return &afu_dev->device;
> + }
> + }
> + return NULL;
> +}
> +static int
> +ifpga_parse(const char *name, void *addr) {
> + int *out = addr;
> + struct rte_rawdev *rawdev = NULL;
> + char rawdev_name[RTE_RAWDEV_NAME_MAX_LEN];
> + char *c1 = NULL, *c2 = NULL;
> + int port = IFPGA_BUS_DEV_PORT_MAX;
> + char str_port[8];
> + int str_port_len = 0;
> + int ret;
> +
> + memset(str_port, 0, 8);
> + c1 = strchr(name, '|');
> + if (c1 != NULL) {
> + str_port_len = c1-name;
> + c2 = c1+1;
> + }
> +
> + if (str_port_len < 8 &&
> + str_port_len > 0) {
> + memcpy(str_port, name, str_port_len);
> + ret = sscanf(str_port, "%d", &port);
> + if (ret == -1)
> + return 0;
> + }
> +
> + memset(rawdev_name, 0, sizeof(rawdev_name));
> + snprintf(rawdev_name, RTE_RAWDEV_NAME_MAX_LEN, "IFPGA:%s", c2);
> + rawdev = rte_rawdev_pmd_get_named_dev(rawdev_name);
> +
> + if ((port < IFPGA_BUS_DEV_PORT_MAX) &&
> + rawdev &&
> + (addr != NULL))
> + *out = port;
> +
> + if ((port < IFPGA_BUS_DEV_PORT_MAX) &&
> + rawdev)
> + return 0;
> + else
> + return 1;
> +}
> +
> +static struct rte_bus rte_ifpga_bus = {
> + .scan = ifpga_scan,
> + .probe = ifpga_probe,
> + .find_device = ifpga_find_device,
> + .plug = ifpga_plug,
> + .unplug = ifpga_unplug,
> + .parse = ifpga_parse,
> +};
> +
> +RTE_REGISTER_BUS(IFPGA_BUS_NAME, rte_ifpga_bus);
> +
> +RTE_INIT(ifpga_init_log)
> +{
> + ifpga_bus_logtype = rte_log_register("bus.ifpga");
> + if (ifpga_bus_logtype >= 0)
> + rte_log_set_level(ifpga_bus_logtype, RTE_LOG_NOTICE); }
> diff --git a/drivers/bus/ifpga/ifpga_common.c
> b/drivers/bus/ifpga/ifpga_common.c
> new file mode 100644
> index 0000000..78e2eae
> --- /dev/null
> +++ b/drivers/bus/ifpga/ifpga_common.c
> @@ -0,0 +1,88 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2010-2018 Intel Corporation */
> +
> +#include <string.h>
> +#include <inttypes.h>
> +#include <stdint.h>
> +#include <stdlib.h>
> +#include <stdio.h>
> +#include <sys/queue.h>
> +#include <sys/mman.h>
> +#include <sys/types.h>
> +#include <unistd.h>
> +#include <fcntl.h>
> +
> +#include <rte_errno.h>
> +#include <rte_bus.h>
> +#include <rte_per_lcore.h>
> +#include <rte_memory.h>
> +#include <rte_memzone.h>
> +#include <rte_eal.h>
> +#include <rte_common.h>
> +
> +#include <rte_devargs.h>
> +#include <rte_kvargs.h>
> +#include <rte_alarm.h>
> +
> +#include "rte_bus_ifpga.h"
> +#include "ifpga_logs.h"
> +#include "ifpga_common.h"
> +
> +int rte_ifpga_get_string_arg(const char *key __rte_unused,
> + const char *value, void *extra_args)
> +{
> + if (!value || !extra_args)
> + return -EINVAL;
> +
> + *(char **)extra_args = strdup(value);
> +
> + if (!*(char **)extra_args)
> + return -ENOMEM;
> +
> + return 0;
> +}
> +int rte_ifpga_get_integer32_arg(const char *key __rte_unused,
> + const char *value, void *extra_args)
> +{
> + if (!value || !extra_args)
> + return -EINVAL;
> +
> + *(int *)extra_args = strtoull(value, NULL, 0);
> +
> + return 0;
> +}
> +int ifpga_get_integer64_arg(const char *key __rte_unused,
> + const char *value, void *extra_args)
> +{
> + if (!value || !extra_args)
> + return -EINVAL;
> +
> + *(uint64_t *)extra_args = strtoull(value, NULL, 0);
> +
> + return 0;
> +}
> +int ifpga_get_unsigned_long(const char *str, int base) {
> + unsigned long num;
> + char *end = NULL;
> +
> + errno = 0;
> +
> + num = strtoul(str, &end, base);
> + if ((str[0] == '\0') || (end == NULL) || (*end != '\0') || (errno != 0))
> + return -1;
> +
> + return num;
> +}
> +
> +int ifpga_afu_id_cmp(const struct rte_afu_id *afu_id0,
> + const struct rte_afu_id *afu_id1)
> +{
> + if ((afu_id0->uuid.uuid_low == afu_id1->uuid.uuid_low) &&
> + (afu_id0->uuid.uuid_high == afu_id1->uuid.uuid_high) &&
> + (afu_id0->port == afu_id1->port)) {
> + return 0;
> + } else
> + return 1;
> +}
> diff --git a/drivers/bus/ifpga/ifpga_common.h
> b/drivers/bus/ifpga/ifpga_common.h
> new file mode 100644
> index 0000000..f9254b9
> --- /dev/null
> +++ b/drivers/bus/ifpga/ifpga_common.h
> @@ -0,0 +1,18 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2010-2018 Intel Corporation */
> +
> +#ifndef _IFPGA_COMMON_H_
> +#define _IFPGA_COMMON_H_
> +
> +int rte_ifpga_get_string_arg(const char *key __rte_unused,
> + const char *value, void *extra_args);
> +int rte_ifpga_get_integer32_arg(const char *key __rte_unused,
> + const char *value, void *extra_args);
> +int ifpga_get_integer64_arg(const char *key __rte_unused,
> + const char *value, void *extra_args);
> +int ifpga_get_unsigned_long(const char *str, int base); int
> +ifpga_afu_id_cmp(const struct rte_afu_id *afu_id0,
> + const struct rte_afu_id *afu_id1);
> +
> +#endif /* _IFPGA_COMMON_H_ */
> diff --git a/drivers/bus/ifpga/ifpga_logs.h b/drivers/bus/ifpga/ifpga_logs.h
> new file mode 100644 index 0000000..873e0a4
> --- /dev/null
> +++ b/drivers/bus/ifpga/ifpga_logs.h
> @@ -0,0 +1,31 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2010-2018 Intel Corporation */
> +
> +#ifndef _IFPGA_LOGS_H_
> +#define _IFPGA_LOGS_H_
> +
> +#include <rte_log.h>
> +
> +extern int ifpga_bus_logtype;
> +
> +#define IFPGA_LOG(level, fmt, args...) \
> + rte_log(RTE_LOG_ ## level, ifpga_bus_logtype, "%s(): " fmt "\n", \
> + __func__, ##args)
> +
> +#define IFPGA_BUS_LOG(level, fmt, args...) \
> + rte_log(RTE_LOG_ ## level, ifpga_bus_logtype, "%s(): " fmt "\n", \
> + __func__, ##args)
> +
> +#define IFPGA_BUS_FUNC_TRACE() IFPGA_BUS_LOG(DEBUG, ">>")
> +
> +#define IFPGA_BUS_DEBUG(fmt, args...) \
> + IFPGA_BUS_LOG(DEBUG, fmt, ## args)
> +#define IFPGA_BUS_INFO(fmt, args...) \
> + IFPGA_BUS_LOG(INFO, fmt, ## args)
> +#define IFPGA_BUS_ERR(fmt, args...) \
> + IFPGA_BUS_LOG(ERR, fmt, ## args)
> +#define IFPGA_BUS_WARN(fmt, args...) \
> + IFPGA_BUS_LOG(WARNING, fmt, ## args)
> +
> +#endif /* _IFPGA_BUS_LOGS_H_ */
> diff --git a/drivers/bus/ifpga/meson.build b/drivers/bus/ifpga/meson.build
> new file mode 100644 index 0000000..c9b08c8
> --- /dev/null
> +++ b/drivers/bus/ifpga/meson.build
> @@ -0,0 +1,8 @@
> +# SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2010-2018 Intel
> +Corporation
> +
> +deps += ['pci', 'kvargs', 'rawdev']
> +install_headers('rte_bus_ifpga.h')
> +sources = files('ifpga_common.c', 'ifpga_bus.c')
> +
> +allow_experimental_apis = true
> diff --git a/drivers/bus/ifpga/rte_bus_ifpga.h
> b/drivers/bus/ifpga/rte_bus_ifpga.h
> new file mode 100644
> index 0000000..5c559e1
> --- /dev/null
> +++ b/drivers/bus/ifpga/rte_bus_ifpga.h
> @@ -0,0 +1,160 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2010-2018 Intel Corporation */
> +
> +#ifndef _RTE_BUS_IFPGA_H_
> +#define _RTE_BUS_IFPGA_H_
> +
> +/**
> + * @file
> + *
> + * RTE Intel FPGA Bus Interface
> + */
> +
> +#ifdef __cplusplus
> +extern "C" {
> +#endif
> +
> +#include <rte_bus.h>
> +#include <rte_pci.h>
> +
> +/** Name of Intel FPGA Bus */
> +#define IFPGA_BUS_NAME ifpga
> +
> +/* Forward declarations */
> +struct rte_afu_device;
> +
> +/** List of Intel AFU devices */
> +TAILQ_HEAD(afu_device_list, rte_afu_device);
> +/** Double linked list of AFU device drivers. */
> +TAILQ_HEAD(afu_driver_list, rte_afu_driver);
> +
> +#define IFPGA_BUS_BITSTREAM_PATH_MAX_LEN 256
> +
> +struct rte_afu_uuid {
> + uint64_t uuid_low;
> + uint64_t uuid_high;
> +} __attribute__ ((packed));
> +
> +#define IFPGA_BUS_DEV_PORT_MAX 4
> +
> +/**
> + * A structure describing an ID for a AFU driver. Each driver provides
> +a
> + * table of these IDs for each device that it supports.
> + */
> +struct rte_afu_id {
> + struct rte_afu_uuid uuid;
> + int port; /**< port number */
> +} __attribute__ ((packed));
> +
> +/**
> + * A structure PR (Partial Reconfiguration) configuration AFU driver.
> + */
> +
> +struct rte_afu_pr_conf {
> + struct rte_afu_id afu_id;
> + int pr_enable;
> + char bs_path[IFPGA_BUS_BITSTREAM_PATH_MAX_LEN];
> +};
> +
> +#define AFU_PRI_STR_SIZE (PCI_PRI_STR_SIZE + 8)
> +
> +/**
> + * A structure describing a fpga device.
> + */
> +struct rte_ifpga_device {
> + TAILQ_ENTRY(rte_ifpga_device) next; /**< Next in device list. */
> + struct rte_rawdev *rdev;
> + struct afu_device_list afu_list; /**< List of AFU devices */ };
> +
> +/**
> + * A structure describing a AFU device.
> + */
> +struct rte_afu_device {
> + TAILQ_ENTRY(rte_afu_device) next; /**< Next in device list. */
> + struct rte_device device; /**< Inherit core device */
> + struct rte_rawdev *rawdev; /**< Point Rawdev */
> + struct rte_ifpga_device *ifpga_dev; /**< Point ifpga device */
> + struct rte_afu_id id; /**< AFU id within FPGA. */
> + uint32_t num_region; /**< number of regions found */
> + struct rte_mem_resource mem_resource[PCI_MAX_RESOURCE];
> + /**< AFU Memory Resource */
> + struct rte_intr_handle intr_handle; /**< Interrupt handle */
> + struct rte_afu_driver *driver; /**< Associated driver */
> + char path[IFPGA_BUS_BITSTREAM_PATH_MAX_LEN];
> +} __attribute__ ((packed));
> +
> +/**
> + * @internal
> + * Helper macro for drivers that need to convert to struct rte_afu_device.
> + */
> +#define RTE_DEV_TO_AFU(ptr) \
> + container_of(ptr, struct rte_afu_device, device)
> +
> +#define RTE_DRV_TO_AFU_CONST(ptr) \
> + container_of(ptr, const struct rte_afu_driver, driver)
> +
> +/**
> + * Initialisation function for the driver called during FPGA BUS probing.
typo: Initialization
> + */
> +typedef int (afu_probe_t)(struct rte_afu_device *);
> +
> +/**
> + * Uninitialisation function for the driver called during hotplugging.
typo: Unintialization
> + */
> +typedef int (afu_remove_t)(struct rte_afu_device *);
> +
> +/**
> + * A structure describing a AFU device.
> + */
> +struct rte_afu_driver {
> + TAILQ_ENTRY(rte_afu_driver) next; /**< Next afu driver. */
> + struct rte_driver driver; /**< Inherit core driver. */
> + afu_probe_t *probe; /**< Device Probe function.
> */
> + afu_remove_t *remove; /**< Device Remove
> function. */
> + const struct rte_afu_uuid *id_table; /**< AFU uuid within FPGA. */
> + uint32_t drv_flags; /**< Flags contolling handling of device. */
Typo Controlling
And what is "flags controlling handling of device?"
> +};
> +
> +static inline const char *
> +rte_ifpga_device_name(const struct rte_afu_device *afu) {
> + if (afu && afu->device.name)
> + return afu->device.name;
> + return NULL;
> +}
> +
> +/**
> + * Register a ifpga afu device driver.
> + *
> + * @param driver
> + * A pointer to a rte_afu_driver structure describing the driver
> + * to be registered.
> + */
> +void rte_ifpga_driver_register(struct rte_afu_driver *driver);
> +
> +/**
> + * Unregister a ifpga afu device driver.
> + *
> + * @param driver
> + * A pointer to a rte_afu_driver structure describing the driver
> + * to be unregistered.
> + */
> +void rte_ifpga_driver_unregister(struct rte_afu_driver *driver);
> +
> +#define RTE_PMD_REGISTER_AFU(nm, afudrv)\ RTE_INIT(afudrvinitfn_
> +##afudrv);\ static const char *afudrvinit_ ## nm ## _alias;\ static
> +void afudrvinitfn_ ##afudrv(void)\ {\
> + (afudrv).driver.name = RTE_STR(nm);\
> + (afudrv).driver.alias = afudrvinit_ ## nm ## _alias;\
> + rte_ifpga_driver_register(&afudrv);\
> +} \
> +RTE_PMD_EXPORT_NAME(nm, __COUNTER__)
> +
> +#define RTE_PMD_REGISTER_AFU_ALIAS(nm, alias)\ static const char
> +*afudrvinit_ ## nm ## _alias = RTE_STR(alias)
> +
> +#endif /* _RTE_BUS_IFPGA_H_ */
> diff --git a/drivers/bus/ifpga/rte_bus_ifpga_version.map
> b/drivers/bus/ifpga/rte_bus_ifpga_version.map
> new file mode 100644
> index 0000000..a027979
> --- /dev/null
> +++ b/drivers/bus/ifpga/rte_bus_ifpga_version.map
> @@ -0,0 +1,10 @@
> +DPDK_18.05 {
> + global:
> +
> + rte_ifpga_get_integer32_arg;
> + rte_ifpga_get_string_arg;
> + rte_ifpga_driver_register;
> + rte_ifpga_driver_unregister;
> +
> + local: *;
> +};
> diff --git a/drivers/bus/meson.build b/drivers/bus/meson.build index
> 58dfbe2..52c755d 100644
> --- a/drivers/bus/meson.build
> +++ b/drivers/bus/meson.build
> @@ -1,7 +1,7 @@
> # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2017 Intel
> Corporation
>
> -drivers = ['dpaa', 'fslmc', 'pci', 'vdev']
> +drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'vdev']
> std_deps = ['eal']
> config_flag_fmt = 'RTE_LIBRTE_@0@_BUS'
> driver_name_fmt = 'rte_bus_@0@'
> diff --git a/mk/rte.app.mk b/mk/rte.app.mk index 26f3563..3861e1a 100644
> --- a/mk/rte.app.mk
> +++ b/mk/rte.app.mk
> @@ -255,6 +255,7 @@ ifeq
> ($(CONFIG_RTE_EAL_VFIO)$(CONFIG_RTE_LIBRTE_FSLMC_BUS),yy)
> _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_CMDIF_RAWDEV) +=
> -lrte_pmd_dpaa2_cmdif
> _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV) +=
> -lrte_pmd_dpaa2_qdma endif # CONFIG_RTE_LIBRTE_FSLMC_BUS
> +_LDLIBS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += -lrte_bus_ifpga
> endif # CONFIG_RTE_LIBRTE_RAWDEV
>
>
> --
> 1.8.3.1
Regards
Qi
Hi Qi,
> -----Original Message-----
> From: Zhang, Qi Z
> Sent: Thursday, May 10, 2018 20:27
> To: Xu, Rosen <rosen.xu@intel.com>; dev@dpdk.org; thomas@monjalon.net
> Cc: Xu, Rosen <rosen.xu@intel.com>; Zhang, Roy Fan
> <roy.fan.zhang@intel.com>; Doherty, Declan <declan.doherty@intel.com>;
> Richardson, Bruce <bruce.richardson@intel.com>; shreyansh.jain@nxp.com;
> Yigit, Ferruh <ferruh.yigit@intel.com>; Ananyev, Konstantin
> <konstantin.ananyev@intel.com>; Zhang, Tianfei <tianfei.zhang@intel.com>;
> Liu, Song <song.liu@intel.com>; Wu, Hao <hao.wu@intel.com>;
> gaetan.rivet@6wind.com
> Subject: RE: [dpdk-dev] [PATCH v10 1/3] bus/ifpga: Add Intel FPGA BUS
> Library
>
> Hi Rosen:
>
> > -----Original Message-----
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Xu, Rosen
> > Sent: Wednesday, May 9, 2018 3:43 PM
> > To: dev@dpdk.org; thomas@monjalon.net
> > Cc: Xu, Rosen <rosen.xu@intel.com>; Zhang, Roy Fan
> > <roy.fan.zhang@intel.com>; Doherty, Declan <declan.doherty@intel.com>;
> > Richardson, Bruce <bruce.richardson@intel.com>;
> > shreyansh.jain@nxp.com; Yigit, Ferruh <ferruh.yigit@intel.com>;
> > Ananyev, Konstantin <konstantin.ananyev@intel.com>; Zhang, Tianfei
> > <tianfei.zhang@intel.com>; Liu, Song <song.liu@intel.com>; Wu, Hao
> > <hao.wu@intel.com>; gaetan.rivet@6wind.com
> > Subject: [dpdk-dev] [PATCH v10 1/3] bus/ifpga: Add Intel FPGA BUS
> > Library
> >
> > From: Rosen Xu <rosen.xu@intel.com>
> >
> > Defined FPGA-BUS for Acceleration Drivers of AFUs
> >
> > 1. FPGA PCI Scan (1st Scan) follows DPDK UIO/VFIO PCI Scan Process,
> > probe Intel FPGA Rawdev Driver, it will be covered in following patches.
> >
> > 2. AFU Scan(2nd Scan) bind DPDK driver to FPGA Partial-Bitstream.
> > This scan is trigged by hotplug of IFPGA Rawdev probe, in this scan
> > the AFUs will be created and their drivers are also probed.
> >
> > This patch will introduce rte_afu_device which describe the AFU device
> > listed in the FPGA-BUS.
> >
> > Signed-off-by: Rosen Xu <rosen.xu@intel.com>
> > Signed-off-by: Tianfei Zhang <tianfei.zhang@intel.com>
> > ---
> > MAINTAINERS | 4 +
> > config/common_base | 5 +
> > doc/guides/rel_notes/release_18_05.rst | 5 +
> > drivers/bus/Makefile | 1 +
> > drivers/bus/ifpga/Makefile | 32 ++
> > drivers/bus/ifpga/ifpga_bus.c | 501
> > ++++++++++++++++++++++++++++
> > drivers/bus/ifpga/ifpga_common.c | 88 +++++
> > drivers/bus/ifpga/ifpga_common.h | 18 +
> > drivers/bus/ifpga/ifpga_logs.h | 31 ++
> > drivers/bus/ifpga/meson.build | 8 +
> > drivers/bus/ifpga/rte_bus_ifpga.h | 160 +++++++++
> > drivers/bus/ifpga/rte_bus_ifpga_version.map | 10 +
> > drivers/bus/meson.build | 2 +-
> > mk/rte.app.mk | 1 +
> > 14 files changed, 865 insertions(+), 1 deletion(-) create mode
> > 100644 drivers/bus/ifpga/Makefile create mode 100644
> > drivers/bus/ifpga/ifpga_bus.c create mode 100644
> > drivers/bus/ifpga/ifpga_common.c create mode 100644
> > drivers/bus/ifpga/ifpga_common.h create mode 100644
> > drivers/bus/ifpga/ifpga_logs.h create mode 100644
> > drivers/bus/ifpga/meson.build create mode 100644
> > drivers/bus/ifpga/rte_bus_ifpga.h create mode 100644
> > drivers/bus/ifpga/rte_bus_ifpga_version.map
> >
> > diff --git a/MAINTAINERS b/MAINTAINERS index 7105920..fa0c5b1 100644
> > --- a/MAINTAINERS
> > +++ b/MAINTAINERS
> > @@ -383,6 +383,10 @@ F: drivers/mempool/bucket/ Bus Drivers
> > -----------
> >
> > +Intel FPGA buses
> > +M: Rosen Xu <rosen.xu@intel.com>
> > +F: drivers/bus/ifpga/
> > +
> > NXP buses
> > M: Hemant Agrawal <hemant.agrawal@nxp.com>
> > M: Shreyansh Jain <shreyansh.jain@nxp.com> diff --git
> > a/config/common_base b/config/common_base index 0d181ac..1440316
> > 100644
> > --- a/config/common_base
> > +++ b/config/common_base
> > @@ -139,6 +139,11 @@
> > CONFIG_RTE_ETHDEV_PROFILE_ITT_WASTED_RX_ITERATIONS=n
> > CONFIG_RTE_ETHDEV_TX_PREPARE_NOOP=n
> >
> > #
> > +# Compile the Intel FPGA bus
> > +#
> > +CONFIG_RTE_LIBRTE_IFPGA_BUS=y
> > +
> > +#
> > # Compile PCI bus driver
> > #
> > CONFIG_RTE_LIBRTE_PCI_BUS=y
> > diff --git a/doc/guides/rel_notes/release_18_05.rst
> > b/doc/guides/rel_notes/release_18_05.rst
> > index 7187348..265950a 100644
> > --- a/doc/guides/rel_notes/release_18_05.rst
> > +++ b/doc/guides/rel_notes/release_18_05.rst
> > @@ -183,6 +183,11 @@ New Features
> > stats/xstats on shared memory from secondary process, and also
> > pdump packets on
> > those virtual devices.
> >
> > +* **Added Ifpga Bus, a generic Intel FPGA Bus library.**
> > +
> > + The Ifpga Bus library provides support for integrating any Intel
> > + FPGA device with the DPDK framework. It provides Intel FPGA Partial
> > + Bit Stream AFU(Accelerated Function Unit) scan and drivers prove.
> >
> > API Changes
> > -----------
> > diff --git a/drivers/bus/Makefile b/drivers/bus/Makefile index
> > c251b65..ef7f247 100644
> > --- a/drivers/bus/Makefile
> > +++ b/drivers/bus/Makefile
> > @@ -7,6 +7,7 @@ DIRS-$(CONFIG_RTE_LIBRTE_DPAA_BUS) += dpaa ifeq
> > ($(CONFIG_RTE_EAL_VFIO),y)
> > DIRS-$(CONFIG_RTE_LIBRTE_FSLMC_BUS) += fslmc endif
> > +DIRS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += ifpga
> > DIRS-$(CONFIG_RTE_LIBRTE_PCI_BUS) += pci
> > DIRS-$(CONFIG_RTE_LIBRTE_VDEV_BUS) += vdev
> >
> > diff --git a/drivers/bus/ifpga/Makefile b/drivers/bus/ifpga/Makefile
> > new file mode 100644 index 0000000..3ff3bdb
> > --- /dev/null
> > +++ b/drivers/bus/ifpga/Makefile
> > @@ -0,0 +1,32 @@
> > +# SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2018 Intel
> > +Corporation
> > +
> > +include $(RTE_SDK)/mk/rte.vars.mk
> > +
> > +#
> > +# library name
> > +#
> > +LIB = librte_bus_ifpga.a
> > +
> > +CFLAGS += -DALLOW_EXPERIMENTAL_API
> > +CFLAGS += -O3
> > +CFLAGS += $(WERROR_FLAGS)
> > +LDLIBS += -lrte_eal
> > +LDLIBS += -lrte_rawdev
> > +LDLIBS += -lrte_kvargs
> > +
> > +# versioning export map
> > +EXPORT_MAP := rte_bus_ifpga_version.map
> > +
> > +# library version
> > +LIBABIVER := 1
> > +
> > +SRCS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += ifpga_bus.c
> > +SRCS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += ifpga_common.c
> > +
> > +#
> > +# Export include files
> > +#
> > +SYMLINK-$(CONFIG_RTE_LIBRTE_IFPGA_BUS)-include += rte_bus_ifpga.h
> > +
> > +include $(RTE_SDK)/mk/rte.lib.mk
> > diff --git a/drivers/bus/ifpga/ifpga_bus.c
> > b/drivers/bus/ifpga/ifpga_bus.c new file mode 100644 index
> > 0000000..e144c01
> > --- /dev/null
> > +++ b/drivers/bus/ifpga/ifpga_bus.c
> > @@ -0,0 +1,501 @@
> > +/* SPDX-License-Identifier: BSD-3-Clause
> > + * Copyright(c) 2010-2018 Intel Corporation */
> > +
> > +#include <string.h>
> > +#include <inttypes.h>
> > +#include <stdint.h>
> > +#include <stdlib.h>
> > +#include <stdio.h>
> > +#include <sys/queue.h>
> > +#include <sys/mman.h>
> > +#include <sys/types.h>
> > +#include <unistd.h>
> > +#include <fcntl.h>
> > +
> > +#include <rte_errno.h>
> > +#include <rte_bus.h>
> > +#include <rte_per_lcore.h>
> > +#include <rte_memory.h>
> > +#include <rte_memzone.h>
> > +#include <rte_eal.h>
> > +#include <rte_common.h>
> > +
> > +#include <rte_devargs.h>
> > +#include <rte_kvargs.h>
> > +#include <rte_alarm.h>
> > +
> > +#include "rte_rawdev.h"
> > +#include "rte_rawdev_pmd.h"
> > +#include "rte_bus_ifpga.h"
> > +#include "ifpga_logs.h"
> > +#include "ifpga_common.h"
> > +
> > +int ifpga_bus_logtype;
> > +
> > +/* Forward declaration to access Intel FPGA bus
> > + * on which iFPGA devices are connected */ static struct rte_bus
> > +rte_ifpga_bus;
> > +
> > +/** Double linked list of IFPGA device. */
> > +TAILQ_HEAD(ifpga_device_list, rte_ifpga_device);
> > +
> > +static struct ifpga_device_list ifpga_device_list =
> > + TAILQ_HEAD_INITIALIZER(ifpga_device_list);
> > +static struct afu_driver_list afu_driver_list =
> > + TAILQ_HEAD_INITIALIZER(afu_driver_list);
> > +
> > +
> > +/* register a ifpga bus based driver */ void
> > +rte_ifpga_driver_register(struct rte_afu_driver *driver) {
> > + RTE_VERIFY(driver);
> > +
> > + TAILQ_INSERT_TAIL(&afu_driver_list, driver, next); }
> > +
> > +/* un-register a fpga bus based driver */ void
> > +rte_ifpga_driver_unregister(struct rte_afu_driver *driver) {
> > + TAILQ_REMOVE(&afu_driver_list, driver, next); }
> > +
> > +static struct rte_ifpga_device *
> > +ifpga_find_ifpga_dev(const struct rte_rawdev *rdev) {
> > + struct rte_ifpga_device *ifpga_dev = NULL;
> > +
> > + TAILQ_FOREACH(ifpga_dev, &ifpga_device_list, next) {
> > + if (rdev &&
> > + ifpga_dev->rdev &&
> > + ifpga_dev->rdev == rdev)
> > + return ifpga_dev;
> > + }
> > + return NULL;
> > +}
> > +
> > +static struct rte_afu_device *
> > +ifpga_find_afu_dev(const struct rte_ifpga_device *ifpga_dev,
> > + const struct rte_afu_id *afu_id)
> > +{
> > + struct rte_afu_device *afu_dev = NULL;
> > +
> > + TAILQ_FOREACH(afu_dev, &ifpga_dev->afu_list, next) {
> > + if (!ifpga_afu_id_cmp(&afu_dev->id, afu_id))
> > + return afu_dev;
> > + }
> > + return NULL;
> > +}
> > +
> > +static const char * const valid_args[] = {
> > +#define IFPGA_ARG_NAME "ifpga"
> > + IFPGA_ARG_NAME,
> > +#define IFPGA_ARG_PORT "port"
> > + IFPGA_ARG_PORT,
> > +#define IFPGA_AFU_BTS "afu_bts"
> > + IFPGA_AFU_BTS,
> > + NULL
> > +};
> > +
> > +/*
> > + * Scan the content of the FPGA bus, and the devices in the devices
> > + * list
> > + */
> > +static struct rte_afu_device *
> > +ifpga_scan_one(struct rte_devargs *devargs,
> > + struct rte_ifpga_device *ifpga_dev)
>
> usually ifpag_dev should be the first parameter here, and devargs follows
Fixed.
> > +{
> > + struct rte_kvargs *kvlist = NULL;
> > + struct rte_rawdev *rawdev = NULL;
> > + struct rte_afu_device *afu_dev = NULL;
> > + struct rte_afu_pr_conf afu_pr_conf;
> > + int ret = 0;
> > + char *path = NULL;
> > +
> > + memset(&afu_pr_conf, 0, sizeof(struct rte_afu_pr_conf));
> > +
> > + kvlist = rte_kvargs_parse(devargs->args, valid_args);
> > + if (!kvlist) {
> > + IFPGA_BUS_ERR("error when parsing param");
> > + goto end;
> > + }
> > +
> > + if (rte_kvargs_count(kvlist, IFPGA_ARG_PORT) == 1) {
> > + if (rte_kvargs_process(kvlist, IFPGA_ARG_PORT,
> > + &rte_ifpga_get_integer32_arg, &afu_pr_conf.afu_id.port) < 0)
> {
> > + IFPGA_BUS_ERR("error to parse %s",
> > + IFPGA_ARG_PORT);
> > + goto end;
> > + }
> > + } else {
> > + IFPGA_BUS_ERR("arg %s is mandatory for ifpga bus",
> > + IFPGA_ARG_PORT);
> > + goto end;
> > + }
> > +
> > + if (rte_kvargs_count(kvlist, IFPGA_AFU_BTS) == 1) {
> > + if (rte_kvargs_process(kvlist, IFPGA_AFU_BTS,
> > + &rte_ifpga_get_string_arg, &path) < 0) {
> > + IFPGA_BUS_ERR("Failed to parse %s",
> > + IFPGA_AFU_BTS);
> > + goto end;
> > + }
> > + } else {
> > + IFPGA_BUS_ERR("arg %s is mandatory for ifpga bus",
> > + IFPGA_AFU_BTS);
> > + goto end;
> > + }
> > +
> > + afu_pr_conf.afu_id.uuid.uuid_low = 0;
> > + afu_pr_conf.afu_id.uuid.uuid_high = 0;
> > + afu_pr_conf.pr_enable = path?1:0;
> > +
> > + rawdev = ifpga_dev->rdev;
> > + if (ifpga_find_afu_dev(ifpga_dev, &afu_pr_conf.afu_id))
> > + goto end;
> > +
> > + afu_dev = calloc(1, sizeof(*afu_dev));
> > + if (!afu_dev)
> > + goto end;
> > +
> > + afu_dev->device.devargs = devargs;
> > + afu_dev->device.numa_node = SOCKET_ID_ANY;
> > + afu_dev->device.name = devargs->name;
> > + afu_dev->rawdev = rawdev;
> > + afu_dev->id.uuid.uuid_low = 0;
> > + afu_dev->id.uuid.uuid_high = 0;
> > + afu_dev->id.port = afu_pr_conf.afu_id.port;
> > + afu_dev->ifpga_dev = ifpga_dev;
> > +
> > + if (rawdev->dev_ops && rawdev->dev_ops->dev_info_get)
> > + rawdev->dev_ops->dev_info_get(rawdev, afu_dev);
> > +
> > + if (rawdev->dev_ops &&
> > + rawdev->dev_ops->dev_start &&
> > + rawdev->dev_ops->dev_start(rawdev))
> > + goto free_dev;
> > +
> > + strncpy(afu_pr_conf.bs_path, path, sizeof(afu_pr_conf.bs_path));
> > + if (rawdev->dev_ops->firmware_load &&
> > + rawdev->dev_ops->firmware_load(rawdev,
> > + &afu_pr_conf)){
> > + IFPGA_BUS_ERR("firmware load error %d\n", ret);
> > + goto free_dev;
> > + }
> > + afu_dev->id.uuid.uuid_low = afu_pr_conf.afu_id.uuid.uuid_low;
> > + afu_dev->id.uuid.uuid_high = afu_pr_conf.afu_id.uuid.uuid_high;
> > +
> > + return afu_dev;
> > +
> > +free_dev:
> > + free(afu_dev);
> > +end:
> > + if (kvlist)
> > + rte_kvargs_free(kvlist);
> > + if (path)
> > + free(path);
> > +
> > + return NULL;
> > +}
> > +
> > +/*
> > + * Scan the content of the FPGA bus, and the devices in the devices
> > + * list
> > + */
> > +static int
> > +ifpga_scan(void)
> > +{
> > + struct rte_ifpga_device *ifpga_dev;
> > + struct rte_devargs *devargs;
> > + struct rte_kvargs *kvlist = NULL;
> > + struct rte_rawdev *rawdev = NULL;
> > + char *name = NULL;
> > + char name1[RTE_RAWDEV_NAME_MAX_LEN];
> > + struct rte_afu_device *afu_dev = NULL;
> > +
> > + /* for FPGA devices we scan the devargs_list populated via cmdline
> > +*/
>
> I didn't see "--ifpga" is supported by cmdline, either you need to add
> corresponding parser or correct the comment here.
To be honestly, ifpga_scan() is called by hotplug, lib_rte hotplug function will construct it.
> > + RTE_EAL_DEVARGS_FOREACH("ifpga", devargs) {
>
>
> IFPGA_BUS_NAME to replace "ifpga" here.
>
> > + if (devargs->bus != &rte_ifpga_bus)
> > + continue;
> > +
> > + kvlist = rte_kvargs_parse(devargs->args, valid_args);
> > + if (!kvlist) {
> > + IFPGA_BUS_ERR("error when parsing param");
> > + goto end;
> > + }
> > +
> > + if (rte_kvargs_count(kvlist, IFPGA_ARG_NAME) == 1) {
> > + if (rte_kvargs_process(kvlist, IFPGA_ARG_NAME,
> > + &rte_ifpga_get_string_arg, &name) < 0) {
> > + IFPGA_BUS_ERR("error to parse %s",
> > + IFPGA_ARG_NAME);
> > + goto end;
> > + }
> > + } else {
> > + IFPGA_BUS_ERR("arg %s is mandatory for ifpga bus",
> > + IFPGA_ARG_NAME);
> > + goto end;
> > + }
> > +
> > + memset(name1, 0, sizeof(name1));
> > + snprintf(name1, RTE_RAWDEV_NAME_MAX_LEN,
> "IFPGA:%s",
> > name);
> > +
> > + rawdev = rte_rawdev_pmd_get_named_dev(name1);
> > + if (!rawdev)
> > + goto end;
> > +
> > + if (ifpga_find_ifpga_dev(rawdev))
> > + continue;
> > +
> > + ifpga_dev = calloc(1, sizeof(*ifpga_dev));
> > + if (!ifpga_dev)
> > + goto end;
> > +
> > + ifpga_dev->rdev = rawdev;
> > + TAILQ_INIT(&ifpga_dev->afu_list);
> > +
> > + TAILQ_INSERT_TAIL(&ifpga_device_list, ifpga_dev, next);
> > + afu_dev = ifpga_scan_one(devargs, ifpga_dev);
> > + if (afu_dev != NULL)
> > + TAILQ_INSERT_TAIL(&ifpga_dev->afu_list, afu_dev,
> next);
>
> I don't understand why we need afu_list?, seems we only add a new afu_dev
> into the list after create a new ifpga_dev, Is there any another place that we
> add one to list to make it reasonable? it looks like either a 1:1 or 1:0 for
> ifpga_dev:afu_dev?
One FPAG may support more than one AFU.
> > + }
> > +
> > +end:
> > + if (kvlist)
> > + rte_kvargs_free(kvlist);
> > + if (name)
> > + free(name);
> > +
> > + return 0;
> > +}
> > +
> > +/*
> > + * Match the AFU Driver and AFU Device using the ID Table */ static
> > +int rte_afu_match(const struct rte_afu_driver *afu_drv,
> > + const struct rte_afu_device *afu_dev) {
> > + const struct rte_afu_uuid *id_table;
> > +
> > + for (id_table = afu_drv->id_table;
> > + ((id_table->uuid_low != 0) && (id_table->uuid_high != 0));
> > + id_table++) {
> > + /* check if device's identifiers match the driver's ones */
> > + if ((id_table->uuid_low != afu_dev->id.uuid.uuid_low) ||
> > + id_table->uuid_high !=
> > + afu_dev->id.uuid.uuid_high)
> > + continue;
> > +
> > + return 1;
> > + }
> > +
> > + return 0;
> > +}
> > +
> > +static int
> > +ifpga_probe_one_driver(struct rte_afu_driver *drv,
> > + struct rte_afu_device *afu_dev)
> > +{
> > + int ret;
> > +
> > + if (!rte_afu_match(drv, afu_dev))
> > + /* Match of device and driver failed */
> > + return 1;
> > +
> > + /* reference driver structure */
> > + afu_dev->driver = drv;
> > + afu_dev->device.driver = &drv->driver;
> > +
> > + /* call the driver probe() function */
> > + ret = drv->probe(afu_dev);
> > + if (ret) {
> > + afu_dev->driver = NULL;
> > + afu_dev->device.driver = NULL;
> > + }
> > +
> > + return ret;
> > +}
> > +
> > +static int
> > +ifpga_probe_all_drivers(struct rte_afu_device *afu_dev) {
> > + struct rte_afu_driver *drv = NULL;
> > + int ret = 0;
> > +
> > + if (afu_dev == NULL)
> > + return -1;
> > +
> > + /* Check if a driver is already loaded */
> > + if (afu_dev->driver != NULL)
> > + return 0;
> > +
> > + TAILQ_FOREACH(drv, &afu_driver_list, next) {
> > + if (ifpga_probe_one_driver(drv, afu_dev)) {
> > + ret = -1;
> > + break;
> > + }
> > + }
> > + return ret;
> > +}
> > +
> > +/*
> > + * Scan the content of the Intel FPGA bus, and call the probe()
> > +function for
> > + * all registered drivers that have a matching entry in its id_table
> > + * for discovered devices.
> > + */
> > +static int
> > +ifpga_probe(void)
> > +{
> > + struct rte_ifpga_device *ifpga_dev;
> > + struct rte_afu_device *afu_dev = NULL;
> > + int ret = 0;
> > +
> > + TAILQ_FOREACH(ifpga_dev, &ifpga_device_list, next) {
> > + TAILQ_FOREACH(afu_dev, &ifpga_dev->afu_list, next) {
> > +
> > + if (afu_dev->device.driver)
> > + continue;
> > +
> > + ret = ifpga_probe_all_drivers(afu_dev);
> > + if (ret < 0)
> > + IFPGA_BUS_ERR("failed to initialize %s
> device\n",
> > + rte_ifpga_device_name(afu_dev));
> > + }
> > + }
> > +
> > + return ret;
> > +}
> > +
> > +static int
> > +ifpga_plug(struct rte_device *dev)
> > +{
> > + return ifpga_probe_all_drivers(RTE_DEV_TO_AFU(dev));
> > +}
> > +
> > +static int
> > +ifpga_remove_driver(struct rte_afu_device *afu_dev) {
> > + const char *name;
> > + const struct rte_afu_driver *driver;
> > +
> > + name = rte_ifpga_device_name(afu_dev);
> > + if (!afu_dev->device.driver) {
> > + IFPGA_BUS_DEBUG("no driver attach to device %s\n", name);
> > + return 1;
> > + }
> > +
> > + driver = RTE_DRV_TO_AFU_CONST(afu_dev->device.driver);
> > + return driver->remove(afu_dev);
> > +}
> > +
> > +static int
> > +ifpga_unplug(struct rte_device *dev)
> > +{
> > + struct rte_ifpga_device *ifpga_dev = NULL;
> > + struct rte_afu_device *afu_dev = NULL;
> > + struct rte_devargs *devargs = NULL;
> > + int ret;
> > +
> > + if (dev == NULL)
> > + return -EINVAL;
> > +
> > + afu_dev = RTE_DEV_TO_AFU(dev);
> > + if (!dev)
> > + return -ENOENT;
> > +
> > + ifpga_dev = afu_dev->ifpga_dev;
> > + devargs = dev->devargs;
> > +
> > + ret = ifpga_remove_driver(afu_dev);
>
> So what is the device type that be plugged into the ifgpa bus?
> ifpga_dev or afu_dev? If its afu_dev, why we need ifpga_dev?
> If it is ifpga_dev, we need remove ifpga_dev from ifpga_device_list here.
The device type that be plugged into the ifpga bus is afu_dev.
And each afu_dev will be added into one ifpga_dev.
Each FPGA device bind to ifpga_dev.
> > + if (ret)
> > + return ret;
> > +
> > + TAILQ_REMOVE(&ifpga_dev->afu_list, afu_dev, next);
> > +
> > + rte_devargs_remove(devargs->bus->name, devargs->name);
> > + free(afu_dev);
> > + return 0;
> > +
> > +}
> > +
> > +static struct rte_device *
> > +ifpga_find_device(const struct rte_device *start,
> > + rte_dev_cmp_t cmp, const void *data) {
> > + struct rte_ifpga_device *ifpga_dev;
> > + struct rte_afu_device *afu_dev;
> > +
> > + TAILQ_FOREACH(ifpga_dev, &ifpga_device_list, next) {
> > + TAILQ_FOREACH(afu_dev, &ifpga_dev->afu_list, next) {
> > + if (start && &afu_dev->device == start) {
> > + start = NULL;
> > + continue;
> > + }
> > + if (cmp(&afu_dev->device, data) == 0)
> > + return &afu_dev->device;
> > + }
> > + }
> > + return NULL;
> > +}
> > +static int
> > +ifpga_parse(const char *name, void *addr) {
> > + int *out = addr;
> > + struct rte_rawdev *rawdev = NULL;
> > + char rawdev_name[RTE_RAWDEV_NAME_MAX_LEN];
> > + char *c1 = NULL, *c2 = NULL;
> > + int port = IFPGA_BUS_DEV_PORT_MAX;
> > + char str_port[8];
> > + int str_port_len = 0;
> > + int ret;
> > +
> > + memset(str_port, 0, 8);
> > + c1 = strchr(name, '|');
> > + if (c1 != NULL) {
> > + str_port_len = c1-name;
> > + c2 = c1+1;
> > + }
> > +
> > + if (str_port_len < 8 &&
> > + str_port_len > 0) {
> > + memcpy(str_port, name, str_port_len);
> > + ret = sscanf(str_port, "%d", &port);
> > + if (ret == -1)
> > + return 0;
> > + }
> > +
> > + memset(rawdev_name, 0, sizeof(rawdev_name));
> > + snprintf(rawdev_name, RTE_RAWDEV_NAME_MAX_LEN, "IFPGA:%s",
> c2);
> > + rawdev = rte_rawdev_pmd_get_named_dev(rawdev_name);
> > +
> > + if ((port < IFPGA_BUS_DEV_PORT_MAX) &&
> > + rawdev &&
> > + (addr != NULL))
> > + *out = port;
> > +
> > + if ((port < IFPGA_BUS_DEV_PORT_MAX) &&
> > + rawdev)
> > + return 0;
> > + else
> > + return 1;
> > +}
> > +
> > +static struct rte_bus rte_ifpga_bus = {
> > + .scan = ifpga_scan,
> > + .probe = ifpga_probe,
> > + .find_device = ifpga_find_device,
> > + .plug = ifpga_plug,
> > + .unplug = ifpga_unplug,
> > + .parse = ifpga_parse,
> > +};
> > +
> > +RTE_REGISTER_BUS(IFPGA_BUS_NAME, rte_ifpga_bus);
> > +
> > +RTE_INIT(ifpga_init_log)
> > +{
> > + ifpga_bus_logtype = rte_log_register("bus.ifpga");
> > + if (ifpga_bus_logtype >= 0)
> > + rte_log_set_level(ifpga_bus_logtype, RTE_LOG_NOTICE); }
> > diff --git a/drivers/bus/ifpga/ifpga_common.c
> > b/drivers/bus/ifpga/ifpga_common.c
> > new file mode 100644
> > index 0000000..78e2eae
> > --- /dev/null
> > +++ b/drivers/bus/ifpga/ifpga_common.c
> > @@ -0,0 +1,88 @@
> > +/* SPDX-License-Identifier: BSD-3-Clause
> > + * Copyright(c) 2010-2018 Intel Corporation */
> > +
> > +#include <string.h>
> > +#include <inttypes.h>
> > +#include <stdint.h>
> > +#include <stdlib.h>
> > +#include <stdio.h>
> > +#include <sys/queue.h>
> > +#include <sys/mman.h>
> > +#include <sys/types.h>
> > +#include <unistd.h>
> > +#include <fcntl.h>
> > +
> > +#include <rte_errno.h>
> > +#include <rte_bus.h>
> > +#include <rte_per_lcore.h>
> > +#include <rte_memory.h>
> > +#include <rte_memzone.h>
> > +#include <rte_eal.h>
> > +#include <rte_common.h>
> > +
> > +#include <rte_devargs.h>
> > +#include <rte_kvargs.h>
> > +#include <rte_alarm.h>
> > +
> > +#include "rte_bus_ifpga.h"
> > +#include "ifpga_logs.h"
> > +#include "ifpga_common.h"
> > +
> > +int rte_ifpga_get_string_arg(const char *key __rte_unused,
> > + const char *value, void *extra_args) {
> > + if (!value || !extra_args)
> > + return -EINVAL;
> > +
> > + *(char **)extra_args = strdup(value);
> > +
> > + if (!*(char **)extra_args)
> > + return -ENOMEM;
> > +
> > + return 0;
> > +}
> > +int rte_ifpga_get_integer32_arg(const char *key __rte_unused,
> > + const char *value, void *extra_args) {
> > + if (!value || !extra_args)
> > + return -EINVAL;
> > +
> > + *(int *)extra_args = strtoull(value, NULL, 0);
> > +
> > + return 0;
> > +}
> > +int ifpga_get_integer64_arg(const char *key __rte_unused,
> > + const char *value, void *extra_args) {
> > + if (!value || !extra_args)
> > + return -EINVAL;
> > +
> > + *(uint64_t *)extra_args = strtoull(value, NULL, 0);
> > +
> > + return 0;
> > +}
> > +int ifpga_get_unsigned_long(const char *str, int base) {
> > + unsigned long num;
> > + char *end = NULL;
> > +
> > + errno = 0;
> > +
> > + num = strtoul(str, &end, base);
> > + if ((str[0] == '\0') || (end == NULL) || (*end != '\0') || (errno != 0))
> > + return -1;
> > +
> > + return num;
> > +}
> > +
> > +int ifpga_afu_id_cmp(const struct rte_afu_id *afu_id0,
> > + const struct rte_afu_id *afu_id1)
> > +{
> > + if ((afu_id0->uuid.uuid_low == afu_id1->uuid.uuid_low) &&
> > + (afu_id0->uuid.uuid_high == afu_id1->uuid.uuid_high) &&
> > + (afu_id0->port == afu_id1->port)) {
> > + return 0;
> > + } else
> > + return 1;
> > +}
> > diff --git a/drivers/bus/ifpga/ifpga_common.h
> > b/drivers/bus/ifpga/ifpga_common.h
> > new file mode 100644
> > index 0000000..f9254b9
> > --- /dev/null
> > +++ b/drivers/bus/ifpga/ifpga_common.h
> > @@ -0,0 +1,18 @@
> > +/* SPDX-License-Identifier: BSD-3-Clause
> > + * Copyright(c) 2010-2018 Intel Corporation */
> > +
> > +#ifndef _IFPGA_COMMON_H_
> > +#define _IFPGA_COMMON_H_
> > +
> > +int rte_ifpga_get_string_arg(const char *key __rte_unused,
> > + const char *value, void *extra_args); int
> > +rte_ifpga_get_integer32_arg(const char *key __rte_unused,
> > + const char *value, void *extra_args); int
> > +ifpga_get_integer64_arg(const char *key __rte_unused,
> > + const char *value, void *extra_args); int
> > +ifpga_get_unsigned_long(const char *str, int base); int
> > +ifpga_afu_id_cmp(const struct rte_afu_id *afu_id0,
> > + const struct rte_afu_id *afu_id1);
> > +
> > +#endif /* _IFPGA_COMMON_H_ */
> > diff --git a/drivers/bus/ifpga/ifpga_logs.h
> > b/drivers/bus/ifpga/ifpga_logs.h new file mode 100644 index
> > 0000000..873e0a4
> > --- /dev/null
> > +++ b/drivers/bus/ifpga/ifpga_logs.h
> > @@ -0,0 +1,31 @@
> > +/* SPDX-License-Identifier: BSD-3-Clause
> > + * Copyright(c) 2010-2018 Intel Corporation */
> > +
> > +#ifndef _IFPGA_LOGS_H_
> > +#define _IFPGA_LOGS_H_
> > +
> > +#include <rte_log.h>
> > +
> > +extern int ifpga_bus_logtype;
> > +
> > +#define IFPGA_LOG(level, fmt, args...) \
> > + rte_log(RTE_LOG_ ## level, ifpga_bus_logtype, "%s(): " fmt "\n", \
> > + __func__, ##args)
> > +
> > +#define IFPGA_BUS_LOG(level, fmt, args...) \
> > + rte_log(RTE_LOG_ ## level, ifpga_bus_logtype, "%s(): " fmt "\n", \
> > + __func__, ##args)
> > +
> > +#define IFPGA_BUS_FUNC_TRACE() IFPGA_BUS_LOG(DEBUG, ">>")
> > +
> > +#define IFPGA_BUS_DEBUG(fmt, args...) \
> > + IFPGA_BUS_LOG(DEBUG, fmt, ## args)
> > +#define IFPGA_BUS_INFO(fmt, args...) \
> > + IFPGA_BUS_LOG(INFO, fmt, ## args)
> > +#define IFPGA_BUS_ERR(fmt, args...) \
> > + IFPGA_BUS_LOG(ERR, fmt, ## args)
> > +#define IFPGA_BUS_WARN(fmt, args...) \
> > + IFPGA_BUS_LOG(WARNING, fmt, ## args)
> > +
> > +#endif /* _IFPGA_BUS_LOGS_H_ */
> > diff --git a/drivers/bus/ifpga/meson.build
> > b/drivers/bus/ifpga/meson.build new file mode 100644 index
> > 0000000..c9b08c8
> > --- /dev/null
> > +++ b/drivers/bus/ifpga/meson.build
> > @@ -0,0 +1,8 @@
> > +# SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2010-2018
> > +Intel Corporation
> > +
> > +deps += ['pci', 'kvargs', 'rawdev']
> > +install_headers('rte_bus_ifpga.h')
> > +sources = files('ifpga_common.c', 'ifpga_bus.c')
> > +
> > +allow_experimental_apis = true
> > diff --git a/drivers/bus/ifpga/rte_bus_ifpga.h
> > b/drivers/bus/ifpga/rte_bus_ifpga.h
> > new file mode 100644
> > index 0000000..5c559e1
> > --- /dev/null
> > +++ b/drivers/bus/ifpga/rte_bus_ifpga.h
> > @@ -0,0 +1,160 @@
> > +/* SPDX-License-Identifier: BSD-3-Clause
> > + * Copyright(c) 2010-2018 Intel Corporation */
> > +
> > +#ifndef _RTE_BUS_IFPGA_H_
> > +#define _RTE_BUS_IFPGA_H_
> > +
> > +/**
> > + * @file
> > + *
> > + * RTE Intel FPGA Bus Interface
> > + */
> > +
> > +#ifdef __cplusplus
> > +extern "C" {
> > +#endif
> > +
> > +#include <rte_bus.h>
> > +#include <rte_pci.h>
> > +
> > +/** Name of Intel FPGA Bus */
> > +#define IFPGA_BUS_NAME ifpga
> > +
> > +/* Forward declarations */
> > +struct rte_afu_device;
> > +
> > +/** List of Intel AFU devices */
> > +TAILQ_HEAD(afu_device_list, rte_afu_device);
> > +/** Double linked list of AFU device drivers. */
> > +TAILQ_HEAD(afu_driver_list, rte_afu_driver);
> > +
> > +#define IFPGA_BUS_BITSTREAM_PATH_MAX_LEN 256
> > +
> > +struct rte_afu_uuid {
> > + uint64_t uuid_low;
> > + uint64_t uuid_high;
> > +} __attribute__ ((packed));
> > +
> > +#define IFPGA_BUS_DEV_PORT_MAX 4
> > +
> > +/**
> > + * A structure describing an ID for a AFU driver. Each driver
> > +provides a
> > + * table of these IDs for each device that it supports.
> > + */
> > +struct rte_afu_id {
> > + struct rte_afu_uuid uuid;
> > + int port; /**< port number */
> > +} __attribute__ ((packed));
> > +
> > +/**
> > + * A structure PR (Partial Reconfiguration) configuration AFU driver.
> > + */
> > +
> > +struct rte_afu_pr_conf {
> > + struct rte_afu_id afu_id;
> > + int pr_enable;
> > + char bs_path[IFPGA_BUS_BITSTREAM_PATH_MAX_LEN];
> > +};
> > +
> > +#define AFU_PRI_STR_SIZE (PCI_PRI_STR_SIZE + 8)
> > +
> > +/**
> > + * A structure describing a fpga device.
> > + */
> > +struct rte_ifpga_device {
> > + TAILQ_ENTRY(rte_ifpga_device) next; /**< Next in device list. */
> > + struct rte_rawdev *rdev;
> > + struct afu_device_list afu_list; /**< List of AFU devices */ };
> > +
> > +/**
> > + * A structure describing a AFU device.
> > + */
> > +struct rte_afu_device {
> > + TAILQ_ENTRY(rte_afu_device) next; /**< Next in device list. */
> > + struct rte_device device; /**< Inherit core device */
> > + struct rte_rawdev *rawdev; /**< Point Rawdev */
> > + struct rte_ifpga_device *ifpga_dev; /**< Point ifpga device */
> > + struct rte_afu_id id; /**< AFU id within FPGA. */
> > + uint32_t num_region; /**< number of regions found */
> > + struct rte_mem_resource mem_resource[PCI_MAX_RESOURCE];
> > + /**< AFU Memory Resource
> */
> > + struct rte_intr_handle intr_handle; /**< Interrupt handle */
> > + struct rte_afu_driver *driver; /**< Associated driver */
> > + char path[IFPGA_BUS_BITSTREAM_PATH_MAX_LEN];
> > +} __attribute__ ((packed));
> > +
> > +/**
> > + * @internal
> > + * Helper macro for drivers that need to convert to struct rte_afu_device.
> > + */
> > +#define RTE_DEV_TO_AFU(ptr) \
> > + container_of(ptr, struct rte_afu_device, device)
> > +
> > +#define RTE_DRV_TO_AFU_CONST(ptr) \
> > + container_of(ptr, const struct rte_afu_driver, driver)
> > +
> > +/**
> > + * Initialisation function for the driver called during FPGA BUS probing.
>
>
> typo: Initialization
Fixed.
> > + */
> > +typedef int (afu_probe_t)(struct rte_afu_device *);
> > +
> > +/**
> > + * Uninitialisation function for the driver called during hotplugging.
>
> typo: Unintialization
Fixed.
> > + */
> > +typedef int (afu_remove_t)(struct rte_afu_device *);
> > +
> > +/**
> > + * A structure describing a AFU device.
> > + */
> > +struct rte_afu_driver {
> > + TAILQ_ENTRY(rte_afu_driver) next; /**< Next afu driver. */
> > + struct rte_driver driver; /**< Inherit core driver. */
> > + afu_probe_t *probe; /**< Device Probe function.
> > */
> > + afu_remove_t *remove; /**< Device Remove
> > function. */
> > + const struct rte_afu_uuid *id_table; /**< AFU uuid within FPGA. */
> > + uint32_t drv_flags; /**< Flags contolling handling of device. */
>
> Typo Controlling
> And what is "flags controlling handling of device?"
This variable is not used, so I have removed it.
> > +};
> > +
> > +static inline const char *
> > +rte_ifpga_device_name(const struct rte_afu_device *afu) {
> > + if (afu && afu->device.name)
> > + return afu->device.name;
> > + return NULL;
> > +}
> > +
> > +/**
> > + * Register a ifpga afu device driver.
> > + *
> > + * @param driver
> > + * A pointer to a rte_afu_driver structure describing the driver
> > + * to be registered.
> > + */
> > +void rte_ifpga_driver_register(struct rte_afu_driver *driver);
> > +
> > +/**
> > + * Unregister a ifpga afu device driver.
> > + *
> > + * @param driver
> > + * A pointer to a rte_afu_driver structure describing the driver
> > + * to be unregistered.
> > + */
> > +void rte_ifpga_driver_unregister(struct rte_afu_driver *driver);
> > +
> > +#define RTE_PMD_REGISTER_AFU(nm, afudrv)\ RTE_INIT(afudrvinitfn_
> > +##afudrv);\ static const char *afudrvinit_ ## nm ## _alias;\ static
> > +void afudrvinitfn_ ##afudrv(void)\ {\
> > + (afudrv).driver.name = RTE_STR(nm);\
> > + (afudrv).driver.alias = afudrvinit_ ## nm ## _alias;\
> > + rte_ifpga_driver_register(&afudrv);\
> > +} \
> > +RTE_PMD_EXPORT_NAME(nm, __COUNTER__)
> > +
> > +#define RTE_PMD_REGISTER_AFU_ALIAS(nm, alias)\ static const char
> > +*afudrvinit_ ## nm ## _alias = RTE_STR(alias)
> > +
> > +#endif /* _RTE_BUS_IFPGA_H_ */
> > diff --git a/drivers/bus/ifpga/rte_bus_ifpga_version.map
> > b/drivers/bus/ifpga/rte_bus_ifpga_version.map
> > new file mode 100644
> > index 0000000..a027979
> > --- /dev/null
> > +++ b/drivers/bus/ifpga/rte_bus_ifpga_version.map
> > @@ -0,0 +1,10 @@
> > +DPDK_18.05 {
> > + global:
> > +
> > + rte_ifpga_get_integer32_arg;
> > + rte_ifpga_get_string_arg;
> > + rte_ifpga_driver_register;
> > + rte_ifpga_driver_unregister;
> > +
> > + local: *;
> > +};
> > diff --git a/drivers/bus/meson.build b/drivers/bus/meson.build index
> > 58dfbe2..52c755d 100644
> > --- a/drivers/bus/meson.build
> > +++ b/drivers/bus/meson.build
> > @@ -1,7 +1,7 @@
> > # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2017 Intel
> > Corporation
> >
> > -drivers = ['dpaa', 'fslmc', 'pci', 'vdev']
> > +drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'vdev']
> > std_deps = ['eal']
> > config_flag_fmt = 'RTE_LIBRTE_@0@_BUS'
> > driver_name_fmt = 'rte_bus_@0@'
> > diff --git a/mk/rte.app.mk b/mk/rte.app.mk index 26f3563..3861e1a
> > 100644
> > --- a/mk/rte.app.mk
> > +++ b/mk/rte.app.mk
> > @@ -255,6 +255,7 @@ ifeq
> > ($(CONFIG_RTE_EAL_VFIO)$(CONFIG_RTE_LIBRTE_FSLMC_BUS),yy)
> > _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_CMDIF_RAWDEV) +=
> > -lrte_pmd_dpaa2_cmdif
> > _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV) +=
> > -lrte_pmd_dpaa2_qdma endif # CONFIG_RTE_LIBRTE_FSLMC_BUS
> > +_LDLIBS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += -lrte_bus_ifpga
> > endif # CONFIG_RTE_LIBRTE_RAWDEV
> >
> >
> > --
> > 1.8.3.1
>
> Regards
> Qi
> -----Original Message-----
> From: Xu, Rosen
> Sent: Thursday, May 10, 2018 9:29 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; dev@dpdk.org;
> thomas@monjalon.net
> Cc: Zhang, Roy Fan <roy.fan.zhang@intel.com>; Doherty, Declan
> <declan.doherty@intel.com>; Richardson, Bruce
> <bruce.richardson@intel.com>; shreyansh.jain@nxp.com; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Ananyev, Konstantin
> <konstantin.ananyev@intel.com>; Zhang, Tianfei <tianfei.zhang@intel.com>;
> Liu, Song <song.liu@intel.com>; Wu, Hao <hao.wu@intel.com>;
> gaetan.rivet@6wind.com
> Subject: RE: [dpdk-dev] [PATCH v10 1/3] bus/ifpga: Add Intel FPGA BUS Library
>
> Hi Qi,
>
> > -----Original Message-----
> > From: Zhang, Qi Z
> > Sent: Thursday, May 10, 2018 20:27
> > To: Xu, Rosen <rosen.xu@intel.com>; dev@dpdk.org;
> thomas@monjalon.net
> > Cc: Xu, Rosen <rosen.xu@intel.com>; Zhang, Roy Fan
> > <roy.fan.zhang@intel.com>; Doherty, Declan <declan.doherty@intel.com>;
> > Richardson, Bruce <bruce.richardson@intel.com>;
> > shreyansh.jain@nxp.com; Yigit, Ferruh <ferruh.yigit@intel.com>;
> > Ananyev, Konstantin <konstantin.ananyev@intel.com>; Zhang, Tianfei
> > <tianfei.zhang@intel.com>; Liu, Song <song.liu@intel.com>; Wu, Hao
> > <hao.wu@intel.com>; gaetan.rivet@6wind.com
> > Subject: RE: [dpdk-dev] [PATCH v10 1/3] bus/ifpga: Add Intel FPGA BUS
> > Library
> >
> > Hi Rosen:
> >
> > > -----Original Message-----
> > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Xu, Rosen
> > > Sent: Wednesday, May 9, 2018 3:43 PM
> > > To: dev@dpdk.org; thomas@monjalon.net
> > > Cc: Xu, Rosen <rosen.xu@intel.com>; Zhang, Roy Fan
> > > <roy.fan.zhang@intel.com>; Doherty, Declan
> > > <declan.doherty@intel.com>; Richardson, Bruce
> > > <bruce.richardson@intel.com>; shreyansh.jain@nxp.com; Yigit, Ferruh
> > > <ferruh.yigit@intel.com>; Ananyev, Konstantin
> > > <konstantin.ananyev@intel.com>; Zhang, Tianfei
> > > <tianfei.zhang@intel.com>; Liu, Song <song.liu@intel.com>; Wu, Hao
> > > <hao.wu@intel.com>; gaetan.rivet@6wind.com
> > > Subject: [dpdk-dev] [PATCH v10 1/3] bus/ifpga: Add Intel FPGA BUS
> > > Library
> > >
> > > From: Rosen Xu <rosen.xu@intel.com>
> > >
> > > Defined FPGA-BUS for Acceleration Drivers of AFUs
> > >
> > > 1. FPGA PCI Scan (1st Scan) follows DPDK UIO/VFIO PCI Scan Process,
> > > probe Intel FPGA Rawdev Driver, it will be covered in following patches.
> > >
> > > 2. AFU Scan(2nd Scan) bind DPDK driver to FPGA Partial-Bitstream.
> > > This scan is trigged by hotplug of IFPGA Rawdev probe, in this scan
> > > the AFUs will be created and their drivers are also probed.
> > >
> > > This patch will introduce rte_afu_device which describe the AFU
> > > device listed in the FPGA-BUS.
> > >
> > > Signed-off-by: Rosen Xu <rosen.xu@intel.com>
> > > Signed-off-by: Tianfei Zhang <tianfei.zhang@intel.com>
> > > ---
> > > MAINTAINERS | 4 +
> > > config/common_base | 5 +
> > > doc/guides/rel_notes/release_18_05.rst | 5 +
> > > drivers/bus/Makefile | 1 +
> > > drivers/bus/ifpga/Makefile | 32 ++
> > > drivers/bus/ifpga/ifpga_bus.c | 501
> > > ++++++++++++++++++++++++++++
> > > drivers/bus/ifpga/ifpga_common.c | 88 +++++
> > > drivers/bus/ifpga/ifpga_common.h | 18 +
> > > drivers/bus/ifpga/ifpga_logs.h | 31 ++
> > > drivers/bus/ifpga/meson.build | 8 +
> > > drivers/bus/ifpga/rte_bus_ifpga.h | 160 +++++++++
> > > drivers/bus/ifpga/rte_bus_ifpga_version.map | 10 +
> > > drivers/bus/meson.build | 2 +-
> > > mk/rte.app.mk | 1 +
> > > 14 files changed, 865 insertions(+), 1 deletion(-) create mode
> > > 100644 drivers/bus/ifpga/Makefile create mode 100644
> > > drivers/bus/ifpga/ifpga_bus.c create mode 100644
> > > drivers/bus/ifpga/ifpga_common.c create mode 100644
> > > drivers/bus/ifpga/ifpga_common.h create mode 100644
> > > drivers/bus/ifpga/ifpga_logs.h create mode 100644
> > > drivers/bus/ifpga/meson.build create mode 100644
> > > drivers/bus/ifpga/rte_bus_ifpga.h create mode 100644
> > > drivers/bus/ifpga/rte_bus_ifpga_version.map
> > >
> > > diff --git a/MAINTAINERS b/MAINTAINERS index 7105920..fa0c5b1 100644
> > > --- a/MAINTAINERS
> > > +++ b/MAINTAINERS
> > > @@ -383,6 +383,10 @@ F: drivers/mempool/bucket/ Bus Drivers
> > > -----------
> > >
> > > +Intel FPGA buses
> > > +M: Rosen Xu <rosen.xu@intel.com>
> > > +F: drivers/bus/ifpga/
> > > +
> > > NXP buses
> > > M: Hemant Agrawal <hemant.agrawal@nxp.com>
> > > M: Shreyansh Jain <shreyansh.jain@nxp.com> diff --git
> > > a/config/common_base b/config/common_base index 0d181ac..1440316
> > > 100644
> > > --- a/config/common_base
> > > +++ b/config/common_base
> > > @@ -139,6 +139,11 @@
> > > CONFIG_RTE_ETHDEV_PROFILE_ITT_WASTED_RX_ITERATIONS=n
> > > CONFIG_RTE_ETHDEV_TX_PREPARE_NOOP=n
> > >
> > > #
> > > +# Compile the Intel FPGA bus
> > > +#
> > > +CONFIG_RTE_LIBRTE_IFPGA_BUS=y
> > > +
> > > +#
> > > # Compile PCI bus driver
> > > #
> > > CONFIG_RTE_LIBRTE_PCI_BUS=y
> > > diff --git a/doc/guides/rel_notes/release_18_05.rst
> > > b/doc/guides/rel_notes/release_18_05.rst
> > > index 7187348..265950a 100644
> > > --- a/doc/guides/rel_notes/release_18_05.rst
> > > +++ b/doc/guides/rel_notes/release_18_05.rst
> > > @@ -183,6 +183,11 @@ New Features
> > > stats/xstats on shared memory from secondary process, and also
> > > pdump packets on
> > > those virtual devices.
> > >
> > > +* **Added Ifpga Bus, a generic Intel FPGA Bus library.**
> > > +
> > > + The Ifpga Bus library provides support for integrating any Intel
> > > + FPGA device with the DPDK framework. It provides Intel FPGA
> > > + Partial Bit Stream AFU(Accelerated Function Unit) scan and drivers
> prove.
> > >
> > > API Changes
> > > -----------
> > > diff --git a/drivers/bus/Makefile b/drivers/bus/Makefile index
> > > c251b65..ef7f247 100644
> > > --- a/drivers/bus/Makefile
> > > +++ b/drivers/bus/Makefile
> > > @@ -7,6 +7,7 @@ DIRS-$(CONFIG_RTE_LIBRTE_DPAA_BUS) += dpaa ifeq
> > > ($(CONFIG_RTE_EAL_VFIO),y)
> > > DIRS-$(CONFIG_RTE_LIBRTE_FSLMC_BUS) += fslmc endif
> > > +DIRS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += ifpga
> > > DIRS-$(CONFIG_RTE_LIBRTE_PCI_BUS) += pci
> > > DIRS-$(CONFIG_RTE_LIBRTE_VDEV_BUS) += vdev
> > >
> > > diff --git a/drivers/bus/ifpga/Makefile b/drivers/bus/ifpga/Makefile
> > > new file mode 100644 index 0000000..3ff3bdb
> > > --- /dev/null
> > > +++ b/drivers/bus/ifpga/Makefile
> > > @@ -0,0 +1,32 @@
> > > +# SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2018 Intel
> > > +Corporation
> > > +
> > > +include $(RTE_SDK)/mk/rte.vars.mk
> > > +
> > > +#
> > > +# library name
> > > +#
> > > +LIB = librte_bus_ifpga.a
> > > +
> > > +CFLAGS += -DALLOW_EXPERIMENTAL_API
> > > +CFLAGS += -O3
> > > +CFLAGS += $(WERROR_FLAGS)
> > > +LDLIBS += -lrte_eal
> > > +LDLIBS += -lrte_rawdev
> > > +LDLIBS += -lrte_kvargs
> > > +
> > > +# versioning export map
> > > +EXPORT_MAP := rte_bus_ifpga_version.map
> > > +
> > > +# library version
> > > +LIBABIVER := 1
> > > +
> > > +SRCS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += ifpga_bus.c
> > > +SRCS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += ifpga_common.c
> > > +
> > > +#
> > > +# Export include files
> > > +#
> > > +SYMLINK-$(CONFIG_RTE_LIBRTE_IFPGA_BUS)-include += rte_bus_ifpga.h
> > > +
> > > +include $(RTE_SDK)/mk/rte.lib.mk
> > > diff --git a/drivers/bus/ifpga/ifpga_bus.c
> > > b/drivers/bus/ifpga/ifpga_bus.c new file mode 100644 index
> > > 0000000..e144c01
> > > --- /dev/null
> > > +++ b/drivers/bus/ifpga/ifpga_bus.c
> > > @@ -0,0 +1,501 @@
> > > +/* SPDX-License-Identifier: BSD-3-Clause
> > > + * Copyright(c) 2010-2018 Intel Corporation */
> > > +
> > > +#include <string.h>
> > > +#include <inttypes.h>
> > > +#include <stdint.h>
> > > +#include <stdlib.h>
> > > +#include <stdio.h>
> > > +#include <sys/queue.h>
> > > +#include <sys/mman.h>
> > > +#include <sys/types.h>
> > > +#include <unistd.h>
> > > +#include <fcntl.h>
> > > +
> > > +#include <rte_errno.h>
> > > +#include <rte_bus.h>
> > > +#include <rte_per_lcore.h>
> > > +#include <rte_memory.h>
> > > +#include <rte_memzone.h>
> > > +#include <rte_eal.h>
> > > +#include <rte_common.h>
> > > +
> > > +#include <rte_devargs.h>
> > > +#include <rte_kvargs.h>
> > > +#include <rte_alarm.h>
> > > +
> > > +#include "rte_rawdev.h"
> > > +#include "rte_rawdev_pmd.h"
> > > +#include "rte_bus_ifpga.h"
> > > +#include "ifpga_logs.h"
> > > +#include "ifpga_common.h"
> > > +
> > > +int ifpga_bus_logtype;
> > > +
> > > +/* Forward declaration to access Intel FPGA bus
> > > + * on which iFPGA devices are connected */ static struct rte_bus
> > > +rte_ifpga_bus;
> > > +
> > > +/** Double linked list of IFPGA device. */
> > > +TAILQ_HEAD(ifpga_device_list, rte_ifpga_device);
> > > +
> > > +static struct ifpga_device_list ifpga_device_list =
> > > + TAILQ_HEAD_INITIALIZER(ifpga_device_list);
> > > +static struct afu_driver_list afu_driver_list =
> > > + TAILQ_HEAD_INITIALIZER(afu_driver_list);
> > > +
> > > +
> > > +/* register a ifpga bus based driver */ void
> > > +rte_ifpga_driver_register(struct rte_afu_driver *driver) {
> > > + RTE_VERIFY(driver);
> > > +
> > > + TAILQ_INSERT_TAIL(&afu_driver_list, driver, next); }
> > > +
> > > +/* un-register a fpga bus based driver */ void
> > > +rte_ifpga_driver_unregister(struct rte_afu_driver *driver) {
> > > + TAILQ_REMOVE(&afu_driver_list, driver, next); }
> > > +
> > > +static struct rte_ifpga_device *
> > > +ifpga_find_ifpga_dev(const struct rte_rawdev *rdev) {
> > > + struct rte_ifpga_device *ifpga_dev = NULL;
> > > +
> > > + TAILQ_FOREACH(ifpga_dev, &ifpga_device_list, next) {
> > > + if (rdev &&
> > > + ifpga_dev->rdev &&
> > > + ifpga_dev->rdev == rdev)
> > > + return ifpga_dev;
> > > + }
> > > + return NULL;
> > > +}
> > > +
> > > +static struct rte_afu_device *
> > > +ifpga_find_afu_dev(const struct rte_ifpga_device *ifpga_dev,
> > > + const struct rte_afu_id *afu_id)
> > > +{
> > > + struct rte_afu_device *afu_dev = NULL;
> > > +
> > > + TAILQ_FOREACH(afu_dev, &ifpga_dev->afu_list, next) {
> > > + if (!ifpga_afu_id_cmp(&afu_dev->id, afu_id))
> > > + return afu_dev;
> > > + }
> > > + return NULL;
> > > +}
> > > +
> > > +static const char * const valid_args[] = {
> > > +#define IFPGA_ARG_NAME "ifpga"
> > > + IFPGA_ARG_NAME,
> > > +#define IFPGA_ARG_PORT "port"
> > > + IFPGA_ARG_PORT,
> > > +#define IFPGA_AFU_BTS "afu_bts"
> > > + IFPGA_AFU_BTS,
> > > + NULL
> > > +};
> > > +
> > > +/*
> > > + * Scan the content of the FPGA bus, and the devices in the devices
> > > + * list
> > > + */
> > > +static struct rte_afu_device *
> > > +ifpga_scan_one(struct rte_devargs *devargs,
> > > + struct rte_ifpga_device *ifpga_dev)
> >
> > usually ifpag_dev should be the first parameter here, and devargs
> > follows
>
> Fixed.
>
> > > +{
> > > + struct rte_kvargs *kvlist = NULL;
> > > + struct rte_rawdev *rawdev = NULL;
> > > + struct rte_afu_device *afu_dev = NULL;
> > > + struct rte_afu_pr_conf afu_pr_conf;
> > > + int ret = 0;
> > > + char *path = NULL;
> > > +
> > > + memset(&afu_pr_conf, 0, sizeof(struct rte_afu_pr_conf));
> > > +
> > > + kvlist = rte_kvargs_parse(devargs->args, valid_args);
> > > + if (!kvlist) {
> > > + IFPGA_BUS_ERR("error when parsing param");
> > > + goto end;
> > > + }
> > > +
> > > + if (rte_kvargs_count(kvlist, IFPGA_ARG_PORT) == 1) {
> > > + if (rte_kvargs_process(kvlist, IFPGA_ARG_PORT,
> > > + &rte_ifpga_get_integer32_arg, &afu_pr_conf.afu_id.port) < 0)
> > {
> > > + IFPGA_BUS_ERR("error to parse %s",
> > > + IFPGA_ARG_PORT);
> > > + goto end;
> > > + }
> > > + } else {
> > > + IFPGA_BUS_ERR("arg %s is mandatory for ifpga bus",
> > > + IFPGA_ARG_PORT);
> > > + goto end;
> > > + }
> > > +
> > > + if (rte_kvargs_count(kvlist, IFPGA_AFU_BTS) == 1) {
> > > + if (rte_kvargs_process(kvlist, IFPGA_AFU_BTS,
> > > + &rte_ifpga_get_string_arg, &path) < 0) {
> > > + IFPGA_BUS_ERR("Failed to parse %s",
> > > + IFPGA_AFU_BTS);
> > > + goto end;
> > > + }
> > > + } else {
> > > + IFPGA_BUS_ERR("arg %s is mandatory for ifpga bus",
> > > + IFPGA_AFU_BTS);
> > > + goto end;
> > > + }
> > > +
> > > + afu_pr_conf.afu_id.uuid.uuid_low = 0;
> > > + afu_pr_conf.afu_id.uuid.uuid_high = 0;
> > > + afu_pr_conf.pr_enable = path?1:0;
> > > +
> > > + rawdev = ifpga_dev->rdev;
> > > + if (ifpga_find_afu_dev(ifpga_dev, &afu_pr_conf.afu_id))
> > > + goto end;
> > > +
> > > + afu_dev = calloc(1, sizeof(*afu_dev));
> > > + if (!afu_dev)
> > > + goto end;
> > > +
> > > + afu_dev->device.devargs = devargs;
> > > + afu_dev->device.numa_node = SOCKET_ID_ANY;
> > > + afu_dev->device.name = devargs->name;
> > > + afu_dev->rawdev = rawdev;
> > > + afu_dev->id.uuid.uuid_low = 0;
> > > + afu_dev->id.uuid.uuid_high = 0;
> > > + afu_dev->id.port = afu_pr_conf.afu_id.port;
> > > + afu_dev->ifpga_dev = ifpga_dev;
> > > +
> > > + if (rawdev->dev_ops && rawdev->dev_ops->dev_info_get)
> > > + rawdev->dev_ops->dev_info_get(rawdev, afu_dev);
> > > +
> > > + if (rawdev->dev_ops &&
> > > + rawdev->dev_ops->dev_start &&
> > > + rawdev->dev_ops->dev_start(rawdev))
> > > + goto free_dev;
> > > +
> > > + strncpy(afu_pr_conf.bs_path, path, sizeof(afu_pr_conf.bs_path));
> > > + if (rawdev->dev_ops->firmware_load &&
> > > + rawdev->dev_ops->firmware_load(rawdev,
> > > + &afu_pr_conf)){
> > > + IFPGA_BUS_ERR("firmware load error %d\n", ret);
> > > + goto free_dev;
> > > + }
> > > + afu_dev->id.uuid.uuid_low = afu_pr_conf.afu_id.uuid.uuid_low;
> > > + afu_dev->id.uuid.uuid_high = afu_pr_conf.afu_id.uuid.uuid_high;
> > > +
> > > + return afu_dev;
> > > +
> > > +free_dev:
> > > + free(afu_dev);
> > > +end:
> > > + if (kvlist)
> > > + rte_kvargs_free(kvlist);
> > > + if (path)
> > > + free(path);
> > > +
> > > + return NULL;
> > > +}
> > > +
> > > +/*
> > > + * Scan the content of the FPGA bus, and the devices in the devices
> > > + * list
> > > + */
> > > +static int
> > > +ifpga_scan(void)
> > > +{
> > > + struct rte_ifpga_device *ifpga_dev;
> > > + struct rte_devargs *devargs;
> > > + struct rte_kvargs *kvlist = NULL;
> > > + struct rte_rawdev *rawdev = NULL;
> > > + char *name = NULL;
> > > + char name1[RTE_RAWDEV_NAME_MAX_LEN];
> > > + struct rte_afu_device *afu_dev = NULL;
> > > +
> > > + /* for FPGA devices we scan the devargs_list populated via cmdline
> > > +*/
> >
> > I didn't see "--ifpga" is supported by cmdline, either you need to add
> > corresponding parser or correct the comment here.
>
> To be honestly, ifpga_scan() is called by hotplug, lib_rte hotplug function will
> construct it.
OK, after review patch 3, I saw it, that make sense.
>
> > > + RTE_EAL_DEVARGS_FOREACH("ifpga", devargs) {
> >
> >
> > IFPGA_BUS_NAME to replace "ifpga" here.
> >
> > > + if (devargs->bus != &rte_ifpga_bus)
> > > + continue;
> > > +
> > > + kvlist = rte_kvargs_parse(devargs->args, valid_args);
> > > + if (!kvlist) {
> > > + IFPGA_BUS_ERR("error when parsing param");
> > > + goto end;
> > > + }
> > > +
> > > + if (rte_kvargs_count(kvlist, IFPGA_ARG_NAME) == 1) {
> > > + if (rte_kvargs_process(kvlist, IFPGA_ARG_NAME,
> > > + &rte_ifpga_get_string_arg, &name) < 0) {
> > > + IFPGA_BUS_ERR("error to parse %s",
> > > + IFPGA_ARG_NAME);
> > > + goto end;
> > > + }
> > > + } else {
> > > + IFPGA_BUS_ERR("arg %s is mandatory for ifpga bus",
> > > + IFPGA_ARG_NAME);
> > > + goto end;
> > > + }
> > > +
> > > + memset(name1, 0, sizeof(name1));
> > > + snprintf(name1, RTE_RAWDEV_NAME_MAX_LEN,
> > "IFPGA:%s",
> > > name);
> > > +
> > > + rawdev = rte_rawdev_pmd_get_named_dev(name1);
> > > + if (!rawdev)
> > > + goto end;
> > > +
> > > + if (ifpga_find_ifpga_dev(rawdev))
> > > + continue;
> > > +
> > > + ifpga_dev = calloc(1, sizeof(*ifpga_dev));
> > > + if (!ifpga_dev)
> > > + goto end;
> > > +
> > > + ifpga_dev->rdev = rawdev;
> > > + TAILQ_INIT(&ifpga_dev->afu_list);
> > > +
> > > + TAILQ_INSERT_TAIL(&ifpga_device_list, ifpga_dev, next);
> > > + afu_dev = ifpga_scan_one(devargs, ifpga_dev);
> > > + if (afu_dev != NULL)
> > > + TAILQ_INSERT_TAIL(&ifpga_dev->afu_list, afu_dev,
> > next);
> >
> > I don't understand why we need afu_list?, seems we only add a new
> > afu_dev into the list after create a new ifpga_dev, Is there any
> > another place that we add one to list to make it reasonable? it looks
> > like either a 1:1 or 1:0 for ifpga_dev:afu_dev?
>
> One FPAG may support more than one AFU.
But in your implementation, it is not possible that an ifpga_dev contains multiple afu_dev
>
> > > + }
> > > +
> > > +end:
> > > + if (kvlist)
> > > + rte_kvargs_free(kvlist);
> > > + if (name)
> > > + free(name);
> > > +
> > > + return 0;
> > > +}
> > > +
> > > +/*
> > > + * Match the AFU Driver and AFU Device using the ID Table */
> > > +static int rte_afu_match(const struct rte_afu_driver *afu_drv,
> > > + const struct rte_afu_device *afu_dev) {
> > > + const struct rte_afu_uuid *id_table;
> > > +
> > > + for (id_table = afu_drv->id_table;
> > > + ((id_table->uuid_low != 0) && (id_table->uuid_high != 0));
> > > + id_table++) {
> > > + /* check if device's identifiers match the driver's ones */
> > > + if ((id_table->uuid_low != afu_dev->id.uuid.uuid_low) ||
> > > + id_table->uuid_high !=
> > > + afu_dev->id.uuid.uuid_high)
> > > + continue;
> > > +
> > > + return 1;
> > > + }
> > > +
> > > + return 0;
> > > +}
> > > +
> > > +static int
> > > +ifpga_probe_one_driver(struct rte_afu_driver *drv,
> > > + struct rte_afu_device *afu_dev)
> > > +{
> > > + int ret;
> > > +
> > > + if (!rte_afu_match(drv, afu_dev))
> > > + /* Match of device and driver failed */
> > > + return 1;
> > > +
> > > + /* reference driver structure */
> > > + afu_dev->driver = drv;
> > > + afu_dev->device.driver = &drv->driver;
> > > +
> > > + /* call the driver probe() function */
> > > + ret = drv->probe(afu_dev);
> > > + if (ret) {
> > > + afu_dev->driver = NULL;
> > > + afu_dev->device.driver = NULL;
> > > + }
> > > +
> > > + return ret;
> > > +}
> > > +
> > > +static int
> > > +ifpga_probe_all_drivers(struct rte_afu_device *afu_dev) {
> > > + struct rte_afu_driver *drv = NULL;
> > > + int ret = 0;
> > > +
> > > + if (afu_dev == NULL)
> > > + return -1;
> > > +
> > > + /* Check if a driver is already loaded */
> > > + if (afu_dev->driver != NULL)
> > > + return 0;
> > > +
> > > + TAILQ_FOREACH(drv, &afu_driver_list, next) {
> > > + if (ifpga_probe_one_driver(drv, afu_dev)) {
> > > + ret = -1;
> > > + break;
> > > + }
> > > + }
> > > + return ret;
> > > +}
> > > +
> > > +/*
> > > + * Scan the content of the Intel FPGA bus, and call the probe()
> > > +function for
> > > + * all registered drivers that have a matching entry in its
> > > +id_table
> > > + * for discovered devices.
> > > + */
> > > +static int
> > > +ifpga_probe(void)
> > > +{
> > > + struct rte_ifpga_device *ifpga_dev;
> > > + struct rte_afu_device *afu_dev = NULL;
> > > + int ret = 0;
> > > +
> > > + TAILQ_FOREACH(ifpga_dev, &ifpga_device_list, next) {
> > > + TAILQ_FOREACH(afu_dev, &ifpga_dev->afu_list, next) {
> > > +
> > > + if (afu_dev->device.driver)
> > > + continue;
> > > +
> > > + ret = ifpga_probe_all_drivers(afu_dev);
> > > + if (ret < 0)
> > > + IFPGA_BUS_ERR("failed to initialize %s
> > device\n",
> > > + rte_ifpga_device_name(afu_dev));
> > > + }
> > > + }
> > > +
> > > + return ret;
> > > +}
> > > +
> > > +static int
> > > +ifpga_plug(struct rte_device *dev)
> > > +{
> > > + return ifpga_probe_all_drivers(RTE_DEV_TO_AFU(dev));
> > > +}
> > > +
> > > +static int
> > > +ifpga_remove_driver(struct rte_afu_device *afu_dev) {
> > > + const char *name;
> > > + const struct rte_afu_driver *driver;
> > > +
> > > + name = rte_ifpga_device_name(afu_dev);
> > > + if (!afu_dev->device.driver) {
> > > + IFPGA_BUS_DEBUG("no driver attach to device %s\n", name);
> > > + return 1;
> > > + }
> > > +
> > > + driver = RTE_DRV_TO_AFU_CONST(afu_dev->device.driver);
> > > + return driver->remove(afu_dev);
> > > +}
> > > +
> > > +static int
> > > +ifpga_unplug(struct rte_device *dev) {
> > > + struct rte_ifpga_device *ifpga_dev = NULL;
> > > + struct rte_afu_device *afu_dev = NULL;
> > > + struct rte_devargs *devargs = NULL;
> > > + int ret;
> > > +
> > > + if (dev == NULL)
> > > + return -EINVAL;
> > > +
> > > + afu_dev = RTE_DEV_TO_AFU(dev);
> > > + if (!dev)
> > > + return -ENOENT;
> > > +
> > > + ifpga_dev = afu_dev->ifpga_dev;
> > > + devargs = dev->devargs;
> > > +
> > > + ret = ifpga_remove_driver(afu_dev);
> >
> > So what is the device type that be plugged into the ifgpa bus?
> > ifpga_dev or afu_dev? If its afu_dev, why we need ifpga_dev?
> > If it is ifpga_dev, we need remove ifpga_dev from ifpga_device_list here.
>
> The device type that be plugged into the ifpga bus is afu_dev.
In DPDK, we have rte_pci_device for pci bus
and rte_vdev_device for vdev bus
Now we have rte_ifpga_device not for ifgpa bus but rte_afu_device, it is a little bit confuse here
> And each afu_dev will be added into one ifpga_dev.
> Each FPGA device bind to ifpga_dev.
So what is the relationship between ifpga_dev and raw_dev?
>
> > > + if (ret)
> > > + return ret;
> > > +
> > > + TAILQ_REMOVE(&ifpga_dev->afu_list, afu_dev, next);
> > > +
> > > + rte_devargs_remove(devargs->bus->name, devargs->name);
> > > + free(afu_dev);
> > > + return 0;
> > > +
> > > +}
> > > +
> > > +static struct rte_device *
> > > +ifpga_find_device(const struct rte_device *start,
> > > + rte_dev_cmp_t cmp, const void *data) {
> > > + struct rte_ifpga_device *ifpga_dev;
> > > + struct rte_afu_device *afu_dev;
> > > +
> > > + TAILQ_FOREACH(ifpga_dev, &ifpga_device_list, next) {
> > > + TAILQ_FOREACH(afu_dev, &ifpga_dev->afu_list, next) {
> > > + if (start && &afu_dev->device == start) {
> > > + start = NULL;
> > > + continue;
> > > + }
> > > + if (cmp(&afu_dev->device, data) == 0)
> > > + return &afu_dev->device;
> > > + }
> > > + }
> > > + return NULL;
> > > +}
> > > +static int
> > > +ifpga_parse(const char *name, void *addr) {
> > > + int *out = addr;
> > > + struct rte_rawdev *rawdev = NULL;
> > > + char rawdev_name[RTE_RAWDEV_NAME_MAX_LEN];
> > > + char *c1 = NULL, *c2 = NULL;
> > > + int port = IFPGA_BUS_DEV_PORT_MAX;
> > > + char str_port[8];
> > > + int str_port_len = 0;
> > > + int ret;
> > > +
> > > + memset(str_port, 0, 8);
> > > + c1 = strchr(name, '|');
> > > + if (c1 != NULL) {
> > > + str_port_len = c1-name;
> > > + c2 = c1+1;
> > > + }
> > > +
> > > + if (str_port_len < 8 &&
> > > + str_port_len > 0) {
> > > + memcpy(str_port, name, str_port_len);
> > > + ret = sscanf(str_port, "%d", &port);
> > > + if (ret == -1)
> > > + return 0;
> > > + }
> > > +
> > > + memset(rawdev_name, 0, sizeof(rawdev_name));
> > > + snprintf(rawdev_name, RTE_RAWDEV_NAME_MAX_LEN, "IFPGA:%s",
> > c2);
> > > + rawdev = rte_rawdev_pmd_get_named_dev(rawdev_name);
> > > +
> > > + if ((port < IFPGA_BUS_DEV_PORT_MAX) &&
> > > + rawdev &&
> > > + (addr != NULL))
> > > + *out = port;
> > > +
> > > + if ((port < IFPGA_BUS_DEV_PORT_MAX) &&
> > > + rawdev)
> > > + return 0;
> > > + else
> > > + return 1;
> > > +}
> > > +
> > > +static struct rte_bus rte_ifpga_bus = {
> > > + .scan = ifpga_scan,
> > > + .probe = ifpga_probe,
> > > + .find_device = ifpga_find_device,
> > > + .plug = ifpga_plug,
> > > + .unplug = ifpga_unplug,
> > > + .parse = ifpga_parse,
> > > +};
> > > +
> > > +RTE_REGISTER_BUS(IFPGA_BUS_NAME, rte_ifpga_bus);
> > > +
> > > +RTE_INIT(ifpga_init_log)
> > > +{
> > > + ifpga_bus_logtype = rte_log_register("bus.ifpga");
> > > + if (ifpga_bus_logtype >= 0)
> > > + rte_log_set_level(ifpga_bus_logtype, RTE_LOG_NOTICE); }
> > > diff --git a/drivers/bus/ifpga/ifpga_common.c
> > > b/drivers/bus/ifpga/ifpga_common.c
> > > new file mode 100644
> > > index 0000000..78e2eae
> > > --- /dev/null
> > > +++ b/drivers/bus/ifpga/ifpga_common.c
> > > @@ -0,0 +1,88 @@
> > > +/* SPDX-License-Identifier: BSD-3-Clause
> > > + * Copyright(c) 2010-2018 Intel Corporation */
> > > +
> > > +#include <string.h>
> > > +#include <inttypes.h>
> > > +#include <stdint.h>
> > > +#include <stdlib.h>
> > > +#include <stdio.h>
> > > +#include <sys/queue.h>
> > > +#include <sys/mman.h>
> > > +#include <sys/types.h>
> > > +#include <unistd.h>
> > > +#include <fcntl.h>
> > > +
> > > +#include <rte_errno.h>
> > > +#include <rte_bus.h>
> > > +#include <rte_per_lcore.h>
> > > +#include <rte_memory.h>
> > > +#include <rte_memzone.h>
> > > +#include <rte_eal.h>
> > > +#include <rte_common.h>
> > > +
> > > +#include <rte_devargs.h>
> > > +#include <rte_kvargs.h>
> > > +#include <rte_alarm.h>
> > > +
> > > +#include "rte_bus_ifpga.h"
> > > +#include "ifpga_logs.h"
> > > +#include "ifpga_common.h"
> > > +
> > > +int rte_ifpga_get_string_arg(const char *key __rte_unused,
> > > + const char *value, void *extra_args) {
> > > + if (!value || !extra_args)
> > > + return -EINVAL;
> > > +
> > > + *(char **)extra_args = strdup(value);
> > > +
> > > + if (!*(char **)extra_args)
> > > + return -ENOMEM;
> > > +
> > > + return 0;
> > > +}
> > > +int rte_ifpga_get_integer32_arg(const char *key __rte_unused,
> > > + const char *value, void *extra_args) {
> > > + if (!value || !extra_args)
> > > + return -EINVAL;
> > > +
> > > + *(int *)extra_args = strtoull(value, NULL, 0);
> > > +
> > > + return 0;
> > > +}
> > > +int ifpga_get_integer64_arg(const char *key __rte_unused,
> > > + const char *value, void *extra_args) {
> > > + if (!value || !extra_args)
> > > + return -EINVAL;
> > > +
> > > + *(uint64_t *)extra_args = strtoull(value, NULL, 0);
> > > +
> > > + return 0;
> > > +}
> > > +int ifpga_get_unsigned_long(const char *str, int base) {
> > > + unsigned long num;
> > > + char *end = NULL;
> > > +
> > > + errno = 0;
> > > +
> > > + num = strtoul(str, &end, base);
> > > + if ((str[0] == '\0') || (end == NULL) || (*end != '\0') || (errno != 0))
> > > + return -1;
> > > +
> > > + return num;
> > > +}
> > > +
> > > +int ifpga_afu_id_cmp(const struct rte_afu_id *afu_id0,
> > > + const struct rte_afu_id *afu_id1)
> > > +{
> > > + if ((afu_id0->uuid.uuid_low == afu_id1->uuid.uuid_low) &&
> > > + (afu_id0->uuid.uuid_high == afu_id1->uuid.uuid_high) &&
> > > + (afu_id0->port == afu_id1->port)) {
> > > + return 0;
> > > + } else
> > > + return 1;
> > > +}
> > > diff --git a/drivers/bus/ifpga/ifpga_common.h
> > > b/drivers/bus/ifpga/ifpga_common.h
> > > new file mode 100644
> > > index 0000000..f9254b9
> > > --- /dev/null
> > > +++ b/drivers/bus/ifpga/ifpga_common.h
> > > @@ -0,0 +1,18 @@
> > > +/* SPDX-License-Identifier: BSD-3-Clause
> > > + * Copyright(c) 2010-2018 Intel Corporation */
> > > +
> > > +#ifndef _IFPGA_COMMON_H_
> > > +#define _IFPGA_COMMON_H_
> > > +
> > > +int rte_ifpga_get_string_arg(const char *key __rte_unused,
> > > + const char *value, void *extra_args); int
> > > +rte_ifpga_get_integer32_arg(const char *key __rte_unused,
> > > + const char *value, void *extra_args); int
> > > +ifpga_get_integer64_arg(const char *key __rte_unused,
> > > + const char *value, void *extra_args); int
> > > +ifpga_get_unsigned_long(const char *str, int base); int
> > > +ifpga_afu_id_cmp(const struct rte_afu_id *afu_id0,
> > > + const struct rte_afu_id *afu_id1);
> > > +
> > > +#endif /* _IFPGA_COMMON_H_ */
> > > diff --git a/drivers/bus/ifpga/ifpga_logs.h
> > > b/drivers/bus/ifpga/ifpga_logs.h new file mode 100644 index
> > > 0000000..873e0a4
> > > --- /dev/null
> > > +++ b/drivers/bus/ifpga/ifpga_logs.h
> > > @@ -0,0 +1,31 @@
> > > +/* SPDX-License-Identifier: BSD-3-Clause
> > > + * Copyright(c) 2010-2018 Intel Corporation */
> > > +
> > > +#ifndef _IFPGA_LOGS_H_
> > > +#define _IFPGA_LOGS_H_
> > > +
> > > +#include <rte_log.h>
> > > +
> > > +extern int ifpga_bus_logtype;
> > > +
> > > +#define IFPGA_LOG(level, fmt, args...) \
> > > + rte_log(RTE_LOG_ ## level, ifpga_bus_logtype, "%s(): " fmt "\n", \
> > > + __func__, ##args)
> > > +
> > > +#define IFPGA_BUS_LOG(level, fmt, args...) \
> > > + rte_log(RTE_LOG_ ## level, ifpga_bus_logtype, "%s(): " fmt "\n", \
> > > + __func__, ##args)
> > > +
> > > +#define IFPGA_BUS_FUNC_TRACE() IFPGA_BUS_LOG(DEBUG, ">>")
> > > +
> > > +#define IFPGA_BUS_DEBUG(fmt, args...) \
> > > + IFPGA_BUS_LOG(DEBUG, fmt, ## args) #define IFPGA_BUS_INFO(fmt,
> > > +args...) \
> > > + IFPGA_BUS_LOG(INFO, fmt, ## args)
> > > +#define IFPGA_BUS_ERR(fmt, args...) \
> > > + IFPGA_BUS_LOG(ERR, fmt, ## args)
> > > +#define IFPGA_BUS_WARN(fmt, args...) \
> > > + IFPGA_BUS_LOG(WARNING, fmt, ## args)
> > > +
> > > +#endif /* _IFPGA_BUS_LOGS_H_ */
> > > diff --git a/drivers/bus/ifpga/meson.build
> > > b/drivers/bus/ifpga/meson.build new file mode 100644 index
> > > 0000000..c9b08c8
> > > --- /dev/null
> > > +++ b/drivers/bus/ifpga/meson.build
> > > @@ -0,0 +1,8 @@
> > > +# SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2010-2018
> > > +Intel Corporation
> > > +
> > > +deps += ['pci', 'kvargs', 'rawdev']
> > > +install_headers('rte_bus_ifpga.h')
> > > +sources = files('ifpga_common.c', 'ifpga_bus.c')
> > > +
> > > +allow_experimental_apis = true
> > > diff --git a/drivers/bus/ifpga/rte_bus_ifpga.h
> > > b/drivers/bus/ifpga/rte_bus_ifpga.h
> > > new file mode 100644
> > > index 0000000..5c559e1
> > > --- /dev/null
> > > +++ b/drivers/bus/ifpga/rte_bus_ifpga.h
> > > @@ -0,0 +1,160 @@
> > > +/* SPDX-License-Identifier: BSD-3-Clause
> > > + * Copyright(c) 2010-2018 Intel Corporation */
> > > +
> > > +#ifndef _RTE_BUS_IFPGA_H_
> > > +#define _RTE_BUS_IFPGA_H_
> > > +
> > > +/**
> > > + * @file
> > > + *
> > > + * RTE Intel FPGA Bus Interface
> > > + */
> > > +
> > > +#ifdef __cplusplus
> > > +extern "C" {
> > > +#endif
> > > +
> > > +#include <rte_bus.h>
> > > +#include <rte_pci.h>
> > > +
> > > +/** Name of Intel FPGA Bus */
> > > +#define IFPGA_BUS_NAME ifpga
> > > +
> > > +/* Forward declarations */
> > > +struct rte_afu_device;
> > > +
> > > +/** List of Intel AFU devices */
> > > +TAILQ_HEAD(afu_device_list, rte_afu_device);
> > > +/** Double linked list of AFU device drivers. */
> > > +TAILQ_HEAD(afu_driver_list, rte_afu_driver);
> > > +
> > > +#define IFPGA_BUS_BITSTREAM_PATH_MAX_LEN 256
> > > +
> > > +struct rte_afu_uuid {
> > > + uint64_t uuid_low;
> > > + uint64_t uuid_high;
> > > +} __attribute__ ((packed));
> > > +
> > > +#define IFPGA_BUS_DEV_PORT_MAX 4
> > > +
> > > +/**
> > > + * A structure describing an ID for a AFU driver. Each driver
> > > +provides a
> > > + * table of these IDs for each device that it supports.
> > > + */
> > > +struct rte_afu_id {
> > > + struct rte_afu_uuid uuid;
> > > + int port; /**< port number */
> > > +} __attribute__ ((packed));
> > > +
> > > +/**
> > > + * A structure PR (Partial Reconfiguration) configuration AFU driver.
> > > + */
> > > +
> > > +struct rte_afu_pr_conf {
> > > + struct rte_afu_id afu_id;
> > > + int pr_enable;
> > > + char bs_path[IFPGA_BUS_BITSTREAM_PATH_MAX_LEN];
> > > +};
> > > +
> > > +#define AFU_PRI_STR_SIZE (PCI_PRI_STR_SIZE + 8)
> > > +
> > > +/**
> > > + * A structure describing a fpga device.
> > > + */
> > > +struct rte_ifpga_device {
> > > + TAILQ_ENTRY(rte_ifpga_device) next; /**< Next in device list. */
> > > + struct rte_rawdev *rdev;
> > > + struct afu_device_list afu_list; /**< List of AFU devices */ };
> > > +
> > > +/**
> > > + * A structure describing a AFU device.
> > > + */
> > > +struct rte_afu_device {
> > > + TAILQ_ENTRY(rte_afu_device) next; /**< Next in device list. */
> > > + struct rte_device device; /**< Inherit core device */
> > > + struct rte_rawdev *rawdev; /**< Point Rawdev */
> > > + struct rte_ifpga_device *ifpga_dev; /**< Point ifpga device */
> > > + struct rte_afu_id id; /**< AFU id within FPGA. */
> > > + uint32_t num_region; /**< number of regions found */
> > > + struct rte_mem_resource mem_resource[PCI_MAX_RESOURCE];
> > > + /**< AFU Memory Resource
> > */
> > > + struct rte_intr_handle intr_handle; /**< Interrupt handle */
> > > + struct rte_afu_driver *driver; /**< Associated driver */
> > > + char path[IFPGA_BUS_BITSTREAM_PATH_MAX_LEN];
> > > +} __attribute__ ((packed));
> > > +
> > > +/**
> > > + * @internal
> > > + * Helper macro for drivers that need to convert to struct rte_afu_device.
> > > + */
> > > +#define RTE_DEV_TO_AFU(ptr) \
> > > + container_of(ptr, struct rte_afu_device, device)
> > > +
> > > +#define RTE_DRV_TO_AFU_CONST(ptr) \
> > > + container_of(ptr, const struct rte_afu_driver, driver)
> > > +
> > > +/**
> > > + * Initialisation function for the driver called during FPGA BUS probing.
> >
> >
> > typo: Initialization
>
> Fixed.
>
> > > + */
> > > +typedef int (afu_probe_t)(struct rte_afu_device *);
> > > +
> > > +/**
> > > + * Uninitialisation function for the driver called during hotplugging.
> >
> > typo: Unintialization
>
> Fixed.
>
> > > + */
> > > +typedef int (afu_remove_t)(struct rte_afu_device *);
> > > +
> > > +/**
> > > + * A structure describing a AFU device.
> > > + */
> > > +struct rte_afu_driver {
> > > + TAILQ_ENTRY(rte_afu_driver) next; /**< Next afu driver. */
> > > + struct rte_driver driver; /**< Inherit core driver. */
> > > + afu_probe_t *probe; /**< Device Probe function.
> > > */
> > > + afu_remove_t *remove; /**< Device Remove
> > > function. */
> > > + const struct rte_afu_uuid *id_table; /**< AFU uuid within FPGA. */
> > > + uint32_t drv_flags; /**< Flags contolling handling of device. */
> >
> > Typo Controlling
> > And what is "flags controlling handling of device?"
>
> This variable is not used, so I have removed it.
>
> > > +};
> > > +
> > > +static inline const char *
> > > +rte_ifpga_device_name(const struct rte_afu_device *afu) {
> > > + if (afu && afu->device.name)
> > > + return afu->device.name;
> > > + return NULL;
> > > +}
> > > +
> > > +/**
> > > + * Register a ifpga afu device driver.
> > > + *
> > > + * @param driver
> > > + * A pointer to a rte_afu_driver structure describing the driver
> > > + * to be registered.
> > > + */
> > > +void rte_ifpga_driver_register(struct rte_afu_driver *driver);
> > > +
> > > +/**
> > > + * Unregister a ifpga afu device driver.
> > > + *
> > > + * @param driver
> > > + * A pointer to a rte_afu_driver structure describing the driver
> > > + * to be unregistered.
> > > + */
> > > +void rte_ifpga_driver_unregister(struct rte_afu_driver *driver);
> > > +
> > > +#define RTE_PMD_REGISTER_AFU(nm, afudrv)\ RTE_INIT(afudrvinitfn_
> > > +##afudrv);\ static const char *afudrvinit_ ## nm ## _alias;\ static
> > > +void afudrvinitfn_ ##afudrv(void)\ {\
> > > + (afudrv).driver.name = RTE_STR(nm);\
> > > + (afudrv).driver.alias = afudrvinit_ ## nm ## _alias;\
> > > + rte_ifpga_driver_register(&afudrv);\
> > > +} \
> > > +RTE_PMD_EXPORT_NAME(nm, __COUNTER__)
> > > +
> > > +#define RTE_PMD_REGISTER_AFU_ALIAS(nm, alias)\ static const char
> > > +*afudrvinit_ ## nm ## _alias = RTE_STR(alias)
> > > +
> > > +#endif /* _RTE_BUS_IFPGA_H_ */
> > > diff --git a/drivers/bus/ifpga/rte_bus_ifpga_version.map
> > > b/drivers/bus/ifpga/rte_bus_ifpga_version.map
> > > new file mode 100644
> > > index 0000000..a027979
> > > --- /dev/null
> > > +++ b/drivers/bus/ifpga/rte_bus_ifpga_version.map
> > > @@ -0,0 +1,10 @@
> > > +DPDK_18.05 {
> > > + global:
> > > +
> > > + rte_ifpga_get_integer32_arg;
> > > + rte_ifpga_get_string_arg;
> > > + rte_ifpga_driver_register;
> > > + rte_ifpga_driver_unregister;
> > > +
> > > + local: *;
> > > +};
> > > diff --git a/drivers/bus/meson.build b/drivers/bus/meson.build index
> > > 58dfbe2..52c755d 100644
> > > --- a/drivers/bus/meson.build
> > > +++ b/drivers/bus/meson.build
> > > @@ -1,7 +1,7 @@
> > > # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2017 Intel
> > > Corporation
> > >
> > > -drivers = ['dpaa', 'fslmc', 'pci', 'vdev']
> > > +drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'vdev']
> > > std_deps = ['eal']
> > > config_flag_fmt = 'RTE_LIBRTE_@0@_BUS'
> > > driver_name_fmt = 'rte_bus_@0@'
> > > diff --git a/mk/rte.app.mk b/mk/rte.app.mk index 26f3563..3861e1a
> > > 100644
> > > --- a/mk/rte.app.mk
> > > +++ b/mk/rte.app.mk
> > > @@ -255,6 +255,7 @@ ifeq
> > > ($(CONFIG_RTE_EAL_VFIO)$(CONFIG_RTE_LIBRTE_FSLMC_BUS),yy)
> > > _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_CMDIF_RAWDEV) +=
> > > -lrte_pmd_dpaa2_cmdif
> > > _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV) +=
> > > -lrte_pmd_dpaa2_qdma endif # CONFIG_RTE_LIBRTE_FSLMC_BUS
> > > +_LDLIBS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += -lrte_bus_ifpga
> > > endif # CONFIG_RTE_LIBRTE_RAWDEV
> > >
> > >
> > > --
> > > 1.8.3.1
> >
> > Regards
> > Qi
Hi Qi,
I miss one comment, so I fix it in this email.
> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Xu, Rosen
> Sent: Thursday, May 10, 2018 21:29
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; dev@dpdk.org;
> thomas@monjalon.net
> Cc: Zhang, Roy Fan <roy.fan.zhang@intel.com>; Doherty, Declan
> <declan.doherty@intel.com>; Richardson, Bruce
> <bruce.richardson@intel.com>; shreyansh.jain@nxp.com; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Ananyev, Konstantin
> <konstantin.ananyev@intel.com>; Zhang, Tianfei <tianfei.zhang@intel.com>;
> Liu, Song <song.liu@intel.com>; Wu, Hao <hao.wu@intel.com>;
> gaetan.rivet@6wind.com
> Subject: Re: [dpdk-dev] [PATCH v10 1/3] bus/ifpga: Add Intel FPGA BUS
> Library
>
> Hi Qi,
>
> > -----Original Message-----
> > From: Zhang, Qi Z
> > Sent: Thursday, May 10, 2018 20:27
> > To: Xu, Rosen <rosen.xu@intel.com>; dev@dpdk.org;
> thomas@monjalon.net
> > Cc: Xu, Rosen <rosen.xu@intel.com>; Zhang, Roy Fan
> > <roy.fan.zhang@intel.com>; Doherty, Declan <declan.doherty@intel.com>;
> > Richardson, Bruce <bruce.richardson@intel.com>;
> > shreyansh.jain@nxp.com; Yigit, Ferruh <ferruh.yigit@intel.com>;
> > Ananyev, Konstantin <konstantin.ananyev@intel.com>; Zhang, Tianfei
> > <tianfei.zhang@intel.com>; Liu, Song <song.liu@intel.com>; Wu, Hao
> > <hao.wu@intel.com>; gaetan.rivet@6wind.com
> > Subject: RE: [dpdk-dev] [PATCH v10 1/3] bus/ifpga: Add Intel FPGA BUS
> > Library
> >
> > Hi Rosen:
> >
> > > -----Original Message-----
> > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Xu, Rosen
> > > Sent: Wednesday, May 9, 2018 3:43 PM
> > > To: dev@dpdk.org; thomas@monjalon.net
> > > Cc: Xu, Rosen <rosen.xu@intel.com>; Zhang, Roy Fan
> > > <roy.fan.zhang@intel.com>; Doherty, Declan
> > > <declan.doherty@intel.com>; Richardson, Bruce
> > > <bruce.richardson@intel.com>; shreyansh.jain@nxp.com; Yigit, Ferruh
> > > <ferruh.yigit@intel.com>; Ananyev, Konstantin
> > > <konstantin.ananyev@intel.com>; Zhang, Tianfei
> > > <tianfei.zhang@intel.com>; Liu, Song <song.liu@intel.com>; Wu, Hao
> > > <hao.wu@intel.com>; gaetan.rivet@6wind.com
> > > Subject: [dpdk-dev] [PATCH v10 1/3] bus/ifpga: Add Intel FPGA BUS
> > > Library
> > >
> > > From: Rosen Xu <rosen.xu@intel.com>
> > >
> > > Defined FPGA-BUS for Acceleration Drivers of AFUs
> > >
> > > 1. FPGA PCI Scan (1st Scan) follows DPDK UIO/VFIO PCI Scan Process,
> > > probe Intel FPGA Rawdev Driver, it will be covered in following patches.
> > >
> > > 2. AFU Scan(2nd Scan) bind DPDK driver to FPGA Partial-Bitstream.
> > > This scan is trigged by hotplug of IFPGA Rawdev probe, in this scan
> > > the AFUs will be created and their drivers are also probed.
> > >
> > > This patch will introduce rte_afu_device which describe the AFU
> > > device listed in the FPGA-BUS.
> > >
> > > Signed-off-by: Rosen Xu <rosen.xu@intel.com>
> > > Signed-off-by: Tianfei Zhang <tianfei.zhang@intel.com>
> > > ---
> > > MAINTAINERS | 4 +
> > > config/common_base | 5 +
> > > doc/guides/rel_notes/release_18_05.rst | 5 +
> > > drivers/bus/Makefile | 1 +
> > > drivers/bus/ifpga/Makefile | 32 ++
> > > drivers/bus/ifpga/ifpga_bus.c | 501
> > > ++++++++++++++++++++++++++++
> > > drivers/bus/ifpga/ifpga_common.c | 88 +++++
> > > drivers/bus/ifpga/ifpga_common.h | 18 +
> > > drivers/bus/ifpga/ifpga_logs.h | 31 ++
> > > drivers/bus/ifpga/meson.build | 8 +
> > > drivers/bus/ifpga/rte_bus_ifpga.h | 160 +++++++++
> > > drivers/bus/ifpga/rte_bus_ifpga_version.map | 10 +
> > > drivers/bus/meson.build | 2 +-
> > > mk/rte.app.mk | 1 +
> > > 14 files changed, 865 insertions(+), 1 deletion(-) create mode
> > > 100644 drivers/bus/ifpga/Makefile create mode 100644
> > > drivers/bus/ifpga/ifpga_bus.c create mode 100644
> > > drivers/bus/ifpga/ifpga_common.c create mode 100644
> > > drivers/bus/ifpga/ifpga_common.h create mode 100644
> > > drivers/bus/ifpga/ifpga_logs.h create mode 100644
> > > drivers/bus/ifpga/meson.build create mode 100644
> > > drivers/bus/ifpga/rte_bus_ifpga.h create mode 100644
> > > drivers/bus/ifpga/rte_bus_ifpga_version.map
> > >
> > > diff --git a/MAINTAINERS b/MAINTAINERS index 7105920..fa0c5b1 100644
> > > --- a/MAINTAINERS
> > > +++ b/MAINTAINERS
> > > @@ -383,6 +383,10 @@ F: drivers/mempool/bucket/ Bus Drivers
> > > -----------
> > >
> > > +Intel FPGA buses
> > > +M: Rosen Xu <rosen.xu@intel.com>
> > > +F: drivers/bus/ifpga/
> > > +
> > > NXP buses
> > > M: Hemant Agrawal <hemant.agrawal@nxp.com>
> > > M: Shreyansh Jain <shreyansh.jain@nxp.com> diff --git
> > > a/config/common_base b/config/common_base index 0d181ac..1440316
> > > 100644
> > > --- a/config/common_base
> > > +++ b/config/common_base
> > > @@ -139,6 +139,11 @@
> > > CONFIG_RTE_ETHDEV_PROFILE_ITT_WASTED_RX_ITERATIONS=n
> > > CONFIG_RTE_ETHDEV_TX_PREPARE_NOOP=n
> > >
> > > #
> > > +# Compile the Intel FPGA bus
> > > +#
> > > +CONFIG_RTE_LIBRTE_IFPGA_BUS=y
> > > +
> > > +#
> > > # Compile PCI bus driver
> > > #
> > > CONFIG_RTE_LIBRTE_PCI_BUS=y
> > > diff --git a/doc/guides/rel_notes/release_18_05.rst
> > > b/doc/guides/rel_notes/release_18_05.rst
> > > index 7187348..265950a 100644
> > > --- a/doc/guides/rel_notes/release_18_05.rst
> > > +++ b/doc/guides/rel_notes/release_18_05.rst
> > > @@ -183,6 +183,11 @@ New Features
> > > stats/xstats on shared memory from secondary process, and also
> > > pdump packets on
> > > those virtual devices.
> > >
> > > +* **Added Ifpga Bus, a generic Intel FPGA Bus library.**
> > > +
> > > + The Ifpga Bus library provides support for integrating any Intel
> > > + FPGA device with the DPDK framework. It provides Intel FPGA
> > > + Partial Bit Stream AFU(Accelerated Function Unit) scan and drivers
> prove.
> > >
> > > API Changes
> > > -----------
> > > diff --git a/drivers/bus/Makefile b/drivers/bus/Makefile index
> > > c251b65..ef7f247 100644
> > > --- a/drivers/bus/Makefile
> > > +++ b/drivers/bus/Makefile
> > > @@ -7,6 +7,7 @@ DIRS-$(CONFIG_RTE_LIBRTE_DPAA_BUS) += dpaa ifeq
> > > ($(CONFIG_RTE_EAL_VFIO),y)
> > > DIRS-$(CONFIG_RTE_LIBRTE_FSLMC_BUS) += fslmc endif
> > > +DIRS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += ifpga
> > > DIRS-$(CONFIG_RTE_LIBRTE_PCI_BUS) += pci
> > > DIRS-$(CONFIG_RTE_LIBRTE_VDEV_BUS) += vdev
> > >
> > > diff --git a/drivers/bus/ifpga/Makefile b/drivers/bus/ifpga/Makefile
> > > new file mode 100644 index 0000000..3ff3bdb
> > > --- /dev/null
> > > +++ b/drivers/bus/ifpga/Makefile
> > > @@ -0,0 +1,32 @@
> > > +# SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2018 Intel
> > > +Corporation
> > > +
> > > +include $(RTE_SDK)/mk/rte.vars.mk
> > > +
> > > +#
> > > +# library name
> > > +#
> > > +LIB = librte_bus_ifpga.a
> > > +
> > > +CFLAGS += -DALLOW_EXPERIMENTAL_API
> > > +CFLAGS += -O3
> > > +CFLAGS += $(WERROR_FLAGS)
> > > +LDLIBS += -lrte_eal
> > > +LDLIBS += -lrte_rawdev
> > > +LDLIBS += -lrte_kvargs
> > > +
> > > +# versioning export map
> > > +EXPORT_MAP := rte_bus_ifpga_version.map
> > > +
> > > +# library version
> > > +LIBABIVER := 1
> > > +
> > > +SRCS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += ifpga_bus.c
> > > +SRCS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += ifpga_common.c
> > > +
> > > +#
> > > +# Export include files
> > > +#
> > > +SYMLINK-$(CONFIG_RTE_LIBRTE_IFPGA_BUS)-include += rte_bus_ifpga.h
> > > +
> > > +include $(RTE_SDK)/mk/rte.lib.mk
> > > diff --git a/drivers/bus/ifpga/ifpga_bus.c
> > > b/drivers/bus/ifpga/ifpga_bus.c new file mode 100644 index
> > > 0000000..e144c01
> > > --- /dev/null
> > > +++ b/drivers/bus/ifpga/ifpga_bus.c
> > > @@ -0,0 +1,501 @@
> > > +/* SPDX-License-Identifier: BSD-3-Clause
> > > + * Copyright(c) 2010-2018 Intel Corporation */
> > > +
> > > +#include <string.h>
> > > +#include <inttypes.h>
> > > +#include <stdint.h>
> > > +#include <stdlib.h>
> > > +#include <stdio.h>
> > > +#include <sys/queue.h>
> > > +#include <sys/mman.h>
> > > +#include <sys/types.h>
> > > +#include <unistd.h>
> > > +#include <fcntl.h>
> > > +
> > > +#include <rte_errno.h>
> > > +#include <rte_bus.h>
> > > +#include <rte_per_lcore.h>
> > > +#include <rte_memory.h>
> > > +#include <rte_memzone.h>
> > > +#include <rte_eal.h>
> > > +#include <rte_common.h>
> > > +
> > > +#include <rte_devargs.h>
> > > +#include <rte_kvargs.h>
> > > +#include <rte_alarm.h>
> > > +
> > > +#include "rte_rawdev.h"
> > > +#include "rte_rawdev_pmd.h"
> > > +#include "rte_bus_ifpga.h"
> > > +#include "ifpga_logs.h"
> > > +#include "ifpga_common.h"
> > > +
> > > +int ifpga_bus_logtype;
> > > +
> > > +/* Forward declaration to access Intel FPGA bus
> > > + * on which iFPGA devices are connected */ static struct rte_bus
> > > +rte_ifpga_bus;
> > > +
> > > +/** Double linked list of IFPGA device. */
> > > +TAILQ_HEAD(ifpga_device_list, rte_ifpga_device);
> > > +
> > > +static struct ifpga_device_list ifpga_device_list =
> > > + TAILQ_HEAD_INITIALIZER(ifpga_device_list);
> > > +static struct afu_driver_list afu_driver_list =
> > > + TAILQ_HEAD_INITIALIZER(afu_driver_list);
> > > +
> > > +
> > > +/* register a ifpga bus based driver */ void
> > > +rte_ifpga_driver_register(struct rte_afu_driver *driver) {
> > > + RTE_VERIFY(driver);
> > > +
> > > + TAILQ_INSERT_TAIL(&afu_driver_list, driver, next); }
> > > +
> > > +/* un-register a fpga bus based driver */ void
> > > +rte_ifpga_driver_unregister(struct rte_afu_driver *driver) {
> > > + TAILQ_REMOVE(&afu_driver_list, driver, next); }
> > > +
> > > +static struct rte_ifpga_device *
> > > +ifpga_find_ifpga_dev(const struct rte_rawdev *rdev) {
> > > + struct rte_ifpga_device *ifpga_dev = NULL;
> > > +
> > > + TAILQ_FOREACH(ifpga_dev, &ifpga_device_list, next) {
> > > + if (rdev &&
> > > + ifpga_dev->rdev &&
> > > + ifpga_dev->rdev == rdev)
> > > + return ifpga_dev;
> > > + }
> > > + return NULL;
> > > +}
> > > +
> > > +static struct rte_afu_device *
> > > +ifpga_find_afu_dev(const struct rte_ifpga_device *ifpga_dev,
> > > + const struct rte_afu_id *afu_id)
> > > +{
> > > + struct rte_afu_device *afu_dev = NULL;
> > > +
> > > + TAILQ_FOREACH(afu_dev, &ifpga_dev->afu_list, next) {
> > > + if (!ifpga_afu_id_cmp(&afu_dev->id, afu_id))
> > > + return afu_dev;
> > > + }
> > > + return NULL;
> > > +}
> > > +
> > > +static const char * const valid_args[] = {
> > > +#define IFPGA_ARG_NAME "ifpga"
> > > + IFPGA_ARG_NAME,
> > > +#define IFPGA_ARG_PORT "port"
> > > + IFPGA_ARG_PORT,
> > > +#define IFPGA_AFU_BTS "afu_bts"
> > > + IFPGA_AFU_BTS,
> > > + NULL
> > > +};
> > > +
> > > +/*
> > > + * Scan the content of the FPGA bus, and the devices in the devices
> > > + * list
> > > + */
> > > +static struct rte_afu_device *
> > > +ifpga_scan_one(struct rte_devargs *devargs,
> > > + struct rte_ifpga_device *ifpga_dev)
> >
> > usually ifpag_dev should be the first parameter here, and devargs
> > follows
>
> Fixed.
>
> > > +{
> > > + struct rte_kvargs *kvlist = NULL;
> > > + struct rte_rawdev *rawdev = NULL;
> > > + struct rte_afu_device *afu_dev = NULL;
> > > + struct rte_afu_pr_conf afu_pr_conf;
> > > + int ret = 0;
> > > + char *path = NULL;
> > > +
> > > + memset(&afu_pr_conf, 0, sizeof(struct rte_afu_pr_conf));
> > > +
> > > + kvlist = rte_kvargs_parse(devargs->args, valid_args);
> > > + if (!kvlist) {
> > > + IFPGA_BUS_ERR("error when parsing param");
> > > + goto end;
> > > + }
> > > +
> > > + if (rte_kvargs_count(kvlist, IFPGA_ARG_PORT) == 1) {
> > > + if (rte_kvargs_process(kvlist, IFPGA_ARG_PORT,
> > > + &rte_ifpga_get_integer32_arg, &afu_pr_conf.afu_id.port) < 0)
> > {
> > > + IFPGA_BUS_ERR("error to parse %s",
> > > + IFPGA_ARG_PORT);
> > > + goto end;
> > > + }
> > > + } else {
> > > + IFPGA_BUS_ERR("arg %s is mandatory for ifpga bus",
> > > + IFPGA_ARG_PORT);
> > > + goto end;
> > > + }
> > > +
> > > + if (rte_kvargs_count(kvlist, IFPGA_AFU_BTS) == 1) {
> > > + if (rte_kvargs_process(kvlist, IFPGA_AFU_BTS,
> > > + &rte_ifpga_get_string_arg, &path) < 0) {
> > > + IFPGA_BUS_ERR("Failed to parse %s",
> > > + IFPGA_AFU_BTS);
> > > + goto end;
> > > + }
> > > + } else {
> > > + IFPGA_BUS_ERR("arg %s is mandatory for ifpga bus",
> > > + IFPGA_AFU_BTS);
> > > + goto end;
> > > + }
> > > +
> > > + afu_pr_conf.afu_id.uuid.uuid_low = 0;
> > > + afu_pr_conf.afu_id.uuid.uuid_high = 0;
> > > + afu_pr_conf.pr_enable = path?1:0;
> > > +
> > > + rawdev = ifpga_dev->rdev;
> > > + if (ifpga_find_afu_dev(ifpga_dev, &afu_pr_conf.afu_id))
> > > + goto end;
> > > +
> > > + afu_dev = calloc(1, sizeof(*afu_dev));
> > > + if (!afu_dev)
> > > + goto end;
> > > +
> > > + afu_dev->device.devargs = devargs;
> > > + afu_dev->device.numa_node = SOCKET_ID_ANY;
> > > + afu_dev->device.name = devargs->name;
> > > + afu_dev->rawdev = rawdev;
> > > + afu_dev->id.uuid.uuid_low = 0;
> > > + afu_dev->id.uuid.uuid_high = 0;
> > > + afu_dev->id.port = afu_pr_conf.afu_id.port;
> > > + afu_dev->ifpga_dev = ifpga_dev;
> > > +
> > > + if (rawdev->dev_ops && rawdev->dev_ops->dev_info_get)
> > > + rawdev->dev_ops->dev_info_get(rawdev, afu_dev);
> > > +
> > > + if (rawdev->dev_ops &&
> > > + rawdev->dev_ops->dev_start &&
> > > + rawdev->dev_ops->dev_start(rawdev))
> > > + goto free_dev;
> > > +
> > > + strncpy(afu_pr_conf.bs_path, path, sizeof(afu_pr_conf.bs_path));
> > > + if (rawdev->dev_ops->firmware_load &&
> > > + rawdev->dev_ops->firmware_load(rawdev,
> > > + &afu_pr_conf)){
> > > + IFPGA_BUS_ERR("firmware load error %d\n", ret);
> > > + goto free_dev;
> > > + }
> > > + afu_dev->id.uuid.uuid_low = afu_pr_conf.afu_id.uuid.uuid_low;
> > > + afu_dev->id.uuid.uuid_high = afu_pr_conf.afu_id.uuid.uuid_high;
> > > +
> > > + return afu_dev;
> > > +
> > > +free_dev:
> > > + free(afu_dev);
> > > +end:
> > > + if (kvlist)
> > > + rte_kvargs_free(kvlist);
> > > + if (path)
> > > + free(path);
> > > +
> > > + return NULL;
> > > +}
> > > +
> > > +/*
> > > + * Scan the content of the FPGA bus, and the devices in the devices
> > > + * list
> > > + */
> > > +static int
> > > +ifpga_scan(void)
> > > +{
> > > + struct rte_ifpga_device *ifpga_dev;
> > > + struct rte_devargs *devargs;
> > > + struct rte_kvargs *kvlist = NULL;
> > > + struct rte_rawdev *rawdev = NULL;
> > > + char *name = NULL;
> > > + char name1[RTE_RAWDEV_NAME_MAX_LEN];
> > > + struct rte_afu_device *afu_dev = NULL;
> > > +
> > > + /* for FPGA devices we scan the devargs_list populated via cmdline
> > > +*/
> >
> > I didn't see "--ifpga" is supported by cmdline, either you need to add
> > corresponding parser or correct the comment here.
>
> To be honestly, ifpga_scan() is called by hotplug, lib_rte hotplug function will
> construct it.
>
> > > + RTE_EAL_DEVARGS_FOREACH("ifpga", devargs) {
> >
> >
> > IFPGA_BUS_NAME to replace "ifpga" here.
IFPGA_BUS_NAME is used in some Macros, so I use IFPGA_ARG_NAME.
> > > + if (devargs->bus != &rte_ifpga_bus)
> > > + continue;
> > > +
> > > + kvlist = rte_kvargs_parse(devargs->args, valid_args);
> > > + if (!kvlist) {
> > > + IFPGA_BUS_ERR("error when parsing param");
> > > + goto end;
> > > + }
> > > +
> > > + if (rte_kvargs_count(kvlist, IFPGA_ARG_NAME) == 1) {
> > > + if (rte_kvargs_process(kvlist, IFPGA_ARG_NAME,
> > > + &rte_ifpga_get_string_arg, &name) < 0) {
> > > + IFPGA_BUS_ERR("error to parse %s",
> > > + IFPGA_ARG_NAME);
> > > + goto end;
> > > + }
> > > + } else {
> > > + IFPGA_BUS_ERR("arg %s is mandatory for ifpga bus",
> > > + IFPGA_ARG_NAME);
> > > + goto end;
> > > + }
> > > +
> > > + memset(name1, 0, sizeof(name1));
> > > + snprintf(name1, RTE_RAWDEV_NAME_MAX_LEN,
> > "IFPGA:%s",
> > > name);
> > > +
> > > + rawdev = rte_rawdev_pmd_get_named_dev(name1);
> > > + if (!rawdev)
> > > + goto end;
> > > +
> > > + if (ifpga_find_ifpga_dev(rawdev))
> > > + continue;
> > > +
> > > + ifpga_dev = calloc(1, sizeof(*ifpga_dev));
> > > + if (!ifpga_dev)
> > > + goto end;
> > > +
> > > + ifpga_dev->rdev = rawdev;
> > > + TAILQ_INIT(&ifpga_dev->afu_list);
> > > +
> > > + TAILQ_INSERT_TAIL(&ifpga_device_list, ifpga_dev, next);
> > > + afu_dev = ifpga_scan_one(devargs, ifpga_dev);
> > > + if (afu_dev != NULL)
> > > + TAILQ_INSERT_TAIL(&ifpga_dev->afu_list, afu_dev,
> > next);
> >
> > I don't understand why we need afu_list?, seems we only add a new
> > afu_dev into the list after create a new ifpga_dev, Is there any
> > another place that we add one to list to make it reasonable? it looks
> > like either a 1:1 or 1:0 for ifpga_dev:afu_dev?
>
> One FPAG may support more than one AFU.
>
> > > + }
> > > +
> > > +end:
> > > + if (kvlist)
> > > + rte_kvargs_free(kvlist);
> > > + if (name)
> > > + free(name);
> > > +
> > > + return 0;
> > > +}
> > > +
> > > +/*
> > > + * Match the AFU Driver and AFU Device using the ID Table */
> > > +static int rte_afu_match(const struct rte_afu_driver *afu_drv,
> > > + const struct rte_afu_device *afu_dev) {
> > > + const struct rte_afu_uuid *id_table;
> > > +
> > > + for (id_table = afu_drv->id_table;
> > > + ((id_table->uuid_low != 0) && (id_table->uuid_high != 0));
> > > + id_table++) {
> > > + /* check if device's identifiers match the driver's ones */
> > > + if ((id_table->uuid_low != afu_dev->id.uuid.uuid_low) ||
> > > + id_table->uuid_high !=
> > > + afu_dev->id.uuid.uuid_high)
> > > + continue;
> > > +
> > > + return 1;
> > > + }
> > > +
> > > + return 0;
> > > +}
> > > +
> > > +static int
> > > +ifpga_probe_one_driver(struct rte_afu_driver *drv,
> > > + struct rte_afu_device *afu_dev)
> > > +{
> > > + int ret;
> > > +
> > > + if (!rte_afu_match(drv, afu_dev))
> > > + /* Match of device and driver failed */
> > > + return 1;
> > > +
> > > + /* reference driver structure */
> > > + afu_dev->driver = drv;
> > > + afu_dev->device.driver = &drv->driver;
> > > +
> > > + /* call the driver probe() function */
> > > + ret = drv->probe(afu_dev);
> > > + if (ret) {
> > > + afu_dev->driver = NULL;
> > > + afu_dev->device.driver = NULL;
> > > + }
> > > +
> > > + return ret;
> > > +}
> > > +
> > > +static int
> > > +ifpga_probe_all_drivers(struct rte_afu_device *afu_dev) {
> > > + struct rte_afu_driver *drv = NULL;
> > > + int ret = 0;
> > > +
> > > + if (afu_dev == NULL)
> > > + return -1;
> > > +
> > > + /* Check if a driver is already loaded */
> > > + if (afu_dev->driver != NULL)
> > > + return 0;
> > > +
> > > + TAILQ_FOREACH(drv, &afu_driver_list, next) {
> > > + if (ifpga_probe_one_driver(drv, afu_dev)) {
> > > + ret = -1;
> > > + break;
> > > + }
> > > + }
> > > + return ret;
> > > +}
> > > +
> > > +/*
> > > + * Scan the content of the Intel FPGA bus, and call the probe()
> > > +function for
> > > + * all registered drivers that have a matching entry in its
> > > +id_table
> > > + * for discovered devices.
> > > + */
> > > +static int
> > > +ifpga_probe(void)
> > > +{
> > > + struct rte_ifpga_device *ifpga_dev;
> > > + struct rte_afu_device *afu_dev = NULL;
> > > + int ret = 0;
> > > +
> > > + TAILQ_FOREACH(ifpga_dev, &ifpga_device_list, next) {
> > > + TAILQ_FOREACH(afu_dev, &ifpga_dev->afu_list, next) {
> > > +
> > > + if (afu_dev->device.driver)
> > > + continue;
> > > +
> > > + ret = ifpga_probe_all_drivers(afu_dev);
> > > + if (ret < 0)
> > > + IFPGA_BUS_ERR("failed to initialize %s
> > device\n",
> > > + rte_ifpga_device_name(afu_dev));
> > > + }
> > > + }
> > > +
> > > + return ret;
> > > +}
> > > +
> > > +static int
> > > +ifpga_plug(struct rte_device *dev)
> > > +{
> > > + return ifpga_probe_all_drivers(RTE_DEV_TO_AFU(dev));
> > > +}
> > > +
> > > +static int
> > > +ifpga_remove_driver(struct rte_afu_device *afu_dev) {
> > > + const char *name;
> > > + const struct rte_afu_driver *driver;
> > > +
> > > + name = rte_ifpga_device_name(afu_dev);
> > > + if (!afu_dev->device.driver) {
> > > + IFPGA_BUS_DEBUG("no driver attach to device %s\n", name);
> > > + return 1;
> > > + }
> > > +
> > > + driver = RTE_DRV_TO_AFU_CONST(afu_dev->device.driver);
> > > + return driver->remove(afu_dev);
> > > +}
> > > +
> > > +static int
> > > +ifpga_unplug(struct rte_device *dev) {
> > > + struct rte_ifpga_device *ifpga_dev = NULL;
> > > + struct rte_afu_device *afu_dev = NULL;
> > > + struct rte_devargs *devargs = NULL;
> > > + int ret;
> > > +
> > > + if (dev == NULL)
> > > + return -EINVAL;
> > > +
> > > + afu_dev = RTE_DEV_TO_AFU(dev);
> > > + if (!dev)
> > > + return -ENOENT;
> > > +
> > > + ifpga_dev = afu_dev->ifpga_dev;
> > > + devargs = dev->devargs;
> > > +
> > > + ret = ifpga_remove_driver(afu_dev);
> >
> > So what is the device type that be plugged into the ifgpa bus?
> > ifpga_dev or afu_dev? If its afu_dev, why we need ifpga_dev?
> > If it is ifpga_dev, we need remove ifpga_dev from ifpga_device_list here.
>
> The device type that be plugged into the ifpga bus is afu_dev.
> And each afu_dev will be added into one ifpga_dev.
> Each FPGA device bind to ifpga_dev.
>
> > > + if (ret)
> > > + return ret;
> > > +
> > > + TAILQ_REMOVE(&ifpga_dev->afu_list, afu_dev, next);
> > > +
> > > + rte_devargs_remove(devargs->bus->name, devargs->name);
> > > + free(afu_dev);
> > > + return 0;
> > > +
> > > +}
> > > +
> > > +static struct rte_device *
> > > +ifpga_find_device(const struct rte_device *start,
> > > + rte_dev_cmp_t cmp, const void *data) {
> > > + struct rte_ifpga_device *ifpga_dev;
> > > + struct rte_afu_device *afu_dev;
> > > +
> > > + TAILQ_FOREACH(ifpga_dev, &ifpga_device_list, next) {
> > > + TAILQ_FOREACH(afu_dev, &ifpga_dev->afu_list, next) {
> > > + if (start && &afu_dev->device == start) {
> > > + start = NULL;
> > > + continue;
> > > + }
> > > + if (cmp(&afu_dev->device, data) == 0)
> > > + return &afu_dev->device;
> > > + }
> > > + }
> > > + return NULL;
> > > +}
> > > +static int
> > > +ifpga_parse(const char *name, void *addr) {
> > > + int *out = addr;
> > > + struct rte_rawdev *rawdev = NULL;
> > > + char rawdev_name[RTE_RAWDEV_NAME_MAX_LEN];
> > > + char *c1 = NULL, *c2 = NULL;
> > > + int port = IFPGA_BUS_DEV_PORT_MAX;
> > > + char str_port[8];
> > > + int str_port_len = 0;
> > > + int ret;
> > > +
> > > + memset(str_port, 0, 8);
> > > + c1 = strchr(name, '|');
> > > + if (c1 != NULL) {
> > > + str_port_len = c1-name;
> > > + c2 = c1+1;
> > > + }
> > > +
> > > + if (str_port_len < 8 &&
> > > + str_port_len > 0) {
> > > + memcpy(str_port, name, str_port_len);
> > > + ret = sscanf(str_port, "%d", &port);
> > > + if (ret == -1)
> > > + return 0;
> > > + }
> > > +
> > > + memset(rawdev_name, 0, sizeof(rawdev_name));
> > > + snprintf(rawdev_name, RTE_RAWDEV_NAME_MAX_LEN, "IFPGA:%s",
> > c2);
> > > + rawdev = rte_rawdev_pmd_get_named_dev(rawdev_name);
> > > +
> > > + if ((port < IFPGA_BUS_DEV_PORT_MAX) &&
> > > + rawdev &&
> > > + (addr != NULL))
> > > + *out = port;
> > > +
> > > + if ((port < IFPGA_BUS_DEV_PORT_MAX) &&
> > > + rawdev)
> > > + return 0;
> > > + else
> > > + return 1;
> > > +}
> > > +
> > > +static struct rte_bus rte_ifpga_bus = {
> > > + .scan = ifpga_scan,
> > > + .probe = ifpga_probe,
> > > + .find_device = ifpga_find_device,
> > > + .plug = ifpga_plug,
> > > + .unplug = ifpga_unplug,
> > > + .parse = ifpga_parse,
> > > +};
> > > +
> > > +RTE_REGISTER_BUS(IFPGA_BUS_NAME, rte_ifpga_bus);
> > > +
> > > +RTE_INIT(ifpga_init_log)
> > > +{
> > > + ifpga_bus_logtype = rte_log_register("bus.ifpga");
> > > + if (ifpga_bus_logtype >= 0)
> > > + rte_log_set_level(ifpga_bus_logtype, RTE_LOG_NOTICE); }
> > > diff --git a/drivers/bus/ifpga/ifpga_common.c
> > > b/drivers/bus/ifpga/ifpga_common.c
> > > new file mode 100644
> > > index 0000000..78e2eae
> > > --- /dev/null
> > > +++ b/drivers/bus/ifpga/ifpga_common.c
> > > @@ -0,0 +1,88 @@
> > > +/* SPDX-License-Identifier: BSD-3-Clause
> > > + * Copyright(c) 2010-2018 Intel Corporation */
> > > +
> > > +#include <string.h>
> > > +#include <inttypes.h>
> > > +#include <stdint.h>
> > > +#include <stdlib.h>
> > > +#include <stdio.h>
> > > +#include <sys/queue.h>
> > > +#include <sys/mman.h>
> > > +#include <sys/types.h>
> > > +#include <unistd.h>
> > > +#include <fcntl.h>
> > > +
> > > +#include <rte_errno.h>
> > > +#include <rte_bus.h>
> > > +#include <rte_per_lcore.h>
> > > +#include <rte_memory.h>
> > > +#include <rte_memzone.h>
> > > +#include <rte_eal.h>
> > > +#include <rte_common.h>
> > > +
> > > +#include <rte_devargs.h>
> > > +#include <rte_kvargs.h>
> > > +#include <rte_alarm.h>
> > > +
> > > +#include "rte_bus_ifpga.h"
> > > +#include "ifpga_logs.h"
> > > +#include "ifpga_common.h"
> > > +
> > > +int rte_ifpga_get_string_arg(const char *key __rte_unused,
> > > + const char *value, void *extra_args) {
> > > + if (!value || !extra_args)
> > > + return -EINVAL;
> > > +
> > > + *(char **)extra_args = strdup(value);
> > > +
> > > + if (!*(char **)extra_args)
> > > + return -ENOMEM;
> > > +
> > > + return 0;
> > > +}
> > > +int rte_ifpga_get_integer32_arg(const char *key __rte_unused,
> > > + const char *value, void *extra_args) {
> > > + if (!value || !extra_args)
> > > + return -EINVAL;
> > > +
> > > + *(int *)extra_args = strtoull(value, NULL, 0);
> > > +
> > > + return 0;
> > > +}
> > > +int ifpga_get_integer64_arg(const char *key __rte_unused,
> > > + const char *value, void *extra_args) {
> > > + if (!value || !extra_args)
> > > + return -EINVAL;
> > > +
> > > + *(uint64_t *)extra_args = strtoull(value, NULL, 0);
> > > +
> > > + return 0;
> > > +}
> > > +int ifpga_get_unsigned_long(const char *str, int base) {
> > > + unsigned long num;
> > > + char *end = NULL;
> > > +
> > > + errno = 0;
> > > +
> > > + num = strtoul(str, &end, base);
> > > + if ((str[0] == '\0') || (end == NULL) || (*end != '\0') || (errno != 0))
> > > + return -1;
> > > +
> > > + return num;
> > > +}
> > > +
> > > +int ifpga_afu_id_cmp(const struct rte_afu_id *afu_id0,
> > > + const struct rte_afu_id *afu_id1)
> > > +{
> > > + if ((afu_id0->uuid.uuid_low == afu_id1->uuid.uuid_low) &&
> > > + (afu_id0->uuid.uuid_high == afu_id1->uuid.uuid_high) &&
> > > + (afu_id0->port == afu_id1->port)) {
> > > + return 0;
> > > + } else
> > > + return 1;
> > > +}
> > > diff --git a/drivers/bus/ifpga/ifpga_common.h
> > > b/drivers/bus/ifpga/ifpga_common.h
> > > new file mode 100644
> > > index 0000000..f9254b9
> > > --- /dev/null
> > > +++ b/drivers/bus/ifpga/ifpga_common.h
> > > @@ -0,0 +1,18 @@
> > > +/* SPDX-License-Identifier: BSD-3-Clause
> > > + * Copyright(c) 2010-2018 Intel Corporation */
> > > +
> > > +#ifndef _IFPGA_COMMON_H_
> > > +#define _IFPGA_COMMON_H_
> > > +
> > > +int rte_ifpga_get_string_arg(const char *key __rte_unused,
> > > + const char *value, void *extra_args); int
> > > +rte_ifpga_get_integer32_arg(const char *key __rte_unused,
> > > + const char *value, void *extra_args); int
> > > +ifpga_get_integer64_arg(const char *key __rte_unused,
> > > + const char *value, void *extra_args); int
> > > +ifpga_get_unsigned_long(const char *str, int base); int
> > > +ifpga_afu_id_cmp(const struct rte_afu_id *afu_id0,
> > > + const struct rte_afu_id *afu_id1);
> > > +
> > > +#endif /* _IFPGA_COMMON_H_ */
> > > diff --git a/drivers/bus/ifpga/ifpga_logs.h
> > > b/drivers/bus/ifpga/ifpga_logs.h new file mode 100644 index
> > > 0000000..873e0a4
> > > --- /dev/null
> > > +++ b/drivers/bus/ifpga/ifpga_logs.h
> > > @@ -0,0 +1,31 @@
> > > +/* SPDX-License-Identifier: BSD-3-Clause
> > > + * Copyright(c) 2010-2018 Intel Corporation */
> > > +
> > > +#ifndef _IFPGA_LOGS_H_
> > > +#define _IFPGA_LOGS_H_
> > > +
> > > +#include <rte_log.h>
> > > +
> > > +extern int ifpga_bus_logtype;
> > > +
> > > +#define IFPGA_LOG(level, fmt, args...) \
> > > + rte_log(RTE_LOG_ ## level, ifpga_bus_logtype, "%s(): " fmt "\n", \
> > > + __func__, ##args)
> > > +
> > > +#define IFPGA_BUS_LOG(level, fmt, args...) \
> > > + rte_log(RTE_LOG_ ## level, ifpga_bus_logtype, "%s(): " fmt "\n", \
> > > + __func__, ##args)
> > > +
> > > +#define IFPGA_BUS_FUNC_TRACE() IFPGA_BUS_LOG(DEBUG, ">>")
> > > +
> > > +#define IFPGA_BUS_DEBUG(fmt, args...) \
> > > + IFPGA_BUS_LOG(DEBUG, fmt, ## args) #define IFPGA_BUS_INFO(fmt,
> > > +args...) \
> > > + IFPGA_BUS_LOG(INFO, fmt, ## args)
> > > +#define IFPGA_BUS_ERR(fmt, args...) \
> > > + IFPGA_BUS_LOG(ERR, fmt, ## args)
> > > +#define IFPGA_BUS_WARN(fmt, args...) \
> > > + IFPGA_BUS_LOG(WARNING, fmt, ## args)
> > > +
> > > +#endif /* _IFPGA_BUS_LOGS_H_ */
> > > diff --git a/drivers/bus/ifpga/meson.build
> > > b/drivers/bus/ifpga/meson.build new file mode 100644 index
> > > 0000000..c9b08c8
> > > --- /dev/null
> > > +++ b/drivers/bus/ifpga/meson.build
> > > @@ -0,0 +1,8 @@
> > > +# SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2010-2018
> > > +Intel Corporation
> > > +
> > > +deps += ['pci', 'kvargs', 'rawdev']
> > > +install_headers('rte_bus_ifpga.h')
> > > +sources = files('ifpga_common.c', 'ifpga_bus.c')
> > > +
> > > +allow_experimental_apis = true
> > > diff --git a/drivers/bus/ifpga/rte_bus_ifpga.h
> > > b/drivers/bus/ifpga/rte_bus_ifpga.h
> > > new file mode 100644
> > > index 0000000..5c559e1
> > > --- /dev/null
> > > +++ b/drivers/bus/ifpga/rte_bus_ifpga.h
> > > @@ -0,0 +1,160 @@
> > > +/* SPDX-License-Identifier: BSD-3-Clause
> > > + * Copyright(c) 2010-2018 Intel Corporation */
> > > +
> > > +#ifndef _RTE_BUS_IFPGA_H_
> > > +#define _RTE_BUS_IFPGA_H_
> > > +
> > > +/**
> > > + * @file
> > > + *
> > > + * RTE Intel FPGA Bus Interface
> > > + */
> > > +
> > > +#ifdef __cplusplus
> > > +extern "C" {
> > > +#endif
> > > +
> > > +#include <rte_bus.h>
> > > +#include <rte_pci.h>
> > > +
> > > +/** Name of Intel FPGA Bus */
> > > +#define IFPGA_BUS_NAME ifpga
> > > +
> > > +/* Forward declarations */
> > > +struct rte_afu_device;
> > > +
> > > +/** List of Intel AFU devices */
> > > +TAILQ_HEAD(afu_device_list, rte_afu_device);
> > > +/** Double linked list of AFU device drivers. */
> > > +TAILQ_HEAD(afu_driver_list, rte_afu_driver);
> > > +
> > > +#define IFPGA_BUS_BITSTREAM_PATH_MAX_LEN 256
> > > +
> > > +struct rte_afu_uuid {
> > > + uint64_t uuid_low;
> > > + uint64_t uuid_high;
> > > +} __attribute__ ((packed));
> > > +
> > > +#define IFPGA_BUS_DEV_PORT_MAX 4
> > > +
> > > +/**
> > > + * A structure describing an ID for a AFU driver. Each driver
> > > +provides a
> > > + * table of these IDs for each device that it supports.
> > > + */
> > > +struct rte_afu_id {
> > > + struct rte_afu_uuid uuid;
> > > + int port; /**< port number */
> > > +} __attribute__ ((packed));
> > > +
> > > +/**
> > > + * A structure PR (Partial Reconfiguration) configuration AFU driver.
> > > + */
> > > +
> > > +struct rte_afu_pr_conf {
> > > + struct rte_afu_id afu_id;
> > > + int pr_enable;
> > > + char bs_path[IFPGA_BUS_BITSTREAM_PATH_MAX_LEN];
> > > +};
> > > +
> > > +#define AFU_PRI_STR_SIZE (PCI_PRI_STR_SIZE + 8)
> > > +
> > > +/**
> > > + * A structure describing a fpga device.
> > > + */
> > > +struct rte_ifpga_device {
> > > + TAILQ_ENTRY(rte_ifpga_device) next; /**< Next in device list. */
> > > + struct rte_rawdev *rdev;
> > > + struct afu_device_list afu_list; /**< List of AFU devices */ };
> > > +
> > > +/**
> > > + * A structure describing a AFU device.
> > > + */
> > > +struct rte_afu_device {
> > > + TAILQ_ENTRY(rte_afu_device) next; /**< Next in device list. */
> > > + struct rte_device device; /**< Inherit core device */
> > > + struct rte_rawdev *rawdev; /**< Point Rawdev */
> > > + struct rte_ifpga_device *ifpga_dev; /**< Point ifpga device */
> > > + struct rte_afu_id id; /**< AFU id within FPGA. */
> > > + uint32_t num_region; /**< number of regions found */
> > > + struct rte_mem_resource mem_resource[PCI_MAX_RESOURCE];
> > > + /**< AFU Memory Resource
> > */
> > > + struct rte_intr_handle intr_handle; /**< Interrupt handle */
> > > + struct rte_afu_driver *driver; /**< Associated driver */
> > > + char path[IFPGA_BUS_BITSTREAM_PATH_MAX_LEN];
> > > +} __attribute__ ((packed));
> > > +
> > > +/**
> > > + * @internal
> > > + * Helper macro for drivers that need to convert to struct rte_afu_device.
> > > + */
> > > +#define RTE_DEV_TO_AFU(ptr) \
> > > + container_of(ptr, struct rte_afu_device, device)
> > > +
> > > +#define RTE_DRV_TO_AFU_CONST(ptr) \
> > > + container_of(ptr, const struct rte_afu_driver, driver)
> > > +
> > > +/**
> > > + * Initialisation function for the driver called during FPGA BUS probing.
> >
> >
> > typo: Initialization
>
> Fixed.
>
> > > + */
> > > +typedef int (afu_probe_t)(struct rte_afu_device *);
> > > +
> > > +/**
> > > + * Uninitialisation function for the driver called during hotplugging.
> >
> > typo: Unintialization
>
> Fixed.
>
> > > + */
> > > +typedef int (afu_remove_t)(struct rte_afu_device *);
> > > +
> > > +/**
> > > + * A structure describing a AFU device.
> > > + */
> > > +struct rte_afu_driver {
> > > + TAILQ_ENTRY(rte_afu_driver) next; /**< Next afu driver. */
> > > + struct rte_driver driver; /**< Inherit core driver. */
> > > + afu_probe_t *probe; /**< Device Probe function.
> > > */
> > > + afu_remove_t *remove; /**< Device Remove
> > > function. */
> > > + const struct rte_afu_uuid *id_table; /**< AFU uuid within FPGA. */
> > > + uint32_t drv_flags; /**< Flags contolling handling of device. */
> >
> > Typo Controlling
> > And what is "flags controlling handling of device?"
>
> This variable is not used, so I have removed it.
>
> > > +};
> > > +
> > > +static inline const char *
> > > +rte_ifpga_device_name(const struct rte_afu_device *afu) {
> > > + if (afu && afu->device.name)
> > > + return afu->device.name;
> > > + return NULL;
> > > +}
> > > +
> > > +/**
> > > + * Register a ifpga afu device driver.
> > > + *
> > > + * @param driver
> > > + * A pointer to a rte_afu_driver structure describing the driver
> > > + * to be registered.
> > > + */
> > > +void rte_ifpga_driver_register(struct rte_afu_driver *driver);
> > > +
> > > +/**
> > > + * Unregister a ifpga afu device driver.
> > > + *
> > > + * @param driver
> > > + * A pointer to a rte_afu_driver structure describing the driver
> > > + * to be unregistered.
> > > + */
> > > +void rte_ifpga_driver_unregister(struct rte_afu_driver *driver);
> > > +
> > > +#define RTE_PMD_REGISTER_AFU(nm, afudrv)\ RTE_INIT(afudrvinitfn_
> > > +##afudrv);\ static const char *afudrvinit_ ## nm ## _alias;\ static
> > > +void afudrvinitfn_ ##afudrv(void)\ {\
> > > + (afudrv).driver.name = RTE_STR(nm);\
> > > + (afudrv).driver.alias = afudrvinit_ ## nm ## _alias;\
> > > + rte_ifpga_driver_register(&afudrv);\
> > > +} \
> > > +RTE_PMD_EXPORT_NAME(nm, __COUNTER__)
> > > +
> > > +#define RTE_PMD_REGISTER_AFU_ALIAS(nm, alias)\ static const char
> > > +*afudrvinit_ ## nm ## _alias = RTE_STR(alias)
> > > +
> > > +#endif /* _RTE_BUS_IFPGA_H_ */
> > > diff --git a/drivers/bus/ifpga/rte_bus_ifpga_version.map
> > > b/drivers/bus/ifpga/rte_bus_ifpga_version.map
> > > new file mode 100644
> > > index 0000000..a027979
> > > --- /dev/null
> > > +++ b/drivers/bus/ifpga/rte_bus_ifpga_version.map
> > > @@ -0,0 +1,10 @@
> > > +DPDK_18.05 {
> > > + global:
> > > +
> > > + rte_ifpga_get_integer32_arg;
> > > + rte_ifpga_get_string_arg;
> > > + rte_ifpga_driver_register;
> > > + rte_ifpga_driver_unregister;
> > > +
> > > + local: *;
> > > +};
> > > diff --git a/drivers/bus/meson.build b/drivers/bus/meson.build index
> > > 58dfbe2..52c755d 100644
> > > --- a/drivers/bus/meson.build
> > > +++ b/drivers/bus/meson.build
> > > @@ -1,7 +1,7 @@
> > > # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2017 Intel
> > > Corporation
> > >
> > > -drivers = ['dpaa', 'fslmc', 'pci', 'vdev']
> > > +drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'vdev']
> > > std_deps = ['eal']
> > > config_flag_fmt = 'RTE_LIBRTE_@0@_BUS'
> > > driver_name_fmt = 'rte_bus_@0@'
> > > diff --git a/mk/rte.app.mk b/mk/rte.app.mk index 26f3563..3861e1a
> > > 100644
> > > --- a/mk/rte.app.mk
> > > +++ b/mk/rte.app.mk
> > > @@ -255,6 +255,7 @@ ifeq
> > > ($(CONFIG_RTE_EAL_VFIO)$(CONFIG_RTE_LIBRTE_FSLMC_BUS),yy)
> > > _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_CMDIF_RAWDEV) +=
> > > -lrte_pmd_dpaa2_cmdif
> > > _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV) +=
> > > -lrte_pmd_dpaa2_qdma endif # CONFIG_RTE_LIBRTE_FSLMC_BUS
> > > +_LDLIBS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += -lrte_bus_ifpga
> > > endif # CONFIG_RTE_LIBRTE_RAWDEV
> > >
> > >
> > > --
> > > 1.8.3.1
> >
> > Regards
> > Qi
Hi Qi,
> -----Original Message-----
> From: Zhang, Qi Z
> Sent: Thursday, May 10, 2018 21:49
> To: Xu, Rosen <rosen.xu@intel.com>; dev@dpdk.org; thomas@monjalon.net
> Cc: Zhang, Roy Fan <roy.fan.zhang@intel.com>; Doherty, Declan
> <declan.doherty@intel.com>; Richardson, Bruce
> <bruce.richardson@intel.com>; shreyansh.jain@nxp.com; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Ananyev, Konstantin
> <konstantin.ananyev@intel.com>; Zhang, Tianfei <tianfei.zhang@intel.com>;
> Liu, Song <song.liu@intel.com>; Wu, Hao <hao.wu@intel.com>;
> gaetan.rivet@6wind.com
> Subject: RE: [dpdk-dev] [PATCH v10 1/3] bus/ifpga: Add Intel FPGA BUS
> Library
>
>
>
> > -----Original Message-----
> > From: Xu, Rosen
> > Sent: Thursday, May 10, 2018 9:29 PM
> > To: Zhang, Qi Z <qi.z.zhang@intel.com>; dev@dpdk.org;
> > thomas@monjalon.net
> > Cc: Zhang, Roy Fan <roy.fan.zhang@intel.com>; Doherty, Declan
> > <declan.doherty@intel.com>; Richardson, Bruce
> > <bruce.richardson@intel.com>; shreyansh.jain@nxp.com; Yigit, Ferruh
> > <ferruh.yigit@intel.com>; Ananyev, Konstantin
> > <konstantin.ananyev@intel.com>; Zhang, Tianfei
> > <tianfei.zhang@intel.com>; Liu, Song <song.liu@intel.com>; Wu, Hao
> > <hao.wu@intel.com>; gaetan.rivet@6wind.com
> > Subject: RE: [dpdk-dev] [PATCH v10 1/3] bus/ifpga: Add Intel FPGA BUS
> > Library
> >
> > Hi Qi,
> >
> > > -----Original Message-----
> > > From: Zhang, Qi Z
> > > Sent: Thursday, May 10, 2018 20:27
> > > To: Xu, Rosen <rosen.xu@intel.com>; dev@dpdk.org;
> > thomas@monjalon.net
> > > Cc: Xu, Rosen <rosen.xu@intel.com>; Zhang, Roy Fan
> > > <roy.fan.zhang@intel.com>; Doherty, Declan
> > > <declan.doherty@intel.com>; Richardson, Bruce
> > > <bruce.richardson@intel.com>; shreyansh.jain@nxp.com; Yigit, Ferruh
> > > <ferruh.yigit@intel.com>; Ananyev, Konstantin
> > > <konstantin.ananyev@intel.com>; Zhang, Tianfei
> > > <tianfei.zhang@intel.com>; Liu, Song <song.liu@intel.com>; Wu, Hao
> > > <hao.wu@intel.com>; gaetan.rivet@6wind.com
> > > Subject: RE: [dpdk-dev] [PATCH v10 1/3] bus/ifpga: Add Intel FPGA
> > > BUS Library
> > >
> > > Hi Rosen:
> > >
> > > > -----Original Message-----
> > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Xu, Rosen
> > > > Sent: Wednesday, May 9, 2018 3:43 PM
> > > > To: dev@dpdk.org; thomas@monjalon.net
> > > > Cc: Xu, Rosen <rosen.xu@intel.com>; Zhang, Roy Fan
> > > > <roy.fan.zhang@intel.com>; Doherty, Declan
> > > > <declan.doherty@intel.com>; Richardson, Bruce
> > > > <bruce.richardson@intel.com>; shreyansh.jain@nxp.com; Yigit,
> > > > Ferruh <ferruh.yigit@intel.com>; Ananyev, Konstantin
> > > > <konstantin.ananyev@intel.com>; Zhang, Tianfei
> > > > <tianfei.zhang@intel.com>; Liu, Song <song.liu@intel.com>; Wu, Hao
> > > > <hao.wu@intel.com>; gaetan.rivet@6wind.com
> > > > Subject: [dpdk-dev] [PATCH v10 1/3] bus/ifpga: Add Intel FPGA BUS
> > > > Library
> > > >
> > > > From: Rosen Xu <rosen.xu@intel.com>
> > > >
> > > > Defined FPGA-BUS for Acceleration Drivers of AFUs
> > > >
> > > > 1. FPGA PCI Scan (1st Scan) follows DPDK UIO/VFIO PCI Scan
> > > > Process, probe Intel FPGA Rawdev Driver, it will be covered in following
> patches.
> > > >
> > > > 2. AFU Scan(2nd Scan) bind DPDK driver to FPGA Partial-Bitstream.
> > > > This scan is trigged by hotplug of IFPGA Rawdev probe, in this
> > > > scan the AFUs will be created and their drivers are also probed.
> > > >
> > > > This patch will introduce rte_afu_device which describe the AFU
> > > > device listed in the FPGA-BUS.
> > > >
> > > > Signed-off-by: Rosen Xu <rosen.xu@intel.com>
> > > > Signed-off-by: Tianfei Zhang <tianfei.zhang@intel.com>
> > > > ---
> > > > MAINTAINERS | 4 +
> > > > config/common_base | 5 +
> > > > doc/guides/rel_notes/release_18_05.rst | 5 +
> > > > drivers/bus/Makefile | 1 +
> > > > drivers/bus/ifpga/Makefile | 32 ++
> > > > drivers/bus/ifpga/ifpga_bus.c | 501
> > > > ++++++++++++++++++++++++++++
> > > > drivers/bus/ifpga/ifpga_common.c | 88 +++++
> > > > drivers/bus/ifpga/ifpga_common.h | 18 +
> > > > drivers/bus/ifpga/ifpga_logs.h | 31 ++
> > > > drivers/bus/ifpga/meson.build | 8 +
> > > > drivers/bus/ifpga/rte_bus_ifpga.h | 160 +++++++++
> > > > drivers/bus/ifpga/rte_bus_ifpga_version.map | 10 +
> > > > drivers/bus/meson.build | 2 +-
> > > > mk/rte.app.mk | 1 +
> > > > 14 files changed, 865 insertions(+), 1 deletion(-) create mode
> > > > 100644 drivers/bus/ifpga/Makefile create mode 100644
> > > > drivers/bus/ifpga/ifpga_bus.c create mode 100644
> > > > drivers/bus/ifpga/ifpga_common.c create mode 100644
> > > > drivers/bus/ifpga/ifpga_common.h create mode 100644
> > > > drivers/bus/ifpga/ifpga_logs.h create mode 100644
> > > > drivers/bus/ifpga/meson.build create mode 100644
> > > > drivers/bus/ifpga/rte_bus_ifpga.h create mode 100644
> > > > drivers/bus/ifpga/rte_bus_ifpga_version.map
> > > >
> > > > diff --git a/MAINTAINERS b/MAINTAINERS index 7105920..fa0c5b1
> > > > 100644
> > > > --- a/MAINTAINERS
> > > > +++ b/MAINTAINERS
> > > > @@ -383,6 +383,10 @@ F: drivers/mempool/bucket/ Bus Drivers
> > > > -----------
> > > >
> > > > +Intel FPGA buses
> > > > +M: Rosen Xu <rosen.xu@intel.com>
> > > > +F: drivers/bus/ifpga/
> > > > +
> > > > NXP buses
> > > > M: Hemant Agrawal <hemant.agrawal@nxp.com>
> > > > M: Shreyansh Jain <shreyansh.jain@nxp.com> diff --git
> > > > a/config/common_base b/config/common_base index
> 0d181ac..1440316
> > > > 100644
> > > > --- a/config/common_base
> > > > +++ b/config/common_base
> > > > @@ -139,6 +139,11 @@
> > > > CONFIG_RTE_ETHDEV_PROFILE_ITT_WASTED_RX_ITERATIONS=n
> > > > CONFIG_RTE_ETHDEV_TX_PREPARE_NOOP=n
> > > >
> > > > #
> > > > +# Compile the Intel FPGA bus
> > > > +#
> > > > +CONFIG_RTE_LIBRTE_IFPGA_BUS=y
> > > > +
> > > > +#
> > > > # Compile PCI bus driver
> > > > #
> > > > CONFIG_RTE_LIBRTE_PCI_BUS=y
> > > > diff --git a/doc/guides/rel_notes/release_18_05.rst
> > > > b/doc/guides/rel_notes/release_18_05.rst
> > > > index 7187348..265950a 100644
> > > > --- a/doc/guides/rel_notes/release_18_05.rst
> > > > +++ b/doc/guides/rel_notes/release_18_05.rst
> > > > @@ -183,6 +183,11 @@ New Features
> > > > stats/xstats on shared memory from secondary process, and also
> > > > pdump packets on
> > > > those virtual devices.
> > > >
> > > > +* **Added Ifpga Bus, a generic Intel FPGA Bus library.**
> > > > +
> > > > + The Ifpga Bus library provides support for integrating any
> > > > + Intel FPGA device with the DPDK framework. It provides Intel
> > > > + FPGA Partial Bit Stream AFU(Accelerated Function Unit) scan and
> > > > + drivers
> > prove.
> > > >
> > > > API Changes
> > > > -----------
> > > > diff --git a/drivers/bus/Makefile b/drivers/bus/Makefile index
> > > > c251b65..ef7f247 100644
> > > > --- a/drivers/bus/Makefile
> > > > +++ b/drivers/bus/Makefile
> > > > @@ -7,6 +7,7 @@ DIRS-$(CONFIG_RTE_LIBRTE_DPAA_BUS) += dpaa
> ifeq
> > > > ($(CONFIG_RTE_EAL_VFIO),y)
> > > > DIRS-$(CONFIG_RTE_LIBRTE_FSLMC_BUS) += fslmc endif
> > > > +DIRS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += ifpga
> > > > DIRS-$(CONFIG_RTE_LIBRTE_PCI_BUS) += pci
> > > > DIRS-$(CONFIG_RTE_LIBRTE_VDEV_BUS) += vdev
> > > >
> > > > diff --git a/drivers/bus/ifpga/Makefile
> > > > b/drivers/bus/ifpga/Makefile new file mode 100644 index
> > > > 0000000..3ff3bdb
> > > > --- /dev/null
> > > > +++ b/drivers/bus/ifpga/Makefile
> > > > @@ -0,0 +1,32 @@
> > > > +# SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2018 Intel
> > > > +Corporation
> > > > +
> > > > +include $(RTE_SDK)/mk/rte.vars.mk
> > > > +
> > > > +#
> > > > +# library name
> > > > +#
> > > > +LIB = librte_bus_ifpga.a
> > > > +
> > > > +CFLAGS += -DALLOW_EXPERIMENTAL_API CFLAGS += -O3 CFLAGS +=
> > > > +$(WERROR_FLAGS) LDLIBS += -lrte_eal LDLIBS += -lrte_rawdev LDLIBS
> > > > ++= -lrte_kvargs
> > > > +
> > > > +# versioning export map
> > > > +EXPORT_MAP := rte_bus_ifpga_version.map
> > > > +
> > > > +# library version
> > > > +LIBABIVER := 1
> > > > +
> > > > +SRCS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += ifpga_bus.c
> > > > +SRCS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += ifpga_common.c
> > > > +
> > > > +#
> > > > +# Export include files
> > > > +#
> > > > +SYMLINK-$(CONFIG_RTE_LIBRTE_IFPGA_BUS)-include +=
> rte_bus_ifpga.h
> > > > +
> > > > +include $(RTE_SDK)/mk/rte.lib.mk
> > > > diff --git a/drivers/bus/ifpga/ifpga_bus.c
> > > > b/drivers/bus/ifpga/ifpga_bus.c new file mode 100644 index
> > > > 0000000..e144c01
> > > > --- /dev/null
> > > > +++ b/drivers/bus/ifpga/ifpga_bus.c
> > > > @@ -0,0 +1,501 @@
> > > > +/* SPDX-License-Identifier: BSD-3-Clause
> > > > + * Copyright(c) 2010-2018 Intel Corporation */
> > > > +
> > > > +#include <string.h>
> > > > +#include <inttypes.h>
> > > > +#include <stdint.h>
> > > > +#include <stdlib.h>
> > > > +#include <stdio.h>
> > > > +#include <sys/queue.h>
> > > > +#include <sys/mman.h>
> > > > +#include <sys/types.h>
> > > > +#include <unistd.h>
> > > > +#include <fcntl.h>
> > > > +
> > > > +#include <rte_errno.h>
> > > > +#include <rte_bus.h>
> > > > +#include <rte_per_lcore.h>
> > > > +#include <rte_memory.h>
> > > > +#include <rte_memzone.h>
> > > > +#include <rte_eal.h>
> > > > +#include <rte_common.h>
> > > > +
> > > > +#include <rte_devargs.h>
> > > > +#include <rte_kvargs.h>
> > > > +#include <rte_alarm.h>
> > > > +
> > > > +#include "rte_rawdev.h"
> > > > +#include "rte_rawdev_pmd.h"
> > > > +#include "rte_bus_ifpga.h"
> > > > +#include "ifpga_logs.h"
> > > > +#include "ifpga_common.h"
> > > > +
> > > > +int ifpga_bus_logtype;
> > > > +
> > > > +/* Forward declaration to access Intel FPGA bus
> > > > + * on which iFPGA devices are connected */ static struct rte_bus
> > > > +rte_ifpga_bus;
> > > > +
> > > > +/** Double linked list of IFPGA device. */
> > > > +TAILQ_HEAD(ifpga_device_list, rte_ifpga_device);
> > > > +
> > > > +static struct ifpga_device_list ifpga_device_list =
> > > > + TAILQ_HEAD_INITIALIZER(ifpga_device_list);
> > > > +static struct afu_driver_list afu_driver_list =
> > > > + TAILQ_HEAD_INITIALIZER(afu_driver_list);
> > > > +
> > > > +
> > > > +/* register a ifpga bus based driver */ void
> > > > +rte_ifpga_driver_register(struct rte_afu_driver *driver) {
> > > > + RTE_VERIFY(driver);
> > > > +
> > > > + TAILQ_INSERT_TAIL(&afu_driver_list, driver, next); }
> > > > +
> > > > +/* un-register a fpga bus based driver */ void
> > > > +rte_ifpga_driver_unregister(struct rte_afu_driver *driver) {
> > > > + TAILQ_REMOVE(&afu_driver_list, driver, next); }
> > > > +
> > > > +static struct rte_ifpga_device *
> > > > +ifpga_find_ifpga_dev(const struct rte_rawdev *rdev) {
> > > > + struct rte_ifpga_device *ifpga_dev = NULL;
> > > > +
> > > > + TAILQ_FOREACH(ifpga_dev, &ifpga_device_list, next) {
> > > > + if (rdev &&
> > > > + ifpga_dev->rdev &&
> > > > + ifpga_dev->rdev == rdev)
> > > > + return ifpga_dev;
> > > > + }
> > > > + return NULL;
> > > > +}
> > > > +
> > > > +static struct rte_afu_device *
> > > > +ifpga_find_afu_dev(const struct rte_ifpga_device *ifpga_dev,
> > > > + const struct rte_afu_id *afu_id) {
> > > > + struct rte_afu_device *afu_dev = NULL;
> > > > +
> > > > + TAILQ_FOREACH(afu_dev, &ifpga_dev->afu_list, next) {
> > > > + if (!ifpga_afu_id_cmp(&afu_dev->id, afu_id))
> > > > + return afu_dev;
> > > > + }
> > > > + return NULL;
> > > > +}
> > > > +
> > > > +static const char * const valid_args[] = {
> > > > +#define IFPGA_ARG_NAME "ifpga"
> > > > + IFPGA_ARG_NAME,
> > > > +#define IFPGA_ARG_PORT "port"
> > > > + IFPGA_ARG_PORT,
> > > > +#define IFPGA_AFU_BTS "afu_bts"
> > > > + IFPGA_AFU_BTS,
> > > > + NULL
> > > > +};
> > > > +
> > > > +/*
> > > > + * Scan the content of the FPGA bus, and the devices in the
> > > > +devices
> > > > + * list
> > > > + */
> > > > +static struct rte_afu_device *
> > > > +ifpga_scan_one(struct rte_devargs *devargs,
> > > > + struct rte_ifpga_device *ifpga_dev)
> > >
> > > usually ifpag_dev should be the first parameter here, and devargs
> > > follows
> >
> > Fixed.
> >
> > > > +{
> > > > + struct rte_kvargs *kvlist = NULL;
> > > > + struct rte_rawdev *rawdev = NULL;
> > > > + struct rte_afu_device *afu_dev = NULL;
> > > > + struct rte_afu_pr_conf afu_pr_conf;
> > > > + int ret = 0;
> > > > + char *path = NULL;
> > > > +
> > > > + memset(&afu_pr_conf, 0, sizeof(struct rte_afu_pr_conf));
> > > > +
> > > > + kvlist = rte_kvargs_parse(devargs->args, valid_args);
> > > > + if (!kvlist) {
> > > > + IFPGA_BUS_ERR("error when parsing param");
> > > > + goto end;
> > > > + }
> > > > +
> > > > + if (rte_kvargs_count(kvlist, IFPGA_ARG_PORT) == 1) {
> > > > + if (rte_kvargs_process(kvlist, IFPGA_ARG_PORT,
> > > > + &rte_ifpga_get_integer32_arg, &afu_pr_conf.afu_id.port) < 0)
> > > {
> > > > + IFPGA_BUS_ERR("error to parse %s",
> > > > + IFPGA_ARG_PORT);
> > > > + goto end;
> > > > + }
> > > > + } else {
> > > > + IFPGA_BUS_ERR("arg %s is mandatory for ifpga bus",
> > > > + IFPGA_ARG_PORT);
> > > > + goto end;
> > > > + }
> > > > +
> > > > + if (rte_kvargs_count(kvlist, IFPGA_AFU_BTS) == 1) {
> > > > + if (rte_kvargs_process(kvlist, IFPGA_AFU_BTS,
> > > > + &rte_ifpga_get_string_arg, &path) < 0) {
> > > > + IFPGA_BUS_ERR("Failed to parse %s",
> > > > + IFPGA_AFU_BTS);
> > > > + goto end;
> > > > + }
> > > > + } else {
> > > > + IFPGA_BUS_ERR("arg %s is mandatory for ifpga bus",
> > > > + IFPGA_AFU_BTS);
> > > > + goto end;
> > > > + }
> > > > +
> > > > + afu_pr_conf.afu_id.uuid.uuid_low = 0;
> > > > + afu_pr_conf.afu_id.uuid.uuid_high = 0;
> > > > + afu_pr_conf.pr_enable = path?1:0;
> > > > +
> > > > + rawdev = ifpga_dev->rdev;
> > > > + if (ifpga_find_afu_dev(ifpga_dev, &afu_pr_conf.afu_id))
> > > > + goto end;
> > > > +
> > > > + afu_dev = calloc(1, sizeof(*afu_dev));
> > > > + if (!afu_dev)
> > > > + goto end;
> > > > +
> > > > + afu_dev->device.devargs = devargs;
> > > > + afu_dev->device.numa_node = SOCKET_ID_ANY;
> > > > + afu_dev->device.name = devargs->name;
> > > > + afu_dev->rawdev = rawdev;
> > > > + afu_dev->id.uuid.uuid_low = 0;
> > > > + afu_dev->id.uuid.uuid_high = 0;
> > > > + afu_dev->id.port = afu_pr_conf.afu_id.port;
> > > > + afu_dev->ifpga_dev = ifpga_dev;
> > > > +
> > > > + if (rawdev->dev_ops && rawdev->dev_ops->dev_info_get)
> > > > + rawdev->dev_ops->dev_info_get(rawdev, afu_dev);
> > > > +
> > > > + if (rawdev->dev_ops &&
> > > > + rawdev->dev_ops->dev_start &&
> > > > + rawdev->dev_ops->dev_start(rawdev))
> > > > + goto free_dev;
> > > > +
> > > > + strncpy(afu_pr_conf.bs_path, path, sizeof(afu_pr_conf.bs_path));
> > > > + if (rawdev->dev_ops->firmware_load &&
> > > > + rawdev->dev_ops->firmware_load(rawdev,
> > > > + &afu_pr_conf)){
> > > > + IFPGA_BUS_ERR("firmware load error %d\n", ret);
> > > > + goto free_dev;
> > > > + }
> > > > + afu_dev->id.uuid.uuid_low = afu_pr_conf.afu_id.uuid.uuid_low;
> > > > + afu_dev->id.uuid.uuid_high = afu_pr_conf.afu_id.uuid.uuid_high;
> > > > +
> > > > + return afu_dev;
> > > > +
> > > > +free_dev:
> > > > + free(afu_dev);
> > > > +end:
> > > > + if (kvlist)
> > > > + rte_kvargs_free(kvlist);
> > > > + if (path)
> > > > + free(path);
> > > > +
> > > > + return NULL;
> > > > +}
> > > > +
> > > > +/*
> > > > + * Scan the content of the FPGA bus, and the devices in the
> > > > +devices
> > > > + * list
> > > > + */
> > > > +static int
> > > > +ifpga_scan(void)
> > > > +{
> > > > + struct rte_ifpga_device *ifpga_dev;
> > > > + struct rte_devargs *devargs;
> > > > + struct rte_kvargs *kvlist = NULL;
> > > > + struct rte_rawdev *rawdev = NULL;
> > > > + char *name = NULL;
> > > > + char name1[RTE_RAWDEV_NAME_MAX_LEN];
> > > > + struct rte_afu_device *afu_dev = NULL;
> > > > +
> > > > + /* for FPGA devices we scan the devargs_list populated via
> > > > +cmdline */
> > >
> > > I didn't see "--ifpga" is supported by cmdline, either you need to
> > > add corresponding parser or correct the comment here.
> >
> > To be honestly, ifpga_scan() is called by hotplug, lib_rte hotplug
> > function will construct it.
>
> OK, after review patch 3, I saw it, that make sense.
> >
> > > > + RTE_EAL_DEVARGS_FOREACH("ifpga", devargs) {
> > >
> > >
> > > IFPGA_BUS_NAME to replace "ifpga" here.
> > >
> > > > + if (devargs->bus != &rte_ifpga_bus)
> > > > + continue;
> > > > +
> > > > + kvlist = rte_kvargs_parse(devargs->args, valid_args);
> > > > + if (!kvlist) {
> > > > + IFPGA_BUS_ERR("error when parsing param");
> > > > + goto end;
> > > > + }
> > > > +
> > > > + if (rte_kvargs_count(kvlist, IFPGA_ARG_NAME) == 1) {
> > > > + if (rte_kvargs_process(kvlist, IFPGA_ARG_NAME,
> > > > + &rte_ifpga_get_string_arg, &name) < 0) {
> > > > + IFPGA_BUS_ERR("error to parse %s",
> > > > + IFPGA_ARG_NAME);
> > > > + goto end;
> > > > + }
> > > > + } else {
> > > > + IFPGA_BUS_ERR("arg %s is mandatory for ifpga bus",
> > > > + IFPGA_ARG_NAME);
> > > > + goto end;
> > > > + }
> > > > +
> > > > + memset(name1, 0, sizeof(name1));
> > > > + snprintf(name1, RTE_RAWDEV_NAME_MAX_LEN,
> > > "IFPGA:%s",
> > > > name);
> > > > +
> > > > + rawdev = rte_rawdev_pmd_get_named_dev(name1);
> > > > + if (!rawdev)
> > > > + goto end;
> > > > +
> > > > + if (ifpga_find_ifpga_dev(rawdev))
> > > > + continue;
> > > > +
> > > > + ifpga_dev = calloc(1, sizeof(*ifpga_dev));
> > > > + if (!ifpga_dev)
> > > > + goto end;
> > > > +
> > > > + ifpga_dev->rdev = rawdev;
> > > > + TAILQ_INIT(&ifpga_dev->afu_list);
> > > > +
> > > > + TAILQ_INSERT_TAIL(&ifpga_device_list, ifpga_dev, next);
> > > > + afu_dev = ifpga_scan_one(devargs, ifpga_dev);
> > > > + if (afu_dev != NULL)
> > > > + TAILQ_INSERT_TAIL(&ifpga_dev->afu_list, afu_dev,
> > > next);
> > >
> > > I don't understand why we need afu_list?, seems we only add a new
> > > afu_dev into the list after create a new ifpga_dev, Is there any
> > > another place that we add one to list to make it reasonable? it
> > > looks like either a 1:1 or 1:0 for ifpga_dev:afu_dev?
> >
> > One FPAG may support more than one AFU.
>
> But in your implementation, it is not possible that an ifpga_dev contains
> multiple afu_dev
All afu_devs are listed to ifpga_dev, and each afu_dev can be added and deleted by user.
> >
> > > > + }
> > > > +
> > > > +end:
> > > > + if (kvlist)
> > > > + rte_kvargs_free(kvlist);
> > > > + if (name)
> > > > + free(name);
> > > > +
> > > > + return 0;
> > > > +}
> > > > +
> > > > +/*
> > > > + * Match the AFU Driver and AFU Device using the ID Table */
> > > > +static int rte_afu_match(const struct rte_afu_driver *afu_drv,
> > > > + const struct rte_afu_device *afu_dev) {
> > > > + const struct rte_afu_uuid *id_table;
> > > > +
> > > > + for (id_table = afu_drv->id_table;
> > > > + ((id_table->uuid_low != 0) && (id_table->uuid_high != 0));
> > > > + id_table++) {
> > > > + /* check if device's identifiers match the driver's ones */
> > > > + if ((id_table->uuid_low != afu_dev->id.uuid.uuid_low) ||
> > > > + id_table->uuid_high !=
> > > > + afu_dev->id.uuid.uuid_high)
> > > > + continue;
> > > > +
> > > > + return 1;
> > > > + }
> > > > +
> > > > + return 0;
> > > > +}
> > > > +
> > > > +static int
> > > > +ifpga_probe_one_driver(struct rte_afu_driver *drv,
> > > > + struct rte_afu_device *afu_dev) {
> > > > + int ret;
> > > > +
> > > > + if (!rte_afu_match(drv, afu_dev))
> > > > + /* Match of device and driver failed */
> > > > + return 1;
> > > > +
> > > > + /* reference driver structure */
> > > > + afu_dev->driver = drv;
> > > > + afu_dev->device.driver = &drv->driver;
> > > > +
> > > > + /* call the driver probe() function */
> > > > + ret = drv->probe(afu_dev);
> > > > + if (ret) {
> > > > + afu_dev->driver = NULL;
> > > > + afu_dev->device.driver = NULL;
> > > > + }
> > > > +
> > > > + return ret;
> > > > +}
> > > > +
> > > > +static int
> > > > +ifpga_probe_all_drivers(struct rte_afu_device *afu_dev) {
> > > > + struct rte_afu_driver *drv = NULL;
> > > > + int ret = 0;
> > > > +
> > > > + if (afu_dev == NULL)
> > > > + return -1;
> > > > +
> > > > + /* Check if a driver is already loaded */
> > > > + if (afu_dev->driver != NULL)
> > > > + return 0;
> > > > +
> > > > + TAILQ_FOREACH(drv, &afu_driver_list, next) {
> > > > + if (ifpga_probe_one_driver(drv, afu_dev)) {
> > > > + ret = -1;
> > > > + break;
> > > > + }
> > > > + }
> > > > + return ret;
> > > > +}
> > > > +
> > > > +/*
> > > > + * Scan the content of the Intel FPGA bus, and call the probe()
> > > > +function for
> > > > + * all registered drivers that have a matching entry in its
> > > > +id_table
> > > > + * for discovered devices.
> > > > + */
> > > > +static int
> > > > +ifpga_probe(void)
> > > > +{
> > > > + struct rte_ifpga_device *ifpga_dev;
> > > > + struct rte_afu_device *afu_dev = NULL;
> > > > + int ret = 0;
> > > > +
> > > > + TAILQ_FOREACH(ifpga_dev, &ifpga_device_list, next) {
> > > > + TAILQ_FOREACH(afu_dev, &ifpga_dev->afu_list, next) {
> > > > +
> > > > + if (afu_dev->device.driver)
> > > > + continue;
> > > > +
> > > > + ret = ifpga_probe_all_drivers(afu_dev);
> > > > + if (ret < 0)
> > > > + IFPGA_BUS_ERR("failed to initialize %s
> > > device\n",
> > > > + rte_ifpga_device_name(afu_dev));
> > > > + }
> > > > + }
> > > > +
> > > > + return ret;
> > > > +}
> > > > +
> > > > +static int
> > > > +ifpga_plug(struct rte_device *dev) {
> > > > + return ifpga_probe_all_drivers(RTE_DEV_TO_AFU(dev));
> > > > +}
> > > > +
> > > > +static int
> > > > +ifpga_remove_driver(struct rte_afu_device *afu_dev) {
> > > > + const char *name;
> > > > + const struct rte_afu_driver *driver;
> > > > +
> > > > + name = rte_ifpga_device_name(afu_dev);
> > > > + if (!afu_dev->device.driver) {
> > > > + IFPGA_BUS_DEBUG("no driver attach to device %s\n", name);
> > > > + return 1;
> > > > + }
> > > > +
> > > > + driver = RTE_DRV_TO_AFU_CONST(afu_dev->device.driver);
> > > > + return driver->remove(afu_dev);
> > > > +}
> > > > +
> > > > +static int
> > > > +ifpga_unplug(struct rte_device *dev) {
> > > > + struct rte_ifpga_device *ifpga_dev = NULL;
> > > > + struct rte_afu_device *afu_dev = NULL;
> > > > + struct rte_devargs *devargs = NULL;
> > > > + int ret;
> > > > +
> > > > + if (dev == NULL)
> > > > + return -EINVAL;
> > > > +
> > > > + afu_dev = RTE_DEV_TO_AFU(dev);
> > > > + if (!dev)
> > > > + return -ENOENT;
> > > > +
> > > > + ifpga_dev = afu_dev->ifpga_dev;
> > > > + devargs = dev->devargs;
> > > > +
> > > > + ret = ifpga_remove_driver(afu_dev);
> > >
> > > So what is the device type that be plugged into the ifgpa bus?
> > > ifpga_dev or afu_dev? If its afu_dev, why we need ifpga_dev?
> > > If it is ifpga_dev, we need remove ifpga_dev from ifpga_device_list here.
> >
> > The device type that be plugged into the ifpga bus is afu_dev.
>
> In DPDK, we have rte_pci_device for pci bus and rte_vdev_device for vdev
> bus Now we have rte_ifpga_device not for ifgpa bus but rte_afu_device, it is
> a little bit confuse here
The relationship of this items is descripted in ifpga_rawdev.rst.
> > And each afu_dev will be added into one ifpga_dev.
> > Each FPGA device bind to ifpga_dev.
>
> So what is the relationship between ifpga_dev and raw_dev?
One FPGA as a PCI device will probe ifpga_rawdev.
For management afu_dev, we create ifpga_dev in ifpga_bus.
> >
> > > > + if (ret)
> > > > + return ret;
> > > > +
> > > > + TAILQ_REMOVE(&ifpga_dev->afu_list, afu_dev, next);
> > > > +
> > > > + rte_devargs_remove(devargs->bus->name, devargs->name);
> > > > + free(afu_dev);
> > > > + return 0;
> > > > +
> > > > +}
> > > > +
> > > > +static struct rte_device *
> > > > +ifpga_find_device(const struct rte_device *start,
> > > > + rte_dev_cmp_t cmp, const void *data) {
> > > > + struct rte_ifpga_device *ifpga_dev;
> > > > + struct rte_afu_device *afu_dev;
> > > > +
> > > > + TAILQ_FOREACH(ifpga_dev, &ifpga_device_list, next) {
> > > > + TAILQ_FOREACH(afu_dev, &ifpga_dev->afu_list, next) {
> > > > + if (start && &afu_dev->device == start) {
> > > > + start = NULL;
> > > > + continue;
> > > > + }
> > > > + if (cmp(&afu_dev->device, data) == 0)
> > > > + return &afu_dev->device;
> > > > + }
> > > > + }
> > > > + return NULL;
> > > > +}
> > > > +static int
> > > > +ifpga_parse(const char *name, void *addr) {
> > > > + int *out = addr;
> > > > + struct rte_rawdev *rawdev = NULL;
> > > > + char rawdev_name[RTE_RAWDEV_NAME_MAX_LEN];
> > > > + char *c1 = NULL, *c2 = NULL;
> > > > + int port = IFPGA_BUS_DEV_PORT_MAX;
> > > > + char str_port[8];
> > > > + int str_port_len = 0;
> > > > + int ret;
> > > > +
> > > > + memset(str_port, 0, 8);
> > > > + c1 = strchr(name, '|');
> > > > + if (c1 != NULL) {
> > > > + str_port_len = c1-name;
> > > > + c2 = c1+1;
> > > > + }
> > > > +
> > > > + if (str_port_len < 8 &&
> > > > + str_port_len > 0) {
> > > > + memcpy(str_port, name, str_port_len);
> > > > + ret = sscanf(str_port, "%d", &port);
> > > > + if (ret == -1)
> > > > + return 0;
> > > > + }
> > > > +
> > > > + memset(rawdev_name, 0, sizeof(rawdev_name));
> > > > + snprintf(rawdev_name, RTE_RAWDEV_NAME_MAX_LEN, "IFPGA:%s",
> > > c2);
> > > > + rawdev = rte_rawdev_pmd_get_named_dev(rawdev_name);
> > > > +
> > > > + if ((port < IFPGA_BUS_DEV_PORT_MAX) &&
> > > > + rawdev &&
> > > > + (addr != NULL))
> > > > + *out = port;
> > > > +
> > > > + if ((port < IFPGA_BUS_DEV_PORT_MAX) &&
> > > > + rawdev)
> > > > + return 0;
> > > > + else
> > > > + return 1;
> > > > +}
> > > > +
> > > > +static struct rte_bus rte_ifpga_bus = {
> > > > + .scan = ifpga_scan,
> > > > + .probe = ifpga_probe,
> > > > + .find_device = ifpga_find_device,
> > > > + .plug = ifpga_plug,
> > > > + .unplug = ifpga_unplug,
> > > > + .parse = ifpga_parse,
> > > > +};
> > > > +
> > > > +RTE_REGISTER_BUS(IFPGA_BUS_NAME, rte_ifpga_bus);
> > > > +
> > > > +RTE_INIT(ifpga_init_log)
> > > > +{
> > > > + ifpga_bus_logtype = rte_log_register("bus.ifpga");
> > > > + if (ifpga_bus_logtype >= 0)
> > > > + rte_log_set_level(ifpga_bus_logtype, RTE_LOG_NOTICE); }
> > > > diff --git a/drivers/bus/ifpga/ifpga_common.c
> > > > b/drivers/bus/ifpga/ifpga_common.c
> > > > new file mode 100644
> > > > index 0000000..78e2eae
> > > > --- /dev/null
> > > > +++ b/drivers/bus/ifpga/ifpga_common.c
> > > > @@ -0,0 +1,88 @@
> > > > +/* SPDX-License-Identifier: BSD-3-Clause
> > > > + * Copyright(c) 2010-2018 Intel Corporation */
> > > > +
> > > > +#include <string.h>
> > > > +#include <inttypes.h>
> > > > +#include <stdint.h>
> > > > +#include <stdlib.h>
> > > > +#include <stdio.h>
> > > > +#include <sys/queue.h>
> > > > +#include <sys/mman.h>
> > > > +#include <sys/types.h>
> > > > +#include <unistd.h>
> > > > +#include <fcntl.h>
> > > > +
> > > > +#include <rte_errno.h>
> > > > +#include <rte_bus.h>
> > > > +#include <rte_per_lcore.h>
> > > > +#include <rte_memory.h>
> > > > +#include <rte_memzone.h>
> > > > +#include <rte_eal.h>
> > > > +#include <rte_common.h>
> > > > +
> > > > +#include <rte_devargs.h>
> > > > +#include <rte_kvargs.h>
> > > > +#include <rte_alarm.h>
> > > > +
> > > > +#include "rte_bus_ifpga.h"
> > > > +#include "ifpga_logs.h"
> > > > +#include "ifpga_common.h"
> > > > +
> > > > +int rte_ifpga_get_string_arg(const char *key __rte_unused,
> > > > + const char *value, void *extra_args) {
> > > > + if (!value || !extra_args)
> > > > + return -EINVAL;
> > > > +
> > > > + *(char **)extra_args = strdup(value);
> > > > +
> > > > + if (!*(char **)extra_args)
> > > > + return -ENOMEM;
> > > > +
> > > > + return 0;
> > > > +}
> > > > +int rte_ifpga_get_integer32_arg(const char *key __rte_unused,
> > > > + const char *value, void *extra_args) {
> > > > + if (!value || !extra_args)
> > > > + return -EINVAL;
> > > > +
> > > > + *(int *)extra_args = strtoull(value, NULL, 0);
> > > > +
> > > > + return 0;
> > > > +}
> > > > +int ifpga_get_integer64_arg(const char *key __rte_unused,
> > > > + const char *value, void *extra_args) {
> > > > + if (!value || !extra_args)
> > > > + return -EINVAL;
> > > > +
> > > > + *(uint64_t *)extra_args = strtoull(value, NULL, 0);
> > > > +
> > > > + return 0;
> > > > +}
> > > > +int ifpga_get_unsigned_long(const char *str, int base) {
> > > > + unsigned long num;
> > > > + char *end = NULL;
> > > > +
> > > > + errno = 0;
> > > > +
> > > > + num = strtoul(str, &end, base);
> > > > + if ((str[0] == '\0') || (end == NULL) || (*end != '\0') || (errno != 0))
> > > > + return -1;
> > > > +
> > > > + return num;
> > > > +}
> > > > +
> > > > +int ifpga_afu_id_cmp(const struct rte_afu_id *afu_id0,
> > > > + const struct rte_afu_id *afu_id1) {
> > > > + if ((afu_id0->uuid.uuid_low == afu_id1->uuid.uuid_low) &&
> > > > + (afu_id0->uuid.uuid_high == afu_id1->uuid.uuid_high) &&
> > > > + (afu_id0->port == afu_id1->port)) {
> > > > + return 0;
> > > > + } else
> > > > + return 1;
> > > > +}
> > > > diff --git a/drivers/bus/ifpga/ifpga_common.h
> > > > b/drivers/bus/ifpga/ifpga_common.h
> > > > new file mode 100644
> > > > index 0000000..f9254b9
> > > > --- /dev/null
> > > > +++ b/drivers/bus/ifpga/ifpga_common.h
> > > > @@ -0,0 +1,18 @@
> > > > +/* SPDX-License-Identifier: BSD-3-Clause
> > > > + * Copyright(c) 2010-2018 Intel Corporation */
> > > > +
> > > > +#ifndef _IFPGA_COMMON_H_
> > > > +#define _IFPGA_COMMON_H_
> > > > +
> > > > +int rte_ifpga_get_string_arg(const char *key __rte_unused,
> > > > + const char *value, void *extra_args); int
> > > > +rte_ifpga_get_integer32_arg(const char *key __rte_unused,
> > > > + const char *value, void *extra_args); int
> > > > +ifpga_get_integer64_arg(const char *key __rte_unused,
> > > > + const char *value, void *extra_args); int
> > > > +ifpga_get_unsigned_long(const char *str, int base); int
> > > > +ifpga_afu_id_cmp(const struct rte_afu_id *afu_id0,
> > > > + const struct rte_afu_id *afu_id1);
> > > > +
> > > > +#endif /* _IFPGA_COMMON_H_ */
> > > > diff --git a/drivers/bus/ifpga/ifpga_logs.h
> > > > b/drivers/bus/ifpga/ifpga_logs.h new file mode 100644 index
> > > > 0000000..873e0a4
> > > > --- /dev/null
> > > > +++ b/drivers/bus/ifpga/ifpga_logs.h
> > > > @@ -0,0 +1,31 @@
> > > > +/* SPDX-License-Identifier: BSD-3-Clause
> > > > + * Copyright(c) 2010-2018 Intel Corporation */
> > > > +
> > > > +#ifndef _IFPGA_LOGS_H_
> > > > +#define _IFPGA_LOGS_H_
> > > > +
> > > > +#include <rte_log.h>
> > > > +
> > > > +extern int ifpga_bus_logtype;
> > > > +
> > > > +#define IFPGA_LOG(level, fmt, args...) \
> > > > + rte_log(RTE_LOG_ ## level, ifpga_bus_logtype, "%s(): " fmt "\n", \
> > > > + __func__, ##args)
> > > > +
> > > > +#define IFPGA_BUS_LOG(level, fmt, args...) \
> > > > + rte_log(RTE_LOG_ ## level, ifpga_bus_logtype, "%s(): " fmt "\n", \
> > > > + __func__, ##args)
> > > > +
> > > > +#define IFPGA_BUS_FUNC_TRACE() IFPGA_BUS_LOG(DEBUG, ">>")
> > > > +
> > > > +#define IFPGA_BUS_DEBUG(fmt, args...) \
> > > > + IFPGA_BUS_LOG(DEBUG, fmt, ## args) #define IFPGA_BUS_INFO(fmt,
> > > > +args...) \
> > > > + IFPGA_BUS_LOG(INFO, fmt, ## args) #define IFPGA_BUS_ERR(fmt,
> > > > +args...) \
> > > > + IFPGA_BUS_LOG(ERR, fmt, ## args) #define IFPGA_BUS_WARN(fmt,
> > > > +args...) \
> > > > + IFPGA_BUS_LOG(WARNING, fmt, ## args)
> > > > +
> > > > +#endif /* _IFPGA_BUS_LOGS_H_ */
> > > > diff --git a/drivers/bus/ifpga/meson.build
> > > > b/drivers/bus/ifpga/meson.build new file mode 100644 index
> > > > 0000000..c9b08c8
> > > > --- /dev/null
> > > > +++ b/drivers/bus/ifpga/meson.build
> > > > @@ -0,0 +1,8 @@
> > > > +# SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2010-2018
> > > > +Intel Corporation
> > > > +
> > > > +deps += ['pci', 'kvargs', 'rawdev']
> > > > +install_headers('rte_bus_ifpga.h')
> > > > +sources = files('ifpga_common.c', 'ifpga_bus.c')
> > > > +
> > > > +allow_experimental_apis = true
> > > > diff --git a/drivers/bus/ifpga/rte_bus_ifpga.h
> > > > b/drivers/bus/ifpga/rte_bus_ifpga.h
> > > > new file mode 100644
> > > > index 0000000..5c559e1
> > > > --- /dev/null
> > > > +++ b/drivers/bus/ifpga/rte_bus_ifpga.h
> > > > @@ -0,0 +1,160 @@
> > > > +/* SPDX-License-Identifier: BSD-3-Clause
> > > > + * Copyright(c) 2010-2018 Intel Corporation */
> > > > +
> > > > +#ifndef _RTE_BUS_IFPGA_H_
> > > > +#define _RTE_BUS_IFPGA_H_
> > > > +
> > > > +/**
> > > > + * @file
> > > > + *
> > > > + * RTE Intel FPGA Bus Interface
> > > > + */
> > > > +
> > > > +#ifdef __cplusplus
> > > > +extern "C" {
> > > > +#endif
> > > > +
> > > > +#include <rte_bus.h>
> > > > +#include <rte_pci.h>
> > > > +
> > > > +/** Name of Intel FPGA Bus */
> > > > +#define IFPGA_BUS_NAME ifpga
> > > > +
> > > > +/* Forward declarations */
> > > > +struct rte_afu_device;
> > > > +
> > > > +/** List of Intel AFU devices */
> > > > +TAILQ_HEAD(afu_device_list, rte_afu_device);
> > > > +/** Double linked list of AFU device drivers. */
> > > > +TAILQ_HEAD(afu_driver_list, rte_afu_driver);
> > > > +
> > > > +#define IFPGA_BUS_BITSTREAM_PATH_MAX_LEN 256
> > > > +
> > > > +struct rte_afu_uuid {
> > > > + uint64_t uuid_low;
> > > > + uint64_t uuid_high;
> > > > +} __attribute__ ((packed));
> > > > +
> > > > +#define IFPGA_BUS_DEV_PORT_MAX 4
> > > > +
> > > > +/**
> > > > + * A structure describing an ID for a AFU driver. Each driver
> > > > +provides a
> > > > + * table of these IDs for each device that it supports.
> > > > + */
> > > > +struct rte_afu_id {
> > > > + struct rte_afu_uuid uuid;
> > > > + int port; /**< port number */
> > > > +} __attribute__ ((packed));
> > > > +
> > > > +/**
> > > > + * A structure PR (Partial Reconfiguration) configuration AFU driver.
> > > > + */
> > > > +
> > > > +struct rte_afu_pr_conf {
> > > > + struct rte_afu_id afu_id;
> > > > + int pr_enable;
> > > > + char bs_path[IFPGA_BUS_BITSTREAM_PATH_MAX_LEN];
> > > > +};
> > > > +
> > > > +#define AFU_PRI_STR_SIZE (PCI_PRI_STR_SIZE + 8)
> > > > +
> > > > +/**
> > > > + * A structure describing a fpga device.
> > > > + */
> > > > +struct rte_ifpga_device {
> > > > + TAILQ_ENTRY(rte_ifpga_device) next; /**< Next in device list. */
> > > > + struct rte_rawdev *rdev;
> > > > + struct afu_device_list afu_list; /**< List of AFU devices */ };
> > > > +
> > > > +/**
> > > > + * A structure describing a AFU device.
> > > > + */
> > > > +struct rte_afu_device {
> > > > + TAILQ_ENTRY(rte_afu_device) next; /**< Next in device list. */
> > > > + struct rte_device device; /**< Inherit core device */
> > > > + struct rte_rawdev *rawdev; /**< Point Rawdev */
> > > > + struct rte_ifpga_device *ifpga_dev; /**< Point ifpga device */
> > > > + struct rte_afu_id id; /**< AFU id within FPGA. */
> > > > + uint32_t num_region; /**< number of regions found */
> > > > + struct rte_mem_resource mem_resource[PCI_MAX_RESOURCE];
> > > > + /**< AFU Memory Resource
> > > */
> > > > + struct rte_intr_handle intr_handle; /**< Interrupt handle */
> > > > + struct rte_afu_driver *driver; /**< Associated driver */
> > > > + char path[IFPGA_BUS_BITSTREAM_PATH_MAX_LEN];
> > > > +} __attribute__ ((packed));
> > > > +
> > > > +/**
> > > > + * @internal
> > > > + * Helper macro for drivers that need to convert to struct
> rte_afu_device.
> > > > + */
> > > > +#define RTE_DEV_TO_AFU(ptr) \
> > > > + container_of(ptr, struct rte_afu_device, device)
> > > > +
> > > > +#define RTE_DRV_TO_AFU_CONST(ptr) \
> > > > + container_of(ptr, const struct rte_afu_driver, driver)
> > > > +
> > > > +/**
> > > > + * Initialisation function for the driver called during FPGA BUS probing.
> > >
> > >
> > > typo: Initialization
> >
> > Fixed.
> >
> > > > + */
> > > > +typedef int (afu_probe_t)(struct rte_afu_device *);
> > > > +
> > > > +/**
> > > > + * Uninitialisation function for the driver called during hotplugging.
> > >
> > > typo: Unintialization
> >
> > Fixed.
> >
> > > > + */
> > > > +typedef int (afu_remove_t)(struct rte_afu_device *);
> > > > +
> > > > +/**
> > > > + * A structure describing a AFU device.
> > > > + */
> > > > +struct rte_afu_driver {
> > > > + TAILQ_ENTRY(rte_afu_driver) next; /**< Next afu driver. */
> > > > + struct rte_driver driver; /**< Inherit core driver. */
> > > > + afu_probe_t *probe; /**< Device Probe function.
> > > > */
> > > > + afu_remove_t *remove; /**< Device Remove
> > > > function. */
> > > > + const struct rte_afu_uuid *id_table; /**< AFU uuid within FPGA. */
> > > > + uint32_t drv_flags; /**< Flags contolling handling of device. */
> > >
> > > Typo Controlling
> > > And what is "flags controlling handling of device?"
> >
> > This variable is not used, so I have removed it.
> >
> > > > +};
> > > > +
> > > > +static inline const char *
> > > > +rte_ifpga_device_name(const struct rte_afu_device *afu) {
> > > > + if (afu && afu->device.name)
> > > > + return afu->device.name;
> > > > + return NULL;
> > > > +}
> > > > +
> > > > +/**
> > > > + * Register a ifpga afu device driver.
> > > > + *
> > > > + * @param driver
> > > > + * A pointer to a rte_afu_driver structure describing the driver
> > > > + * to be registered.
> > > > + */
> > > > +void rte_ifpga_driver_register(struct rte_afu_driver *driver);
> > > > +
> > > > +/**
> > > > + * Unregister a ifpga afu device driver.
> > > > + *
> > > > + * @param driver
> > > > + * A pointer to a rte_afu_driver structure describing the driver
> > > > + * to be unregistered.
> > > > + */
> > > > +void rte_ifpga_driver_unregister(struct rte_afu_driver *driver);
> > > > +
> > > > +#define RTE_PMD_REGISTER_AFU(nm, afudrv)\ RTE_INIT(afudrvinitfn_
> > > > +##afudrv);\ static const char *afudrvinit_ ## nm ## _alias;\
> > > > +static void afudrvinitfn_ ##afudrv(void)\ {\
> > > > + (afudrv).driver.name = RTE_STR(nm);\
> > > > + (afudrv).driver.alias = afudrvinit_ ## nm ## _alias;\
> > > > + rte_ifpga_driver_register(&afudrv);\
> > > > +} \
> > > > +RTE_PMD_EXPORT_NAME(nm, __COUNTER__)
> > > > +
> > > > +#define RTE_PMD_REGISTER_AFU_ALIAS(nm, alias)\ static const char
> > > > +*afudrvinit_ ## nm ## _alias = RTE_STR(alias)
> > > > +
> > > > +#endif /* _RTE_BUS_IFPGA_H_ */
> > > > diff --git a/drivers/bus/ifpga/rte_bus_ifpga_version.map
> > > > b/drivers/bus/ifpga/rte_bus_ifpga_version.map
> > > > new file mode 100644
> > > > index 0000000..a027979
> > > > --- /dev/null
> > > > +++ b/drivers/bus/ifpga/rte_bus_ifpga_version.map
> > > > @@ -0,0 +1,10 @@
> > > > +DPDK_18.05 {
> > > > + global:
> > > > +
> > > > + rte_ifpga_get_integer32_arg;
> > > > + rte_ifpga_get_string_arg;
> > > > + rte_ifpga_driver_register;
> > > > + rte_ifpga_driver_unregister;
> > > > +
> > > > + local: *;
> > > > +};
> > > > diff --git a/drivers/bus/meson.build b/drivers/bus/meson.build
> > > > index 58dfbe2..52c755d 100644
> > > > --- a/drivers/bus/meson.build
> > > > +++ b/drivers/bus/meson.build
> > > > @@ -1,7 +1,7 @@
> > > > # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2017
> > > > Intel Corporation
> > > >
> > > > -drivers = ['dpaa', 'fslmc', 'pci', 'vdev']
> > > > +drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'vdev']
> > > > std_deps = ['eal']
> > > > config_flag_fmt = 'RTE_LIBRTE_@0@_BUS'
> > > > driver_name_fmt = 'rte_bus_@0@'
> > > > diff --git a/mk/rte.app.mk b/mk/rte.app.mk index 26f3563..3861e1a
> > > > 100644
> > > > --- a/mk/rte.app.mk
> > > > +++ b/mk/rte.app.mk
> > > > @@ -255,6 +255,7 @@ ifeq
> > > > ($(CONFIG_RTE_EAL_VFIO)$(CONFIG_RTE_LIBRTE_FSLMC_BUS),yy)
> > > > _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_CMDIF_RAWDEV) +=
> > > > -lrte_pmd_dpaa2_cmdif
> > > > _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV) +=
> > > > -lrte_pmd_dpaa2_qdma endif # CONFIG_RTE_LIBRTE_FSLMC_BUS
> > > > +_LDLIBS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += -lrte_bus_ifpga
> > > > endif # CONFIG_RTE_LIBRTE_RAWDEV
> > > >
> > > >
> > > > --
> > > > 1.8.3.1
> > >
> > > Regards
> > > Qi
> -----Original Message-----
> From: Xu, Rosen
> Sent: Thursday, May 10, 2018 9:51 PM
> To: Xu, Rosen <rosen.xu@intel.com>; Zhang, Qi Z <qi.z.zhang@intel.com>;
> dev@dpdk.org; thomas@monjalon.net
> Cc: Zhang, Roy Fan <roy.fan.zhang@intel.com>; Doherty, Declan
> <declan.doherty@intel.com>; Richardson, Bruce
> <bruce.richardson@intel.com>; shreyansh.jain@nxp.com; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Ananyev, Konstantin
> <konstantin.ananyev@intel.com>; Zhang, Tianfei <tianfei.zhang@intel.com>;
> Liu, Song <song.liu@intel.com>; Wu, Hao <hao.wu@intel.com>;
> gaetan.rivet@6wind.com
> Subject: RE: [dpdk-dev] [PATCH v10 1/3] bus/ifpga: Add Intel FPGA BUS Library
>
> Hi Qi,
>
> I miss one comment, so I fix it in this email.
>
> > -----Original Message-----
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Xu, Rosen
> > Sent: Thursday, May 10, 2018 21:29
> > To: Zhang, Qi Z <qi.z.zhang@intel.com>; dev@dpdk.org;
> > thomas@monjalon.net
> > Cc: Zhang, Roy Fan <roy.fan.zhang@intel.com>; Doherty, Declan
> > <declan.doherty@intel.com>; Richardson, Bruce
> > <bruce.richardson@intel.com>; shreyansh.jain@nxp.com; Yigit, Ferruh
> > <ferruh.yigit@intel.com>; Ananyev, Konstantin
> > <konstantin.ananyev@intel.com>; Zhang, Tianfei
> > <tianfei.zhang@intel.com>; Liu, Song <song.liu@intel.com>; Wu, Hao
> > <hao.wu@intel.com>; gaetan.rivet@6wind.com
> > Subject: Re: [dpdk-dev] [PATCH v10 1/3] bus/ifpga: Add Intel FPGA BUS
> > Library
> >
> > Hi Qi,
> >
> > > -----Original Message-----
> > > From: Zhang, Qi Z
> > > Sent: Thursday, May 10, 2018 20:27
> > > To: Xu, Rosen <rosen.xu@intel.com>; dev@dpdk.org;
> > thomas@monjalon.net
> > > Cc: Xu, Rosen <rosen.xu@intel.com>; Zhang, Roy Fan
> > > <roy.fan.zhang@intel.com>; Doherty, Declan
> > > <declan.doherty@intel.com>; Richardson, Bruce
> > > <bruce.richardson@intel.com>; shreyansh.jain@nxp.com; Yigit, Ferruh
> > > <ferruh.yigit@intel.com>; Ananyev, Konstantin
> > > <konstantin.ananyev@intel.com>; Zhang, Tianfei
> > > <tianfei.zhang@intel.com>; Liu, Song <song.liu@intel.com>; Wu, Hao
> > > <hao.wu@intel.com>; gaetan.rivet@6wind.com
> > > Subject: RE: [dpdk-dev] [PATCH v10 1/3] bus/ifpga: Add Intel FPGA
> > > BUS Library
> > >
> > > Hi Rosen:
> > >
> > > > -----Original Message-----
> > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Xu, Rosen
> > > > Sent: Wednesday, May 9, 2018 3:43 PM
> > > > To: dev@dpdk.org; thomas@monjalon.net
> > > > Cc: Xu, Rosen <rosen.xu@intel.com>; Zhang, Roy Fan
> > > > <roy.fan.zhang@intel.com>; Doherty, Declan
> > > > <declan.doherty@intel.com>; Richardson, Bruce
> > > > <bruce.richardson@intel.com>; shreyansh.jain@nxp.com; Yigit,
> > > > Ferruh <ferruh.yigit@intel.com>; Ananyev, Konstantin
> > > > <konstantin.ananyev@intel.com>; Zhang, Tianfei
> > > > <tianfei.zhang@intel.com>; Liu, Song <song.liu@intel.com>; Wu, Hao
> > > > <hao.wu@intel.com>; gaetan.rivet@6wind.com
> > > > Subject: [dpdk-dev] [PATCH v10 1/3] bus/ifpga: Add Intel FPGA BUS
> > > > Library
> > > >
> > > > From: Rosen Xu <rosen.xu@intel.com>
> > > >
> > > > Defined FPGA-BUS for Acceleration Drivers of AFUs
> > > >
> > > > 1. FPGA PCI Scan (1st Scan) follows DPDK UIO/VFIO PCI Scan
> > > > Process, probe Intel FPGA Rawdev Driver, it will be covered in following
> patches.
> > > >
> > > > 2. AFU Scan(2nd Scan) bind DPDK driver to FPGA Partial-Bitstream.
> > > > This scan is trigged by hotplug of IFPGA Rawdev probe, in this
> > > > scan the AFUs will be created and their drivers are also probed.
> > > >
> > > > This patch will introduce rte_afu_device which describe the AFU
> > > > device listed in the FPGA-BUS.
> > > >
> > > > Signed-off-by: Rosen Xu <rosen.xu@intel.com>
> > > > Signed-off-by: Tianfei Zhang <tianfei.zhang@intel.com>
> > > > ---
> > > > MAINTAINERS | 4 +
> > > > config/common_base | 5 +
> > > > doc/guides/rel_notes/release_18_05.rst | 5 +
> > > > drivers/bus/Makefile | 1 +
> > > > drivers/bus/ifpga/Makefile | 32 ++
> > > > drivers/bus/ifpga/ifpga_bus.c | 501
> > > > ++++++++++++++++++++++++++++
> > > > drivers/bus/ifpga/ifpga_common.c | 88 +++++
> > > > drivers/bus/ifpga/ifpga_common.h | 18 +
> > > > drivers/bus/ifpga/ifpga_logs.h | 31 ++
> > > > drivers/bus/ifpga/meson.build | 8 +
> > > > drivers/bus/ifpga/rte_bus_ifpga.h | 160 +++++++++
> > > > drivers/bus/ifpga/rte_bus_ifpga_version.map | 10 +
> > > > drivers/bus/meson.build | 2 +-
> > > > mk/rte.app.mk | 1 +
> > > > 14 files changed, 865 insertions(+), 1 deletion(-) create mode
> > > > 100644 drivers/bus/ifpga/Makefile create mode 100644
> > > > drivers/bus/ifpga/ifpga_bus.c create mode 100644
> > > > drivers/bus/ifpga/ifpga_common.c create mode 100644
> > > > drivers/bus/ifpga/ifpga_common.h create mode 100644
> > > > drivers/bus/ifpga/ifpga_logs.h create mode 100644
> > > > drivers/bus/ifpga/meson.build create mode 100644
> > > > drivers/bus/ifpga/rte_bus_ifpga.h create mode 100644
> > > > drivers/bus/ifpga/rte_bus_ifpga_version.map
> > > >
> > > > diff --git a/MAINTAINERS b/MAINTAINERS index 7105920..fa0c5b1
> > > > 100644
> > > > --- a/MAINTAINERS
> > > > +++ b/MAINTAINERS
> > > > @@ -383,6 +383,10 @@ F: drivers/mempool/bucket/ Bus Drivers
> > > > -----------
> > > >
> > > > +Intel FPGA buses
> > > > +M: Rosen Xu <rosen.xu@intel.com>
> > > > +F: drivers/bus/ifpga/
> > > > +
> > > > NXP buses
> > > > M: Hemant Agrawal <hemant.agrawal@nxp.com>
> > > > M: Shreyansh Jain <shreyansh.jain@nxp.com> diff --git
> > > > a/config/common_base b/config/common_base index 0d181ac..1440316
> > > > 100644
> > > > --- a/config/common_base
> > > > +++ b/config/common_base
> > > > @@ -139,6 +139,11 @@
> > > > CONFIG_RTE_ETHDEV_PROFILE_ITT_WASTED_RX_ITERATIONS=n
> > > > CONFIG_RTE_ETHDEV_TX_PREPARE_NOOP=n
> > > >
> > > > #
> > > > +# Compile the Intel FPGA bus
> > > > +#
> > > > +CONFIG_RTE_LIBRTE_IFPGA_BUS=y
> > > > +
> > > > +#
> > > > # Compile PCI bus driver
> > > > #
> > > > CONFIG_RTE_LIBRTE_PCI_BUS=y
> > > > diff --git a/doc/guides/rel_notes/release_18_05.rst
> > > > b/doc/guides/rel_notes/release_18_05.rst
> > > > index 7187348..265950a 100644
> > > > --- a/doc/guides/rel_notes/release_18_05.rst
> > > > +++ b/doc/guides/rel_notes/release_18_05.rst
> > > > @@ -183,6 +183,11 @@ New Features
> > > > stats/xstats on shared memory from secondary process, and also
> > > > pdump packets on
> > > > those virtual devices.
> > > >
> > > > +* **Added Ifpga Bus, a generic Intel FPGA Bus library.**
> > > > +
> > > > + The Ifpga Bus library provides support for integrating any
> > > > + Intel FPGA device with the DPDK framework. It provides Intel
> > > > + FPGA Partial Bit Stream AFU(Accelerated Function Unit) scan and
> > > > + drivers
> > prove.
> > > >
> > > > API Changes
> > > > -----------
> > > > diff --git a/drivers/bus/Makefile b/drivers/bus/Makefile index
> > > > c251b65..ef7f247 100644
> > > > --- a/drivers/bus/Makefile
> > > > +++ b/drivers/bus/Makefile
> > > > @@ -7,6 +7,7 @@ DIRS-$(CONFIG_RTE_LIBRTE_DPAA_BUS) += dpaa
> ifeq
> > > > ($(CONFIG_RTE_EAL_VFIO),y)
> > > > DIRS-$(CONFIG_RTE_LIBRTE_FSLMC_BUS) += fslmc endif
> > > > +DIRS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += ifpga
> > > > DIRS-$(CONFIG_RTE_LIBRTE_PCI_BUS) += pci
> > > > DIRS-$(CONFIG_RTE_LIBRTE_VDEV_BUS) += vdev
> > > >
> > > > diff --git a/drivers/bus/ifpga/Makefile
> > > > b/drivers/bus/ifpga/Makefile new file mode 100644 index
> > > > 0000000..3ff3bdb
> > > > --- /dev/null
> > > > +++ b/drivers/bus/ifpga/Makefile
> > > > @@ -0,0 +1,32 @@
> > > > +# SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2018 Intel
> > > > +Corporation
> > > > +
> > > > +include $(RTE_SDK)/mk/rte.vars.mk
> > > > +
> > > > +#
> > > > +# library name
> > > > +#
> > > > +LIB = librte_bus_ifpga.a
> > > > +
> > > > +CFLAGS += -DALLOW_EXPERIMENTAL_API CFLAGS += -O3 CFLAGS +=
> > > > +$(WERROR_FLAGS) LDLIBS += -lrte_eal LDLIBS += -lrte_rawdev LDLIBS
> > > > ++= -lrte_kvargs
> > > > +
> > > > +# versioning export map
> > > > +EXPORT_MAP := rte_bus_ifpga_version.map
> > > > +
> > > > +# library version
> > > > +LIBABIVER := 1
> > > > +
> > > > +SRCS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += ifpga_bus.c
> > > > +SRCS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += ifpga_common.c
> > > > +
> > > > +#
> > > > +# Export include files
> > > > +#
> > > > +SYMLINK-$(CONFIG_RTE_LIBRTE_IFPGA_BUS)-include += rte_bus_ifpga.h
> > > > +
> > > > +include $(RTE_SDK)/mk/rte.lib.mk
> > > > diff --git a/drivers/bus/ifpga/ifpga_bus.c
> > > > b/drivers/bus/ifpga/ifpga_bus.c new file mode 100644 index
> > > > 0000000..e144c01
> > > > --- /dev/null
> > > > +++ b/drivers/bus/ifpga/ifpga_bus.c
> > > > @@ -0,0 +1,501 @@
> > > > +/* SPDX-License-Identifier: BSD-3-Clause
> > > > + * Copyright(c) 2010-2018 Intel Corporation */
> > > > +
> > > > +#include <string.h>
> > > > +#include <inttypes.h>
> > > > +#include <stdint.h>
> > > > +#include <stdlib.h>
> > > > +#include <stdio.h>
> > > > +#include <sys/queue.h>
> > > > +#include <sys/mman.h>
> > > > +#include <sys/types.h>
> > > > +#include <unistd.h>
> > > > +#include <fcntl.h>
> > > > +
> > > > +#include <rte_errno.h>
> > > > +#include <rte_bus.h>
> > > > +#include <rte_per_lcore.h>
> > > > +#include <rte_memory.h>
> > > > +#include <rte_memzone.h>
> > > > +#include <rte_eal.h>
> > > > +#include <rte_common.h>
> > > > +
> > > > +#include <rte_devargs.h>
> > > > +#include <rte_kvargs.h>
> > > > +#include <rte_alarm.h>
> > > > +
> > > > +#include "rte_rawdev.h"
> > > > +#include "rte_rawdev_pmd.h"
> > > > +#include "rte_bus_ifpga.h"
> > > > +#include "ifpga_logs.h"
> > > > +#include "ifpga_common.h"
> > > > +
> > > > +int ifpga_bus_logtype;
> > > > +
> > > > +/* Forward declaration to access Intel FPGA bus
> > > > + * on which iFPGA devices are connected */ static struct rte_bus
> > > > +rte_ifpga_bus;
> > > > +
> > > > +/** Double linked list of IFPGA device. */
> > > > +TAILQ_HEAD(ifpga_device_list, rte_ifpga_device);
> > > > +
> > > > +static struct ifpga_device_list ifpga_device_list =
> > > > + TAILQ_HEAD_INITIALIZER(ifpga_device_list);
> > > > +static struct afu_driver_list afu_driver_list =
> > > > + TAILQ_HEAD_INITIALIZER(afu_driver_list);
> > > > +
> > > > +
> > > > +/* register a ifpga bus based driver */ void
> > > > +rte_ifpga_driver_register(struct rte_afu_driver *driver) {
> > > > + RTE_VERIFY(driver);
> > > > +
> > > > + TAILQ_INSERT_TAIL(&afu_driver_list, driver, next); }
> > > > +
> > > > +/* un-register a fpga bus based driver */ void
> > > > +rte_ifpga_driver_unregister(struct rte_afu_driver *driver) {
> > > > + TAILQ_REMOVE(&afu_driver_list, driver, next); }
> > > > +
> > > > +static struct rte_ifpga_device *
> > > > +ifpga_find_ifpga_dev(const struct rte_rawdev *rdev) {
> > > > + struct rte_ifpga_device *ifpga_dev = NULL;
> > > > +
> > > > + TAILQ_FOREACH(ifpga_dev, &ifpga_device_list, next) {
> > > > + if (rdev &&
> > > > + ifpga_dev->rdev &&
> > > > + ifpga_dev->rdev == rdev)
> > > > + return ifpga_dev;
> > > > + }
> > > > + return NULL;
> > > > +}
> > > > +
> > > > +static struct rte_afu_device *
> > > > +ifpga_find_afu_dev(const struct rte_ifpga_device *ifpga_dev,
> > > > + const struct rte_afu_id *afu_id) {
> > > > + struct rte_afu_device *afu_dev = NULL;
> > > > +
> > > > + TAILQ_FOREACH(afu_dev, &ifpga_dev->afu_list, next) {
> > > > + if (!ifpga_afu_id_cmp(&afu_dev->id, afu_id))
> > > > + return afu_dev;
> > > > + }
> > > > + return NULL;
> > > > +}
> > > > +
> > > > +static const char * const valid_args[] = {
> > > > +#define IFPGA_ARG_NAME "ifpga"
> > > > + IFPGA_ARG_NAME,
> > > > +#define IFPGA_ARG_PORT "port"
> > > > + IFPGA_ARG_PORT,
> > > > +#define IFPGA_AFU_BTS "afu_bts"
> > > > + IFPGA_AFU_BTS,
> > > > + NULL
> > > > +};
> > > > +
> > > > +/*
> > > > + * Scan the content of the FPGA bus, and the devices in the
> > > > +devices
> > > > + * list
> > > > + */
> > > > +static struct rte_afu_device *
> > > > +ifpga_scan_one(struct rte_devargs *devargs,
> > > > + struct rte_ifpga_device *ifpga_dev)
> > >
> > > usually ifpag_dev should be the first parameter here, and devargs
> > > follows
> >
> > Fixed.
> >
> > > > +{
> > > > + struct rte_kvargs *kvlist = NULL;
> > > > + struct rte_rawdev *rawdev = NULL;
> > > > + struct rte_afu_device *afu_dev = NULL;
> > > > + struct rte_afu_pr_conf afu_pr_conf;
> > > > + int ret = 0;
> > > > + char *path = NULL;
> > > > +
> > > > + memset(&afu_pr_conf, 0, sizeof(struct rte_afu_pr_conf));
> > > > +
> > > > + kvlist = rte_kvargs_parse(devargs->args, valid_args);
> > > > + if (!kvlist) {
> > > > + IFPGA_BUS_ERR("error when parsing param");
> > > > + goto end;
> > > > + }
> > > > +
> > > > + if (rte_kvargs_count(kvlist, IFPGA_ARG_PORT) == 1) {
> > > > + if (rte_kvargs_process(kvlist, IFPGA_ARG_PORT,
> > > > + &rte_ifpga_get_integer32_arg, &afu_pr_conf.afu_id.port) < 0)
> > > {
> > > > + IFPGA_BUS_ERR("error to parse %s",
> > > > + IFPGA_ARG_PORT);
> > > > + goto end;
> > > > + }
> > > > + } else {
> > > > + IFPGA_BUS_ERR("arg %s is mandatory for ifpga bus",
> > > > + IFPGA_ARG_PORT);
> > > > + goto end;
> > > > + }
> > > > +
> > > > + if (rte_kvargs_count(kvlist, IFPGA_AFU_BTS) == 1) {
> > > > + if (rte_kvargs_process(kvlist, IFPGA_AFU_BTS,
> > > > + &rte_ifpga_get_string_arg, &path) < 0) {
> > > > + IFPGA_BUS_ERR("Failed to parse %s",
> > > > + IFPGA_AFU_BTS);
> > > > + goto end;
> > > > + }
> > > > + } else {
> > > > + IFPGA_BUS_ERR("arg %s is mandatory for ifpga bus",
> > > > + IFPGA_AFU_BTS);
> > > > + goto end;
> > > > + }
> > > > +
> > > > + afu_pr_conf.afu_id.uuid.uuid_low = 0;
> > > > + afu_pr_conf.afu_id.uuid.uuid_high = 0;
> > > > + afu_pr_conf.pr_enable = path?1:0;
> > > > +
> > > > + rawdev = ifpga_dev->rdev;
> > > > + if (ifpga_find_afu_dev(ifpga_dev, &afu_pr_conf.afu_id))
> > > > + goto end;
> > > > +
> > > > + afu_dev = calloc(1, sizeof(*afu_dev));
> > > > + if (!afu_dev)
> > > > + goto end;
> > > > +
> > > > + afu_dev->device.devargs = devargs;
> > > > + afu_dev->device.numa_node = SOCKET_ID_ANY;
> > > > + afu_dev->device.name = devargs->name;
> > > > + afu_dev->rawdev = rawdev;
> > > > + afu_dev->id.uuid.uuid_low = 0;
> > > > + afu_dev->id.uuid.uuid_high = 0;
> > > > + afu_dev->id.port = afu_pr_conf.afu_id.port;
> > > > + afu_dev->ifpga_dev = ifpga_dev;
> > > > +
> > > > + if (rawdev->dev_ops && rawdev->dev_ops->dev_info_get)
> > > > + rawdev->dev_ops->dev_info_get(rawdev, afu_dev);
> > > > +
> > > > + if (rawdev->dev_ops &&
> > > > + rawdev->dev_ops->dev_start &&
> > > > + rawdev->dev_ops->dev_start(rawdev))
> > > > + goto free_dev;
> > > > +
> > > > + strncpy(afu_pr_conf.bs_path, path, sizeof(afu_pr_conf.bs_path));
> > > > + if (rawdev->dev_ops->firmware_load &&
> > > > + rawdev->dev_ops->firmware_load(rawdev,
> > > > + &afu_pr_conf)){
> > > > + IFPGA_BUS_ERR("firmware load error %d\n", ret);
> > > > + goto free_dev;
> > > > + }
> > > > + afu_dev->id.uuid.uuid_low = afu_pr_conf.afu_id.uuid.uuid_low;
> > > > + afu_dev->id.uuid.uuid_high = afu_pr_conf.afu_id.uuid.uuid_high;
> > > > +
> > > > + return afu_dev;
> > > > +
> > > > +free_dev:
> > > > + free(afu_dev);
> > > > +end:
> > > > + if (kvlist)
> > > > + rte_kvargs_free(kvlist);
> > > > + if (path)
> > > > + free(path);
> > > > +
> > > > + return NULL;
> > > > +}
> > > > +
> > > > +/*
> > > > + * Scan the content of the FPGA bus, and the devices in the
> > > > +devices
> > > > + * list
> > > > + */
> > > > +static int
> > > > +ifpga_scan(void)
> > > > +{
> > > > + struct rte_ifpga_device *ifpga_dev;
> > > > + struct rte_devargs *devargs;
> > > > + struct rte_kvargs *kvlist = NULL;
> > > > + struct rte_rawdev *rawdev = NULL;
> > > > + char *name = NULL;
> > > > + char name1[RTE_RAWDEV_NAME_MAX_LEN];
> > > > + struct rte_afu_device *afu_dev = NULL;
> > > > +
> > > > + /* for FPGA devices we scan the devargs_list populated via
> > > > +cmdline */
> > >
> > > I didn't see "--ifpga" is supported by cmdline, either you need to
> > > add corresponding parser or correct the comment here.
> >
> > To be honestly, ifpga_scan() is called by hotplug, lib_rte hotplug
> > function will construct it.
> >
> > > > + RTE_EAL_DEVARGS_FOREACH("ifpga", devargs) {
> > >
> > >
> > > IFPGA_BUS_NAME to replace "ifpga" here.
>
> IFPGA_BUS_NAME is used in some Macros, so I use IFPGA_ARG_NAME.
Though they have the same value, but different meanings, the first argument of RTE_EAL_DEVARGS_FOREACH actually is "busname", we should parse bus name here.
Though not a big deal, but I don't know why we can't use IFPGA_BUS_NAME?
>
> > > > + if (devargs->bus != &rte_ifpga_bus)
> > > > + continue;
> > > > +
> > > > + kvlist = rte_kvargs_parse(devargs->args, valid_args);
> > > > + if (!kvlist) {
> > > > + IFPGA_BUS_ERR("error when parsing param");
> > > > + goto end;
> > > > + }
> > > > +
> > > > + if (rte_kvargs_count(kvlist, IFPGA_ARG_NAME) == 1) {
> > > > + if (rte_kvargs_process(kvlist, IFPGA_ARG_NAME,
> > > > + &rte_ifpga_get_string_arg, &name) < 0) {
> > > > + IFPGA_BUS_ERR("error to parse %s",
> > > > + IFPGA_ARG_NAME);
> > > > + goto end;
> > > > + }
> > > > + } else {
> > > > + IFPGA_BUS_ERR("arg %s is mandatory for ifpga bus",
> > > > + IFPGA_ARG_NAME);
> > > > + goto end;
> > > > + }
> > > > +
> > > > + memset(name1, 0, sizeof(name1));
> > > > + snprintf(name1, RTE_RAWDEV_NAME_MAX_LEN,
> > > "IFPGA:%s",
> > > > name);
> > > > +
> > > > + rawdev = rte_rawdev_pmd_get_named_dev(name1);
> > > > + if (!rawdev)
> > > > + goto end;
> > > > +
> > > > + if (ifpga_find_ifpga_dev(rawdev))
> > > > + continue;
> > > > +
> > > > + ifpga_dev = calloc(1, sizeof(*ifpga_dev));
> > > > + if (!ifpga_dev)
> > > > + goto end;
> > > > +
> > > > + ifpga_dev->rdev = rawdev;
> > > > + TAILQ_INIT(&ifpga_dev->afu_list);
> > > > +
> > > > + TAILQ_INSERT_TAIL(&ifpga_device_list, ifpga_dev, next);
> > > > + afu_dev = ifpga_scan_one(devargs, ifpga_dev);
> > > > + if (afu_dev != NULL)
> > > > + TAILQ_INSERT_TAIL(&ifpga_dev->afu_list, afu_dev,
> > > next);
> > >
> > > I don't understand why we need afu_list?, seems we only add a new
> > > afu_dev into the list after create a new ifpga_dev, Is there any
> > > another place that we add one to list to make it reasonable? it
> > > looks like either a 1:1 or 1:0 for ifpga_dev:afu_dev?
> >
> > One FPAG may support more than one AFU.
> >
> > > > + }
> > > > +
> > > > +end:
> > > > + if (kvlist)
> > > > + rte_kvargs_free(kvlist);
> > > > + if (name)
> > > > + free(name);
> > > > +
> > > > + return 0;
> > > > +}
> > > > +
> > > > +/*
> > > > + * Match the AFU Driver and AFU Device using the ID Table */
> > > > +static int rte_afu_match(const struct rte_afu_driver *afu_drv,
> > > > + const struct rte_afu_device *afu_dev) {
> > > > + const struct rte_afu_uuid *id_table;
> > > > +
> > > > + for (id_table = afu_drv->id_table;
> > > > + ((id_table->uuid_low != 0) && (id_table->uuid_high != 0));
> > > > + id_table++) {
> > > > + /* check if device's identifiers match the driver's ones */
> > > > + if ((id_table->uuid_low != afu_dev->id.uuid.uuid_low) ||
> > > > + id_table->uuid_high !=
> > > > + afu_dev->id.uuid.uuid_high)
> > > > + continue;
> > > > +
> > > > + return 1;
> > > > + }
> > > > +
> > > > + return 0;
> > > > +}
> > > > +
> > > > +static int
> > > > +ifpga_probe_one_driver(struct rte_afu_driver *drv,
> > > > + struct rte_afu_device *afu_dev) {
> > > > + int ret;
> > > > +
> > > > + if (!rte_afu_match(drv, afu_dev))
> > > > + /* Match of device and driver failed */
> > > > + return 1;
> > > > +
> > > > + /* reference driver structure */
> > > > + afu_dev->driver = drv;
> > > > + afu_dev->device.driver = &drv->driver;
> > > > +
> > > > + /* call the driver probe() function */
> > > > + ret = drv->probe(afu_dev);
> > > > + if (ret) {
> > > > + afu_dev->driver = NULL;
> > > > + afu_dev->device.driver = NULL;
> > > > + }
> > > > +
> > > > + return ret;
> > > > +}
> > > > +
> > > > +static int
> > > > +ifpga_probe_all_drivers(struct rte_afu_device *afu_dev) {
> > > > + struct rte_afu_driver *drv = NULL;
> > > > + int ret = 0;
> > > > +
> > > > + if (afu_dev == NULL)
> > > > + return -1;
> > > > +
> > > > + /* Check if a driver is already loaded */
> > > > + if (afu_dev->driver != NULL)
> > > > + return 0;
> > > > +
> > > > + TAILQ_FOREACH(drv, &afu_driver_list, next) {
> > > > + if (ifpga_probe_one_driver(drv, afu_dev)) {
> > > > + ret = -1;
> > > > + break;
> > > > + }
> > > > + }
> > > > + return ret;
> > > > +}
> > > > +
> > > > +/*
> > > > + * Scan the content of the Intel FPGA bus, and call the probe()
> > > > +function for
> > > > + * all registered drivers that have a matching entry in its
> > > > +id_table
> > > > + * for discovered devices.
> > > > + */
> > > > +static int
> > > > +ifpga_probe(void)
> > > > +{
> > > > + struct rte_ifpga_device *ifpga_dev;
> > > > + struct rte_afu_device *afu_dev = NULL;
> > > > + int ret = 0;
> > > > +
> > > > + TAILQ_FOREACH(ifpga_dev, &ifpga_device_list, next) {
> > > > + TAILQ_FOREACH(afu_dev, &ifpga_dev->afu_list, next) {
> > > > +
> > > > + if (afu_dev->device.driver)
> > > > + continue;
> > > > +
> > > > + ret = ifpga_probe_all_drivers(afu_dev);
> > > > + if (ret < 0)
> > > > + IFPGA_BUS_ERR("failed to initialize %s
> > > device\n",
> > > > + rte_ifpga_device_name(afu_dev));
> > > > + }
> > > > + }
> > > > +
> > > > + return ret;
> > > > +}
> > > > +
> > > > +static int
> > > > +ifpga_plug(struct rte_device *dev) {
> > > > + return ifpga_probe_all_drivers(RTE_DEV_TO_AFU(dev));
> > > > +}
> > > > +
> > > > +static int
> > > > +ifpga_remove_driver(struct rte_afu_device *afu_dev) {
> > > > + const char *name;
> > > > + const struct rte_afu_driver *driver;
> > > > +
> > > > + name = rte_ifpga_device_name(afu_dev);
> > > > + if (!afu_dev->device.driver) {
> > > > + IFPGA_BUS_DEBUG("no driver attach to device %s\n", name);
> > > > + return 1;
> > > > + }
> > > > +
> > > > + driver = RTE_DRV_TO_AFU_CONST(afu_dev->device.driver);
> > > > + return driver->remove(afu_dev);
> > > > +}
> > > > +
> > > > +static int
> > > > +ifpga_unplug(struct rte_device *dev) {
> > > > + struct rte_ifpga_device *ifpga_dev = NULL;
> > > > + struct rte_afu_device *afu_dev = NULL;
> > > > + struct rte_devargs *devargs = NULL;
> > > > + int ret;
> > > > +
> > > > + if (dev == NULL)
> > > > + return -EINVAL;
> > > > +
> > > > + afu_dev = RTE_DEV_TO_AFU(dev);
> > > > + if (!dev)
> > > > + return -ENOENT;
> > > > +
> > > > + ifpga_dev = afu_dev->ifpga_dev;
> > > > + devargs = dev->devargs;
> > > > +
> > > > + ret = ifpga_remove_driver(afu_dev);
> > >
> > > So what is the device type that be plugged into the ifgpa bus?
> > > ifpga_dev or afu_dev? If its afu_dev, why we need ifpga_dev?
> > > If it is ifpga_dev, we need remove ifpga_dev from ifpga_device_list here.
> >
> > The device type that be plugged into the ifpga bus is afu_dev.
> > And each afu_dev will be added into one ifpga_dev.
> > Each FPGA device bind to ifpga_dev.
> >
> > > > + if (ret)
> > > > + return ret;
> > > > +
> > > > + TAILQ_REMOVE(&ifpga_dev->afu_list, afu_dev, next);
> > > > +
> > > > + rte_devargs_remove(devargs->bus->name, devargs->name);
> > > > + free(afu_dev);
> > > > + return 0;
> > > > +
> > > > +}
> > > > +
> > > > +static struct rte_device *
> > > > +ifpga_find_device(const struct rte_device *start,
> > > > + rte_dev_cmp_t cmp, const void *data) {
> > > > + struct rte_ifpga_device *ifpga_dev;
> > > > + struct rte_afu_device *afu_dev;
> > > > +
> > > > + TAILQ_FOREACH(ifpga_dev, &ifpga_device_list, next) {
> > > > + TAILQ_FOREACH(afu_dev, &ifpga_dev->afu_list, next) {
> > > > + if (start && &afu_dev->device == start) {
> > > > + start = NULL;
> > > > + continue;
> > > > + }
> > > > + if (cmp(&afu_dev->device, data) == 0)
> > > > + return &afu_dev->device;
> > > > + }
> > > > + }
> > > > + return NULL;
> > > > +}
> > > > +static int
> > > > +ifpga_parse(const char *name, void *addr) {
> > > > + int *out = addr;
> > > > + struct rte_rawdev *rawdev = NULL;
> > > > + char rawdev_name[RTE_RAWDEV_NAME_MAX_LEN];
> > > > + char *c1 = NULL, *c2 = NULL;
> > > > + int port = IFPGA_BUS_DEV_PORT_MAX;
> > > > + char str_port[8];
> > > > + int str_port_len = 0;
> > > > + int ret;
> > > > +
> > > > + memset(str_port, 0, 8);
> > > > + c1 = strchr(name, '|');
> > > > + if (c1 != NULL) {
> > > > + str_port_len = c1-name;
> > > > + c2 = c1+1;
> > > > + }
> > > > +
> > > > + if (str_port_len < 8 &&
> > > > + str_port_len > 0) {
> > > > + memcpy(str_port, name, str_port_len);
> > > > + ret = sscanf(str_port, "%d", &port);
> > > > + if (ret == -1)
> > > > + return 0;
> > > > + }
> > > > +
> > > > + memset(rawdev_name, 0, sizeof(rawdev_name));
> > > > + snprintf(rawdev_name, RTE_RAWDEV_NAME_MAX_LEN,
> "IFPGA:%s",
> > > c2);
> > > > + rawdev = rte_rawdev_pmd_get_named_dev(rawdev_name);
> > > > +
> > > > + if ((port < IFPGA_BUS_DEV_PORT_MAX) &&
> > > > + rawdev &&
> > > > + (addr != NULL))
> > > > + *out = port;
> > > > +
> > > > + if ((port < IFPGA_BUS_DEV_PORT_MAX) &&
> > > > + rawdev)
> > > > + return 0;
> > > > + else
> > > > + return 1;
> > > > +}
> > > > +
> > > > +static struct rte_bus rte_ifpga_bus = {
> > > > + .scan = ifpga_scan,
> > > > + .probe = ifpga_probe,
> > > > + .find_device = ifpga_find_device,
> > > > + .plug = ifpga_plug,
> > > > + .unplug = ifpga_unplug,
> > > > + .parse = ifpga_parse,
> > > > +};
> > > > +
> > > > +RTE_REGISTER_BUS(IFPGA_BUS_NAME, rte_ifpga_bus);
> > > > +
> > > > +RTE_INIT(ifpga_init_log)
> > > > +{
> > > > + ifpga_bus_logtype = rte_log_register("bus.ifpga");
> > > > + if (ifpga_bus_logtype >= 0)
> > > > + rte_log_set_level(ifpga_bus_logtype, RTE_LOG_NOTICE); }
> > > > diff --git a/drivers/bus/ifpga/ifpga_common.c
> > > > b/drivers/bus/ifpga/ifpga_common.c
> > > > new file mode 100644
> > > > index 0000000..78e2eae
> > > > --- /dev/null
> > > > +++ b/drivers/bus/ifpga/ifpga_common.c
> > > > @@ -0,0 +1,88 @@
> > > > +/* SPDX-License-Identifier: BSD-3-Clause
> > > > + * Copyright(c) 2010-2018 Intel Corporation */
> > > > +
> > > > +#include <string.h>
> > > > +#include <inttypes.h>
> > > > +#include <stdint.h>
> > > > +#include <stdlib.h>
> > > > +#include <stdio.h>
> > > > +#include <sys/queue.h>
> > > > +#include <sys/mman.h>
> > > > +#include <sys/types.h>
> > > > +#include <unistd.h>
> > > > +#include <fcntl.h>
> > > > +
> > > > +#include <rte_errno.h>
> > > > +#include <rte_bus.h>
> > > > +#include <rte_per_lcore.h>
> > > > +#include <rte_memory.h>
> > > > +#include <rte_memzone.h>
> > > > +#include <rte_eal.h>
> > > > +#include <rte_common.h>
> > > > +
> > > > +#include <rte_devargs.h>
> > > > +#include <rte_kvargs.h>
> > > > +#include <rte_alarm.h>
> > > > +
> > > > +#include "rte_bus_ifpga.h"
> > > > +#include "ifpga_logs.h"
> > > > +#include "ifpga_common.h"
> > > > +
> > > > +int rte_ifpga_get_string_arg(const char *key __rte_unused,
> > > > + const char *value, void *extra_args) {
> > > > + if (!value || !extra_args)
> > > > + return -EINVAL;
> > > > +
> > > > + *(char **)extra_args = strdup(value);
> > > > +
> > > > + if (!*(char **)extra_args)
> > > > + return -ENOMEM;
> > > > +
> > > > + return 0;
> > > > +}
> > > > +int rte_ifpga_get_integer32_arg(const char *key __rte_unused,
> > > > + const char *value, void *extra_args) {
> > > > + if (!value || !extra_args)
> > > > + return -EINVAL;
> > > > +
> > > > + *(int *)extra_args = strtoull(value, NULL, 0);
> > > > +
> > > > + return 0;
> > > > +}
> > > > +int ifpga_get_integer64_arg(const char *key __rte_unused,
> > > > + const char *value, void *extra_args) {
> > > > + if (!value || !extra_args)
> > > > + return -EINVAL;
> > > > +
> > > > + *(uint64_t *)extra_args = strtoull(value, NULL, 0);
> > > > +
> > > > + return 0;
> > > > +}
> > > > +int ifpga_get_unsigned_long(const char *str, int base) {
> > > > + unsigned long num;
> > > > + char *end = NULL;
> > > > +
> > > > + errno = 0;
> > > > +
> > > > + num = strtoul(str, &end, base);
> > > > + if ((str[0] == '\0') || (end == NULL) || (*end != '\0') || (errno != 0))
> > > > + return -1;
> > > > +
> > > > + return num;
> > > > +}
> > > > +
> > > > +int ifpga_afu_id_cmp(const struct rte_afu_id *afu_id0,
> > > > + const struct rte_afu_id *afu_id1) {
> > > > + if ((afu_id0->uuid.uuid_low == afu_id1->uuid.uuid_low) &&
> > > > + (afu_id0->uuid.uuid_high == afu_id1->uuid.uuid_high) &&
> > > > + (afu_id0->port == afu_id1->port)) {
> > > > + return 0;
> > > > + } else
> > > > + return 1;
> > > > +}
> > > > diff --git a/drivers/bus/ifpga/ifpga_common.h
> > > > b/drivers/bus/ifpga/ifpga_common.h
> > > > new file mode 100644
> > > > index 0000000..f9254b9
> > > > --- /dev/null
> > > > +++ b/drivers/bus/ifpga/ifpga_common.h
> > > > @@ -0,0 +1,18 @@
> > > > +/* SPDX-License-Identifier: BSD-3-Clause
> > > > + * Copyright(c) 2010-2018 Intel Corporation */
> > > > +
> > > > +#ifndef _IFPGA_COMMON_H_
> > > > +#define _IFPGA_COMMON_H_
> > > > +
> > > > +int rte_ifpga_get_string_arg(const char *key __rte_unused,
> > > > + const char *value, void *extra_args); int
> > > > +rte_ifpga_get_integer32_arg(const char *key __rte_unused,
> > > > + const char *value, void *extra_args); int
> > > > +ifpga_get_integer64_arg(const char *key __rte_unused,
> > > > + const char *value, void *extra_args); int
> > > > +ifpga_get_unsigned_long(const char *str, int base); int
> > > > +ifpga_afu_id_cmp(const struct rte_afu_id *afu_id0,
> > > > + const struct rte_afu_id *afu_id1);
> > > > +
> > > > +#endif /* _IFPGA_COMMON_H_ */
> > > > diff --git a/drivers/bus/ifpga/ifpga_logs.h
> > > > b/drivers/bus/ifpga/ifpga_logs.h new file mode 100644 index
> > > > 0000000..873e0a4
> > > > --- /dev/null
> > > > +++ b/drivers/bus/ifpga/ifpga_logs.h
> > > > @@ -0,0 +1,31 @@
> > > > +/* SPDX-License-Identifier: BSD-3-Clause
> > > > + * Copyright(c) 2010-2018 Intel Corporation */
> > > > +
> > > > +#ifndef _IFPGA_LOGS_H_
> > > > +#define _IFPGA_LOGS_H_
> > > > +
> > > > +#include <rte_log.h>
> > > > +
> > > > +extern int ifpga_bus_logtype;
> > > > +
> > > > +#define IFPGA_LOG(level, fmt, args...) \
> > > > + rte_log(RTE_LOG_ ## level, ifpga_bus_logtype, "%s(): " fmt "\n", \
> > > > + __func__, ##args)
> > > > +
> > > > +#define IFPGA_BUS_LOG(level, fmt, args...) \
> > > > + rte_log(RTE_LOG_ ## level, ifpga_bus_logtype, "%s(): " fmt "\n", \
> > > > + __func__, ##args)
> > > > +
> > > > +#define IFPGA_BUS_FUNC_TRACE() IFPGA_BUS_LOG(DEBUG, ">>")
> > > > +
> > > > +#define IFPGA_BUS_DEBUG(fmt, args...) \
> > > > + IFPGA_BUS_LOG(DEBUG, fmt, ## args) #define IFPGA_BUS_INFO(fmt,
> > > > +args...) \
> > > > + IFPGA_BUS_LOG(INFO, fmt, ## args) #define IFPGA_BUS_ERR(fmt,
> > > > +args...) \
> > > > + IFPGA_BUS_LOG(ERR, fmt, ## args) #define IFPGA_BUS_WARN(fmt,
> > > > +args...) \
> > > > + IFPGA_BUS_LOG(WARNING, fmt, ## args)
> > > > +
> > > > +#endif /* _IFPGA_BUS_LOGS_H_ */
> > > > diff --git a/drivers/bus/ifpga/meson.build
> > > > b/drivers/bus/ifpga/meson.build new file mode 100644 index
> > > > 0000000..c9b08c8
> > > > --- /dev/null
> > > > +++ b/drivers/bus/ifpga/meson.build
> > > > @@ -0,0 +1,8 @@
> > > > +# SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2010-2018
> > > > +Intel Corporation
> > > > +
> > > > +deps += ['pci', 'kvargs', 'rawdev']
> > > > +install_headers('rte_bus_ifpga.h')
> > > > +sources = files('ifpga_common.c', 'ifpga_bus.c')
> > > > +
> > > > +allow_experimental_apis = true
> > > > diff --git a/drivers/bus/ifpga/rte_bus_ifpga.h
> > > > b/drivers/bus/ifpga/rte_bus_ifpga.h
> > > > new file mode 100644
> > > > index 0000000..5c559e1
> > > > --- /dev/null
> > > > +++ b/drivers/bus/ifpga/rte_bus_ifpga.h
> > > > @@ -0,0 +1,160 @@
> > > > +/* SPDX-License-Identifier: BSD-3-Clause
> > > > + * Copyright(c) 2010-2018 Intel Corporation */
> > > > +
> > > > +#ifndef _RTE_BUS_IFPGA_H_
> > > > +#define _RTE_BUS_IFPGA_H_
> > > > +
> > > > +/**
> > > > + * @file
> > > > + *
> > > > + * RTE Intel FPGA Bus Interface
> > > > + */
> > > > +
> > > > +#ifdef __cplusplus
> > > > +extern "C" {
> > > > +#endif
> > > > +
> > > > +#include <rte_bus.h>
> > > > +#include <rte_pci.h>
> > > > +
> > > > +/** Name of Intel FPGA Bus */
> > > > +#define IFPGA_BUS_NAME ifpga
> > > > +
> > > > +/* Forward declarations */
> > > > +struct rte_afu_device;
> > > > +
> > > > +/** List of Intel AFU devices */
> > > > +TAILQ_HEAD(afu_device_list, rte_afu_device);
> > > > +/** Double linked list of AFU device drivers. */
> > > > +TAILQ_HEAD(afu_driver_list, rte_afu_driver);
> > > > +
> > > > +#define IFPGA_BUS_BITSTREAM_PATH_MAX_LEN 256
> > > > +
> > > > +struct rte_afu_uuid {
> > > > + uint64_t uuid_low;
> > > > + uint64_t uuid_high;
> > > > +} __attribute__ ((packed));
> > > > +
> > > > +#define IFPGA_BUS_DEV_PORT_MAX 4
> > > > +
> > > > +/**
> > > > + * A structure describing an ID for a AFU driver. Each driver
> > > > +provides a
> > > > + * table of these IDs for each device that it supports.
> > > > + */
> > > > +struct rte_afu_id {
> > > > + struct rte_afu_uuid uuid;
> > > > + int port; /**< port number */
> > > > +} __attribute__ ((packed));
> > > > +
> > > > +/**
> > > > + * A structure PR (Partial Reconfiguration) configuration AFU driver.
> > > > + */
> > > > +
> > > > +struct rte_afu_pr_conf {
> > > > + struct rte_afu_id afu_id;
> > > > + int pr_enable;
> > > > + char bs_path[IFPGA_BUS_BITSTREAM_PATH_MAX_LEN];
> > > > +};
> > > > +
> > > > +#define AFU_PRI_STR_SIZE (PCI_PRI_STR_SIZE + 8)
> > > > +
> > > > +/**
> > > > + * A structure describing a fpga device.
> > > > + */
> > > > +struct rte_ifpga_device {
> > > > + TAILQ_ENTRY(rte_ifpga_device) next; /**< Next in device list.
> */
> > > > + struct rte_rawdev *rdev;
> > > > + struct afu_device_list afu_list; /**< List of AFU devices */ };
> > > > +
> > > > +/**
> > > > + * A structure describing a AFU device.
> > > > + */
> > > > +struct rte_afu_device {
> > > > + TAILQ_ENTRY(rte_afu_device) next; /**< Next in device list.
> */
> > > > + struct rte_device device; /**< Inherit core device */
> > > > + struct rte_rawdev *rawdev; /**< Point Rawdev */
> > > > + struct rte_ifpga_device *ifpga_dev; /**< Point ifpga device */
> > > > + struct rte_afu_id id; /**< AFU id within FPGA.
> */
> > > > + uint32_t num_region; /**< number of regions found */
> > > > + struct rte_mem_resource mem_resource[PCI_MAX_RESOURCE];
> > > > + /**< AFU Memory Resource
> > > */
> > > > + struct rte_intr_handle intr_handle; /**< Interrupt handle */
> > > > + struct rte_afu_driver *driver; /**< Associated driver */
> > > > + char path[IFPGA_BUS_BITSTREAM_PATH_MAX_LEN];
> > > > +} __attribute__ ((packed));
> > > > +
> > > > +/**
> > > > + * @internal
> > > > + * Helper macro for drivers that need to convert to struct
> rte_afu_device.
> > > > + */
> > > > +#define RTE_DEV_TO_AFU(ptr) \
> > > > + container_of(ptr, struct rte_afu_device, device)
> > > > +
> > > > +#define RTE_DRV_TO_AFU_CONST(ptr) \
> > > > + container_of(ptr, const struct rte_afu_driver, driver)
> > > > +
> > > > +/**
> > > > + * Initialisation function for the driver called during FPGA BUS probing.
> > >
> > >
> > > typo: Initialization
> >
> > Fixed.
> >
> > > > + */
> > > > +typedef int (afu_probe_t)(struct rte_afu_device *);
> > > > +
> > > > +/**
> > > > + * Uninitialisation function for the driver called during hotplugging.
> > >
> > > typo: Unintialization
> >
> > Fixed.
> >
> > > > + */
> > > > +typedef int (afu_remove_t)(struct rte_afu_device *);
> > > > +
> > > > +/**
> > > > + * A structure describing a AFU device.
> > > > + */
> > > > +struct rte_afu_driver {
> > > > + TAILQ_ENTRY(rte_afu_driver) next; /**< Next afu driver. */
> > > > + struct rte_driver driver; /**< Inherit core driver. */
> > > > + afu_probe_t *probe; /**< Device Probe
> function.
> > > > */
> > > > + afu_remove_t *remove; /**< Device Remove
> > > > function. */
> > > > + const struct rte_afu_uuid *id_table; /**< AFU uuid within FPGA.
> */
> > > > + uint32_t drv_flags; /**< Flags contolling handling of device.
> */
> > >
> > > Typo Controlling
> > > And what is "flags controlling handling of device?"
> >
> > This variable is not used, so I have removed it.
> >
> > > > +};
> > > > +
> > > > +static inline const char *
> > > > +rte_ifpga_device_name(const struct rte_afu_device *afu) {
> > > > + if (afu && afu->device.name)
> > > > + return afu->device.name;
> > > > + return NULL;
> > > > +}
> > > > +
> > > > +/**
> > > > + * Register a ifpga afu device driver.
> > > > + *
> > > > + * @param driver
> > > > + * A pointer to a rte_afu_driver structure describing the driver
> > > > + * to be registered.
> > > > + */
> > > > +void rte_ifpga_driver_register(struct rte_afu_driver *driver);
> > > > +
> > > > +/**
> > > > + * Unregister a ifpga afu device driver.
> > > > + *
> > > > + * @param driver
> > > > + * A pointer to a rte_afu_driver structure describing the driver
> > > > + * to be unregistered.
> > > > + */
> > > > +void rte_ifpga_driver_unregister(struct rte_afu_driver *driver);
> > > > +
> > > > +#define RTE_PMD_REGISTER_AFU(nm, afudrv)\ RTE_INIT(afudrvinitfn_
> > > > +##afudrv);\ static const char *afudrvinit_ ## nm ## _alias;\
> > > > +static void afudrvinitfn_ ##afudrv(void)\ {\
> > > > + (afudrv).driver.name = RTE_STR(nm);\
> > > > + (afudrv).driver.alias = afudrvinit_ ## nm ## _alias;\
> > > > + rte_ifpga_driver_register(&afudrv);\
> > > > +} \
> > > > +RTE_PMD_EXPORT_NAME(nm, __COUNTER__)
> > > > +
> > > > +#define RTE_PMD_REGISTER_AFU_ALIAS(nm, alias)\ static const char
> > > > +*afudrvinit_ ## nm ## _alias = RTE_STR(alias)
> > > > +
> > > > +#endif /* _RTE_BUS_IFPGA_H_ */
> > > > diff --git a/drivers/bus/ifpga/rte_bus_ifpga_version.map
> > > > b/drivers/bus/ifpga/rte_bus_ifpga_version.map
> > > > new file mode 100644
> > > > index 0000000..a027979
> > > > --- /dev/null
> > > > +++ b/drivers/bus/ifpga/rte_bus_ifpga_version.map
> > > > @@ -0,0 +1,10 @@
> > > > +DPDK_18.05 {
> > > > + global:
> > > > +
> > > > + rte_ifpga_get_integer32_arg;
> > > > + rte_ifpga_get_string_arg;
> > > > + rte_ifpga_driver_register;
> > > > + rte_ifpga_driver_unregister;
> > > > +
> > > > + local: *;
> > > > +};
> > > > diff --git a/drivers/bus/meson.build b/drivers/bus/meson.build
> > > > index 58dfbe2..52c755d 100644
> > > > --- a/drivers/bus/meson.build
> > > > +++ b/drivers/bus/meson.build
> > > > @@ -1,7 +1,7 @@
> > > > # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2017
> > > > Intel Corporation
> > > >
> > > > -drivers = ['dpaa', 'fslmc', 'pci', 'vdev']
> > > > +drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'vdev']
> > > > std_deps = ['eal']
> > > > config_flag_fmt = 'RTE_LIBRTE_@0@_BUS'
> > > > driver_name_fmt = 'rte_bus_@0@'
> > > > diff --git a/mk/rte.app.mk b/mk/rte.app.mk index 26f3563..3861e1a
> > > > 100644
> > > > --- a/mk/rte.app.mk
> > > > +++ b/mk/rte.app.mk
> > > > @@ -255,6 +255,7 @@ ifeq
> > > > ($(CONFIG_RTE_EAL_VFIO)$(CONFIG_RTE_LIBRTE_FSLMC_BUS),yy)
> > > > _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_CMDIF_RAWDEV) +=
> > > > -lrte_pmd_dpaa2_cmdif
> > > > _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV) +=
> > > > -lrte_pmd_dpaa2_qdma endif # CONFIG_RTE_LIBRTE_FSLMC_BUS
> > > > +_LDLIBS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += -lrte_bus_ifpga
> > > > endif # CONFIG_RTE_LIBRTE_RAWDEV
> > > >
> > > >
> > > > --
> > > > 1.8.3.1
> > >
> > > Regards
> > > Qi
> -----Original Message-----
> From: Xu, Rosen
> Sent: Thursday, May 10, 2018 9:58 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; dev@dpdk.org;
> thomas@monjalon.net
> Cc: Zhang, Roy Fan <roy.fan.zhang@intel.com>; Doherty, Declan
> <declan.doherty@intel.com>; Richardson, Bruce
> <bruce.richardson@intel.com>; shreyansh.jain@nxp.com; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Ananyev, Konstantin
> <konstantin.ananyev@intel.com>; Zhang, Tianfei <tianfei.zhang@intel.com>;
> Liu, Song <song.liu@intel.com>; Wu, Hao <hao.wu@intel.com>;
> gaetan.rivet@6wind.com
> Subject: RE: [dpdk-dev] [PATCH v10 1/3] bus/ifpga: Add Intel FPGA BUS Library
>
> Hi Qi,
>
> > -----Original Message-----
> > From: Zhang, Qi Z
> > Sent: Thursday, May 10, 2018 21:49
> > To: Xu, Rosen <rosen.xu@intel.com>; dev@dpdk.org;
> thomas@monjalon.net
> > Cc: Zhang, Roy Fan <roy.fan.zhang@intel.com>; Doherty, Declan
> > <declan.doherty@intel.com>; Richardson, Bruce
> > <bruce.richardson@intel.com>; shreyansh.jain@nxp.com; Yigit, Ferruh
> > <ferruh.yigit@intel.com>; Ananyev, Konstantin
> > <konstantin.ananyev@intel.com>; Zhang, Tianfei
> > <tianfei.zhang@intel.com>; Liu, Song <song.liu@intel.com>; Wu, Hao
> > <hao.wu@intel.com>; gaetan.rivet@6wind.com
> > Subject: RE: [dpdk-dev] [PATCH v10 1/3] bus/ifpga: Add Intel FPGA BUS
> > Library
> >
> >
> >
> > > -----Original Message-----
> > > From: Xu, Rosen
> > > Sent: Thursday, May 10, 2018 9:29 PM
> > > To: Zhang, Qi Z <qi.z.zhang@intel.com>; dev@dpdk.org;
> > > thomas@monjalon.net
> > > Cc: Zhang, Roy Fan <roy.fan.zhang@intel.com>; Doherty, Declan
> > > <declan.doherty@intel.com>; Richardson, Bruce
> > > <bruce.richardson@intel.com>; shreyansh.jain@nxp.com; Yigit, Ferruh
> > > <ferruh.yigit@intel.com>; Ananyev, Konstantin
> > > <konstantin.ananyev@intel.com>; Zhang, Tianfei
> > > <tianfei.zhang@intel.com>; Liu, Song <song.liu@intel.com>; Wu, Hao
> > > <hao.wu@intel.com>; gaetan.rivet@6wind.com
> > > Subject: RE: [dpdk-dev] [PATCH v10 1/3] bus/ifpga: Add Intel FPGA
> > > BUS Library
> > >
> > > Hi Qi,
> > >
> > > > -----Original Message-----
> > > > From: Zhang, Qi Z
> > > > Sent: Thursday, May 10, 2018 20:27
> > > > To: Xu, Rosen <rosen.xu@intel.com>; dev@dpdk.org;
> > > thomas@monjalon.net
> > > > Cc: Xu, Rosen <rosen.xu@intel.com>; Zhang, Roy Fan
> > > > <roy.fan.zhang@intel.com>; Doherty, Declan
> > > > <declan.doherty@intel.com>; Richardson, Bruce
> > > > <bruce.richardson@intel.com>; shreyansh.jain@nxp.com; Yigit,
> > > > Ferruh <ferruh.yigit@intel.com>; Ananyev, Konstantin
> > > > <konstantin.ananyev@intel.com>; Zhang, Tianfei
> > > > <tianfei.zhang@intel.com>; Liu, Song <song.liu@intel.com>; Wu, Hao
> > > > <hao.wu@intel.com>; gaetan.rivet@6wind.com
> > > > Subject: RE: [dpdk-dev] [PATCH v10 1/3] bus/ifpga: Add Intel FPGA
> > > > BUS Library
> > > >
> > > > Hi Rosen:
> > > >
> > > > > -----Original Message-----
> > > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Xu, Rosen
> > > > > Sent: Wednesday, May 9, 2018 3:43 PM
> > > > > To: dev@dpdk.org; thomas@monjalon.net
> > > > > Cc: Xu, Rosen <rosen.xu@intel.com>; Zhang, Roy Fan
> > > > > <roy.fan.zhang@intel.com>; Doherty, Declan
> > > > > <declan.doherty@intel.com>; Richardson, Bruce
> > > > > <bruce.richardson@intel.com>; shreyansh.jain@nxp.com; Yigit,
> > > > > Ferruh <ferruh.yigit@intel.com>; Ananyev, Konstantin
> > > > > <konstantin.ananyev@intel.com>; Zhang, Tianfei
> > > > > <tianfei.zhang@intel.com>; Liu, Song <song.liu@intel.com>; Wu,
> > > > > Hao <hao.wu@intel.com>; gaetan.rivet@6wind.com
> > > > > Subject: [dpdk-dev] [PATCH v10 1/3] bus/ifpga: Add Intel FPGA
> > > > > BUS Library
> > > > >
> > > > > From: Rosen Xu <rosen.xu@intel.com>
> > > > >
> > > > > Defined FPGA-BUS for Acceleration Drivers of AFUs
> > > > >
> > > > > 1. FPGA PCI Scan (1st Scan) follows DPDK UIO/VFIO PCI Scan
> > > > > Process, probe Intel FPGA Rawdev Driver, it will be covered in
> > > > > following
> > patches.
> > > > >
> > > > > 2. AFU Scan(2nd Scan) bind DPDK driver to FPGA Partial-Bitstream.
> > > > > This scan is trigged by hotplug of IFPGA Rawdev probe, in this
> > > > > scan the AFUs will be created and their drivers are also probed.
> > > > >
> > > > > This patch will introduce rte_afu_device which describe the AFU
> > > > > device listed in the FPGA-BUS.
> > > > >
> > > > > Signed-off-by: Rosen Xu <rosen.xu@intel.com>
> > > > > Signed-off-by: Tianfei Zhang <tianfei.zhang@intel.com>
> > > > > ---
> > > > > MAINTAINERS | 4 +
> > > > > config/common_base | 5 +
> > > > > doc/guides/rel_notes/release_18_05.rst | 5 +
> > > > > drivers/bus/Makefile | 1 +
> > > > > drivers/bus/ifpga/Makefile | 32 ++
> > > > > drivers/bus/ifpga/ifpga_bus.c | 501
> > > > > ++++++++++++++++++++++++++++
> > > > > drivers/bus/ifpga/ifpga_common.c | 88 +++++
> > > > > drivers/bus/ifpga/ifpga_common.h | 18 +
> > > > > drivers/bus/ifpga/ifpga_logs.h | 31 ++
> > > > > drivers/bus/ifpga/meson.build | 8 +
> > > > > drivers/bus/ifpga/rte_bus_ifpga.h | 160 +++++++++
> > > > > drivers/bus/ifpga/rte_bus_ifpga_version.map | 10 +
> > > > > drivers/bus/meson.build | 2 +-
> > > > > mk/rte.app.mk | 1 +
> > > > > 14 files changed, 865 insertions(+), 1 deletion(-) create mode
> > > > > 100644 drivers/bus/ifpga/Makefile create mode 100644
> > > > > drivers/bus/ifpga/ifpga_bus.c create mode 100644
> > > > > drivers/bus/ifpga/ifpga_common.c create mode 100644
> > > > > drivers/bus/ifpga/ifpga_common.h create mode 100644
> > > > > drivers/bus/ifpga/ifpga_logs.h create mode 100644
> > > > > drivers/bus/ifpga/meson.build create mode 100644
> > > > > drivers/bus/ifpga/rte_bus_ifpga.h create mode 100644
> > > > > drivers/bus/ifpga/rte_bus_ifpga_version.map
> > > > >
> > > > > diff --git a/MAINTAINERS b/MAINTAINERS index 7105920..fa0c5b1
> > > > > 100644
> > > > > --- a/MAINTAINERS
> > > > > +++ b/MAINTAINERS
> > > > > @@ -383,6 +383,10 @@ F: drivers/mempool/bucket/ Bus Drivers
> > > > > -----------
> > > > >
> > > > > +Intel FPGA buses
> > > > > +M: Rosen Xu <rosen.xu@intel.com>
> > > > > +F: drivers/bus/ifpga/
> > > > > +
> > > > > NXP buses
> > > > > M: Hemant Agrawal <hemant.agrawal@nxp.com>
> > > > > M: Shreyansh Jain <shreyansh.jain@nxp.com> diff --git
> > > > > a/config/common_base b/config/common_base index
> > 0d181ac..1440316
> > > > > 100644
> > > > > --- a/config/common_base
> > > > > +++ b/config/common_base
> > > > > @@ -139,6 +139,11 @@
> > > > > CONFIG_RTE_ETHDEV_PROFILE_ITT_WASTED_RX_ITERATIONS=n
> > > > > CONFIG_RTE_ETHDEV_TX_PREPARE_NOOP=n
> > > > >
> > > > > #
> > > > > +# Compile the Intel FPGA bus
> > > > > +#
> > > > > +CONFIG_RTE_LIBRTE_IFPGA_BUS=y
> > > > > +
> > > > > +#
> > > > > # Compile PCI bus driver
> > > > > #
> > > > > CONFIG_RTE_LIBRTE_PCI_BUS=y
> > > > > diff --git a/doc/guides/rel_notes/release_18_05.rst
> > > > > b/doc/guides/rel_notes/release_18_05.rst
> > > > > index 7187348..265950a 100644
> > > > > --- a/doc/guides/rel_notes/release_18_05.rst
> > > > > +++ b/doc/guides/rel_notes/release_18_05.rst
> > > > > @@ -183,6 +183,11 @@ New Features
> > > > > stats/xstats on shared memory from secondary process, and
> > > > > also pdump packets on
> > > > > those virtual devices.
> > > > >
> > > > > +* **Added Ifpga Bus, a generic Intel FPGA Bus library.**
> > > > > +
> > > > > + The Ifpga Bus library provides support for integrating any
> > > > > + Intel FPGA device with the DPDK framework. It provides Intel
> > > > > + FPGA Partial Bit Stream AFU(Accelerated Function Unit) scan
> > > > > + and drivers
> > > prove.
> > > > >
> > > > > API Changes
> > > > > -----------
> > > > > diff --git a/drivers/bus/Makefile b/drivers/bus/Makefile index
> > > > > c251b65..ef7f247 100644
> > > > > --- a/drivers/bus/Makefile
> > > > > +++ b/drivers/bus/Makefile
> > > > > @@ -7,6 +7,7 @@ DIRS-$(CONFIG_RTE_LIBRTE_DPAA_BUS) += dpaa
> > ifeq
> > > > > ($(CONFIG_RTE_EAL_VFIO),y)
> > > > > DIRS-$(CONFIG_RTE_LIBRTE_FSLMC_BUS) += fslmc endif
> > > > > +DIRS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += ifpga
> > > > > DIRS-$(CONFIG_RTE_LIBRTE_PCI_BUS) += pci
> > > > > DIRS-$(CONFIG_RTE_LIBRTE_VDEV_BUS) += vdev
> > > > >
> > > > > diff --git a/drivers/bus/ifpga/Makefile
> > > > > b/drivers/bus/ifpga/Makefile new file mode 100644 index
> > > > > 0000000..3ff3bdb
> > > > > --- /dev/null
> > > > > +++ b/drivers/bus/ifpga/Makefile
> > > > > @@ -0,0 +1,32 @@
> > > > > +# SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2018
> > > > > +Intel Corporation
> > > > > +
> > > > > +include $(RTE_SDK)/mk/rte.vars.mk
> > > > > +
> > > > > +#
> > > > > +# library name
> > > > > +#
> > > > > +LIB = librte_bus_ifpga.a
> > > > > +
> > > > > +CFLAGS += -DALLOW_EXPERIMENTAL_API CFLAGS += -O3 CFLAGS +=
> > > > > +$(WERROR_FLAGS) LDLIBS += -lrte_eal LDLIBS += -lrte_rawdev
> > > > > +LDLIBS
> > > > > ++= -lrte_kvargs
> > > > > +
> > > > > +# versioning export map
> > > > > +EXPORT_MAP := rte_bus_ifpga_version.map
> > > > > +
> > > > > +# library version
> > > > > +LIBABIVER := 1
> > > > > +
> > > > > +SRCS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += ifpga_bus.c
> > > > > +SRCS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += ifpga_common.c
> > > > > +
> > > > > +#
> > > > > +# Export include files
> > > > > +#
> > > > > +SYMLINK-$(CONFIG_RTE_LIBRTE_IFPGA_BUS)-include +=
> > rte_bus_ifpga.h
> > > > > +
> > > > > +include $(RTE_SDK)/mk/rte.lib.mk
> > > > > diff --git a/drivers/bus/ifpga/ifpga_bus.c
> > > > > b/drivers/bus/ifpga/ifpga_bus.c new file mode 100644 index
> > > > > 0000000..e144c01
> > > > > --- /dev/null
> > > > > +++ b/drivers/bus/ifpga/ifpga_bus.c
> > > > > @@ -0,0 +1,501 @@
> > > > > +/* SPDX-License-Identifier: BSD-3-Clause
> > > > > + * Copyright(c) 2010-2018 Intel Corporation */
> > > > > +
> > > > > +#include <string.h>
> > > > > +#include <inttypes.h>
> > > > > +#include <stdint.h>
> > > > > +#include <stdlib.h>
> > > > > +#include <stdio.h>
> > > > > +#include <sys/queue.h>
> > > > > +#include <sys/mman.h>
> > > > > +#include <sys/types.h>
> > > > > +#include <unistd.h>
> > > > > +#include <fcntl.h>
> > > > > +
> > > > > +#include <rte_errno.h>
> > > > > +#include <rte_bus.h>
> > > > > +#include <rte_per_lcore.h>
> > > > > +#include <rte_memory.h>
> > > > > +#include <rte_memzone.h>
> > > > > +#include <rte_eal.h>
> > > > > +#include <rte_common.h>
> > > > > +
> > > > > +#include <rte_devargs.h>
> > > > > +#include <rte_kvargs.h>
> > > > > +#include <rte_alarm.h>
> > > > > +
> > > > > +#include "rte_rawdev.h"
> > > > > +#include "rte_rawdev_pmd.h"
> > > > > +#include "rte_bus_ifpga.h"
> > > > > +#include "ifpga_logs.h"
> > > > > +#include "ifpga_common.h"
> > > > > +
> > > > > +int ifpga_bus_logtype;
> > > > > +
> > > > > +/* Forward declaration to access Intel FPGA bus
> > > > > + * on which iFPGA devices are connected */ static struct
> > > > > +rte_bus rte_ifpga_bus;
> > > > > +
> > > > > +/** Double linked list of IFPGA device. */
> > > > > +TAILQ_HEAD(ifpga_device_list, rte_ifpga_device);
> > > > > +
> > > > > +static struct ifpga_device_list ifpga_device_list =
> > > > > + TAILQ_HEAD_INITIALIZER(ifpga_device_list);
> > > > > +static struct afu_driver_list afu_driver_list =
> > > > > + TAILQ_HEAD_INITIALIZER(afu_driver_list);
> > > > > +
> > > > > +
> > > > > +/* register a ifpga bus based driver */ void
> > > > > +rte_ifpga_driver_register(struct rte_afu_driver *driver) {
> > > > > + RTE_VERIFY(driver);
> > > > > +
> > > > > + TAILQ_INSERT_TAIL(&afu_driver_list, driver, next); }
> > > > > +
> > > > > +/* un-register a fpga bus based driver */ void
> > > > > +rte_ifpga_driver_unregister(struct rte_afu_driver *driver) {
> > > > > + TAILQ_REMOVE(&afu_driver_list, driver, next); }
> > > > > +
> > > > > +static struct rte_ifpga_device * ifpga_find_ifpga_dev(const
> > > > > +struct rte_rawdev *rdev) {
> > > > > + struct rte_ifpga_device *ifpga_dev = NULL;
> > > > > +
> > > > > + TAILQ_FOREACH(ifpga_dev, &ifpga_device_list, next) {
> > > > > + if (rdev &&
> > > > > + ifpga_dev->rdev &&
> > > > > + ifpga_dev->rdev == rdev)
> > > > > + return ifpga_dev;
> > > > > + }
> > > > > + return NULL;
> > > > > +}
> > > > > +
> > > > > +static struct rte_afu_device *
> > > > > +ifpga_find_afu_dev(const struct rte_ifpga_device *ifpga_dev,
> > > > > + const struct rte_afu_id *afu_id) {
> > > > > + struct rte_afu_device *afu_dev = NULL;
> > > > > +
> > > > > + TAILQ_FOREACH(afu_dev, &ifpga_dev->afu_list, next) {
> > > > > + if (!ifpga_afu_id_cmp(&afu_dev->id, afu_id))
> > > > > + return afu_dev;
> > > > > + }
> > > > > + return NULL;
> > > > > +}
> > > > > +
> > > > > +static const char * const valid_args[] = {
> > > > > +#define IFPGA_ARG_NAME "ifpga"
> > > > > + IFPGA_ARG_NAME,
> > > > > +#define IFPGA_ARG_PORT "port"
> > > > > + IFPGA_ARG_PORT,
> > > > > +#define IFPGA_AFU_BTS "afu_bts"
> > > > > + IFPGA_AFU_BTS,
> > > > > + NULL
> > > > > +};
> > > > > +
> > > > > +/*
> > > > > + * Scan the content of the FPGA bus, and the devices in the
> > > > > +devices
> > > > > + * list
> > > > > + */
> > > > > +static struct rte_afu_device *
> > > > > +ifpga_scan_one(struct rte_devargs *devargs,
> > > > > + struct rte_ifpga_device *ifpga_dev)
> > > >
> > > > usually ifpag_dev should be the first parameter here, and devargs
> > > > follows
> > >
> > > Fixed.
> > >
> > > > > +{
> > > > > + struct rte_kvargs *kvlist = NULL;
> > > > > + struct rte_rawdev *rawdev = NULL;
> > > > > + struct rte_afu_device *afu_dev = NULL;
> > > > > + struct rte_afu_pr_conf afu_pr_conf;
> > > > > + int ret = 0;
> > > > > + char *path = NULL;
> > > > > +
> > > > > + memset(&afu_pr_conf, 0, sizeof(struct rte_afu_pr_conf));
> > > > > +
> > > > > + kvlist = rte_kvargs_parse(devargs->args, valid_args);
> > > > > + if (!kvlist) {
> > > > > + IFPGA_BUS_ERR("error when parsing param");
> > > > > + goto end;
> > > > > + }
> > > > > +
> > > > > + if (rte_kvargs_count(kvlist, IFPGA_ARG_PORT) == 1) {
> > > > > + if (rte_kvargs_process(kvlist, IFPGA_ARG_PORT,
> > > > > + &rte_ifpga_get_integer32_arg, &afu_pr_conf.afu_id.port) < 0)
> > > > {
> > > > > + IFPGA_BUS_ERR("error to parse %s",
> > > > > + IFPGA_ARG_PORT);
> > > > > + goto end;
> > > > > + }
> > > > > + } else {
> > > > > + IFPGA_BUS_ERR("arg %s is mandatory for ifpga bus",
> > > > > + IFPGA_ARG_PORT);
> > > > > + goto end;
> > > > > + }
> > > > > +
> > > > > + if (rte_kvargs_count(kvlist, IFPGA_AFU_BTS) == 1) {
> > > > > + if (rte_kvargs_process(kvlist, IFPGA_AFU_BTS,
> > > > > + &rte_ifpga_get_string_arg, &path) < 0) {
> > > > > + IFPGA_BUS_ERR("Failed to parse %s",
> > > > > + IFPGA_AFU_BTS);
> > > > > + goto end;
> > > > > + }
> > > > > + } else {
> > > > > + IFPGA_BUS_ERR("arg %s is mandatory for ifpga bus",
> > > > > + IFPGA_AFU_BTS);
> > > > > + goto end;
> > > > > + }
> > > > > +
> > > > > + afu_pr_conf.afu_id.uuid.uuid_low = 0;
> > > > > + afu_pr_conf.afu_id.uuid.uuid_high = 0;
> > > > > + afu_pr_conf.pr_enable = path?1:0;
> > > > > +
> > > > > + rawdev = ifpga_dev->rdev;
> > > > > + if (ifpga_find_afu_dev(ifpga_dev, &afu_pr_conf.afu_id))
> > > > > + goto end;
> > > > > +
> > > > > + afu_dev = calloc(1, sizeof(*afu_dev));
> > > > > + if (!afu_dev)
> > > > > + goto end;
> > > > > +
> > > > > + afu_dev->device.devargs = devargs;
> > > > > + afu_dev->device.numa_node = SOCKET_ID_ANY;
> > > > > + afu_dev->device.name = devargs->name;
> > > > > + afu_dev->rawdev = rawdev;
> > > > > + afu_dev->id.uuid.uuid_low = 0;
> > > > > + afu_dev->id.uuid.uuid_high = 0;
> > > > > + afu_dev->id.port = afu_pr_conf.afu_id.port;
> > > > > + afu_dev->ifpga_dev = ifpga_dev;
> > > > > +
> > > > > + if (rawdev->dev_ops && rawdev->dev_ops->dev_info_get)
> > > > > + rawdev->dev_ops->dev_info_get(rawdev, afu_dev);
> > > > > +
> > > > > + if (rawdev->dev_ops &&
> > > > > + rawdev->dev_ops->dev_start &&
> > > > > + rawdev->dev_ops->dev_start(rawdev))
> > > > > + goto free_dev;
> > > > > +
> > > > > + strncpy(afu_pr_conf.bs_path, path, sizeof(afu_pr_conf.bs_path));
> > > > > + if (rawdev->dev_ops->firmware_load &&
> > > > > + rawdev->dev_ops->firmware_load(rawdev,
> > > > > + &afu_pr_conf)){
> > > > > + IFPGA_BUS_ERR("firmware load error %d\n", ret);
> > > > > + goto free_dev;
> > > > > + }
> > > > > + afu_dev->id.uuid.uuid_low = afu_pr_conf.afu_id.uuid.uuid_low;
> > > > > + afu_dev->id.uuid.uuid_high =
> > > > > +afu_pr_conf.afu_id.uuid.uuid_high;
> > > > > +
> > > > > + return afu_dev;
> > > > > +
> > > > > +free_dev:
> > > > > + free(afu_dev);
> > > > > +end:
> > > > > + if (kvlist)
> > > > > + rte_kvargs_free(kvlist);
> > > > > + if (path)
> > > > > + free(path);
> > > > > +
> > > > > + return NULL;
> > > > > +}
> > > > > +
> > > > > +/*
> > > > > + * Scan the content of the FPGA bus, and the devices in the
> > > > > +devices
> > > > > + * list
> > > > > + */
> > > > > +static int
> > > > > +ifpga_scan(void)
> > > > > +{
> > > > > + struct rte_ifpga_device *ifpga_dev;
> > > > > + struct rte_devargs *devargs;
> > > > > + struct rte_kvargs *kvlist = NULL;
> > > > > + struct rte_rawdev *rawdev = NULL;
> > > > > + char *name = NULL;
> > > > > + char name1[RTE_RAWDEV_NAME_MAX_LEN];
> > > > > + struct rte_afu_device *afu_dev = NULL;
> > > > > +
> > > > > + /* for FPGA devices we scan the devargs_list populated via
> > > > > +cmdline */
> > > >
> > > > I didn't see "--ifpga" is supported by cmdline, either you need to
> > > > add corresponding parser or correct the comment here.
> > >
> > > To be honestly, ifpga_scan() is called by hotplug, lib_rte hotplug
> > > function will construct it.
> >
> > OK, after review patch 3, I saw it, that make sense.
> > >
> > > > > + RTE_EAL_DEVARGS_FOREACH("ifpga", devargs) {
> > > >
> > > >
> > > > IFPGA_BUS_NAME to replace "ifpga" here.
> > > >
> > > > > + if (devargs->bus != &rte_ifpga_bus)
> > > > > + continue;
> > > > > +
> > > > > + kvlist = rte_kvargs_parse(devargs->args, valid_args);
> > > > > + if (!kvlist) {
> > > > > + IFPGA_BUS_ERR("error when parsing param");
> > > > > + goto end;
> > > > > + }
> > > > > +
> > > > > + if (rte_kvargs_count(kvlist, IFPGA_ARG_NAME) == 1) {
> > > > > + if (rte_kvargs_process(kvlist, IFPGA_ARG_NAME,
> > > > > + &rte_ifpga_get_string_arg, &name) < 0) {
> > > > > + IFPGA_BUS_ERR("error to parse %s",
> > > > > + IFPGA_ARG_NAME);
> > > > > + goto end;
> > > > > + }
> > > > > + } else {
> > > > > + IFPGA_BUS_ERR("arg %s is mandatory for ifpga bus",
> > > > > + IFPGA_ARG_NAME);
> > > > > + goto end;
> > > > > + }
> > > > > +
> > > > > + memset(name1, 0, sizeof(name1));
> > > > > + snprintf(name1, RTE_RAWDEV_NAME_MAX_LEN,
> > > > "IFPGA:%s",
> > > > > name);
> > > > > +
> > > > > + rawdev = rte_rawdev_pmd_get_named_dev(name1);
> > > > > + if (!rawdev)
> > > > > + goto end;
> > > > > +
> > > > > + if (ifpga_find_ifpga_dev(rawdev))
> > > > > + continue;
> > > > > +
> > > > > + ifpga_dev = calloc(1, sizeof(*ifpga_dev));
> > > > > + if (!ifpga_dev)
> > > > > + goto end;
> > > > > +
> > > > > + ifpga_dev->rdev = rawdev;
> > > > > + TAILQ_INIT(&ifpga_dev->afu_list);
> > > > > +
> > > > > + TAILQ_INSERT_TAIL(&ifpga_device_list, ifpga_dev, next);
> > > > > + afu_dev = ifpga_scan_one(devargs, ifpga_dev);
> > > > > + if (afu_dev != NULL)
> > > > > + TAILQ_INSERT_TAIL(&ifpga_dev->afu_list, afu_dev,
> > > > next);
> > > >
> > > > I don't understand why we need afu_list?, seems we only add a new
> > > > afu_dev into the list after create a new ifpga_dev, Is there any
> > > > another place that we add one to list to make it reasonable? it
> > > > looks like either a 1:1 or 1:0 for ifpga_dev:afu_dev?
> > >
> > > One FPAG may support more than one AFU.
> >
> > But in your implementation, it is not possible that an ifpga_dev
> > contains multiple afu_dev
>
> All afu_devs are listed to ifpga_dev, and each afu_dev can be added and
> deleted by user.
Could you show me an example, how this happen, what APIs user can add an rte_afu_device to an rte_ifpga_device?
>
> > >
> > > > > + }
> > > > > +
> > > > > +end:
> > > > > + if (kvlist)
> > > > > + rte_kvargs_free(kvlist);
> > > > > + if (name)
> > > > > + free(name);
> > > > > +
> > > > > + return 0;
> > > > > +}
> > > > > +
> > > > > +/*
> > > > > + * Match the AFU Driver and AFU Device using the ID Table */
> > > > > +static int rte_afu_match(const struct rte_afu_driver *afu_drv,
> > > > > + const struct rte_afu_device *afu_dev) {
> > > > > + const struct rte_afu_uuid *id_table;
> > > > > +
> > > > > + for (id_table = afu_drv->id_table;
> > > > > + ((id_table->uuid_low != 0) && (id_table->uuid_high != 0));
> > > > > + id_table++) {
> > > > > + /* check if device's identifiers match the driver's ones */
> > > > > + if ((id_table->uuid_low != afu_dev->id.uuid.uuid_low) ||
> > > > > + id_table->uuid_high !=
> > > > > + afu_dev->id.uuid.uuid_high)
> > > > > + continue;
> > > > > +
> > > > > + return 1;
> > > > > + }
> > > > > +
> > > > > + return 0;
> > > > > +}
> > > > > +
> > > > > +static int
> > > > > +ifpga_probe_one_driver(struct rte_afu_driver *drv,
> > > > > + struct rte_afu_device *afu_dev) {
> > > > > + int ret;
> > > > > +
> > > > > + if (!rte_afu_match(drv, afu_dev))
> > > > > + /* Match of device and driver failed */
> > > > > + return 1;
> > > > > +
> > > > > + /* reference driver structure */
> > > > > + afu_dev->driver = drv;
> > > > > + afu_dev->device.driver = &drv->driver;
> > > > > +
> > > > > + /* call the driver probe() function */
> > > > > + ret = drv->probe(afu_dev);
> > > > > + if (ret) {
> > > > > + afu_dev->driver = NULL;
> > > > > + afu_dev->device.driver = NULL;
> > > > > + }
> > > > > +
> > > > > + return ret;
> > > > > +}
> > > > > +
> > > > > +static int
> > > > > +ifpga_probe_all_drivers(struct rte_afu_device *afu_dev) {
> > > > > + struct rte_afu_driver *drv = NULL;
> > > > > + int ret = 0;
> > > > > +
> > > > > + if (afu_dev == NULL)
> > > > > + return -1;
> > > > > +
> > > > > + /* Check if a driver is already loaded */
> > > > > + if (afu_dev->driver != NULL)
> > > > > + return 0;
> > > > > +
> > > > > + TAILQ_FOREACH(drv, &afu_driver_list, next) {
> > > > > + if (ifpga_probe_one_driver(drv, afu_dev)) {
> > > > > + ret = -1;
> > > > > + break;
> > > > > + }
> > > > > + }
> > > > > + return ret;
> > > > > +}
> > > > > +
> > > > > +/*
> > > > > + * Scan the content of the Intel FPGA bus, and call the probe()
> > > > > +function for
> > > > > + * all registered drivers that have a matching entry in its
> > > > > +id_table
> > > > > + * for discovered devices.
> > > > > + */
> > > > > +static int
> > > > > +ifpga_probe(void)
> > > > > +{
> > > > > + struct rte_ifpga_device *ifpga_dev;
> > > > > + struct rte_afu_device *afu_dev = NULL;
> > > > > + int ret = 0;
> > > > > +
> > > > > + TAILQ_FOREACH(ifpga_dev, &ifpga_device_list, next) {
> > > > > + TAILQ_FOREACH(afu_dev, &ifpga_dev->afu_list, next) {
> > > > > +
> > > > > + if (afu_dev->device.driver)
> > > > > + continue;
> > > > > +
> > > > > + ret = ifpga_probe_all_drivers(afu_dev);
> > > > > + if (ret < 0)
> > > > > + IFPGA_BUS_ERR("failed to initialize %s
> > > > device\n",
> > > > > + rte_ifpga_device_name(afu_dev));
> > > > > + }
> > > > > + }
> > > > > +
> > > > > + return ret;
> > > > > +}
> > > > > +
> > > > > +static int
> > > > > +ifpga_plug(struct rte_device *dev) {
> > > > > + return ifpga_probe_all_drivers(RTE_DEV_TO_AFU(dev));
> > > > > +}
> > > > > +
> > > > > +static int
> > > > > +ifpga_remove_driver(struct rte_afu_device *afu_dev) {
> > > > > + const char *name;
> > > > > + const struct rte_afu_driver *driver;
> > > > > +
> > > > > + name = rte_ifpga_device_name(afu_dev);
> > > > > + if (!afu_dev->device.driver) {
> > > > > + IFPGA_BUS_DEBUG("no driver attach to device %s\n", name);
> > > > > + return 1;
> > > > > + }
> > > > > +
> > > > > + driver = RTE_DRV_TO_AFU_CONST(afu_dev->device.driver);
> > > > > + return driver->remove(afu_dev); }
> > > > > +
> > > > > +static int
> > > > > +ifpga_unplug(struct rte_device *dev) {
> > > > > + struct rte_ifpga_device *ifpga_dev = NULL;
> > > > > + struct rte_afu_device *afu_dev = NULL;
> > > > > + struct rte_devargs *devargs = NULL;
> > > > > + int ret;
> > > > > +
> > > > > + if (dev == NULL)
> > > > > + return -EINVAL;
> > > > > +
> > > > > + afu_dev = RTE_DEV_TO_AFU(dev);
> > > > > + if (!dev)
> > > > > + return -ENOENT;
> > > > > +
> > > > > + ifpga_dev = afu_dev->ifpga_dev;
> > > > > + devargs = dev->devargs;
> > > > > +
> > > > > + ret = ifpga_remove_driver(afu_dev);
> > > >
> > > > So what is the device type that be plugged into the ifgpa bus?
> > > > ifpga_dev or afu_dev? If its afu_dev, why we need ifpga_dev?
> > > > If it is ifpga_dev, we need remove ifpga_dev from ifpga_device_list here.
> > >
> > > The device type that be plugged into the ifpga bus is afu_dev.
> >
> > In DPDK, we have rte_pci_device for pci bus and rte_vdev_device for
> > vdev bus Now we have rte_ifpga_device not for ifgpa bus but
> > rte_afu_device, it is a little bit confuse here
>
> The relationship of this items is descripted in ifpga_rawdev.rst.
>
> > > And each afu_dev will be added into one ifpga_dev.
> > > Each FPGA device bind to ifpga_dev.
> >
> > So what is the relationship between ifpga_dev and raw_dev?
>
> One FPGA as a PCI device will probe ifpga_rawdev.
> For management afu_dev, we create ifpga_dev in ifpga_bus.
So 1 ifpga_dev map to 1 raw device?
>
> > >
> > > > > + if (ret)
> > > > > + return ret;
> > > > > +
> > > > > + TAILQ_REMOVE(&ifpga_dev->afu_list, afu_dev, next);
> > > > > +
> > > > > + rte_devargs_remove(devargs->bus->name, devargs->name);
> > > > > + free(afu_dev);
> > > > > + return 0;
> > > > > +
> > > > > +}
> > > > > +
> > > > > +static struct rte_device *
> > > > > +ifpga_find_device(const struct rte_device *start,
> > > > > + rte_dev_cmp_t cmp, const void *data) {
> > > > > + struct rte_ifpga_device *ifpga_dev;
> > > > > + struct rte_afu_device *afu_dev;
> > > > > +
> > > > > + TAILQ_FOREACH(ifpga_dev, &ifpga_device_list, next) {
> > > > > + TAILQ_FOREACH(afu_dev, &ifpga_dev->afu_list, next) {
> > > > > + if (start && &afu_dev->device == start) {
> > > > > + start = NULL;
> > > > > + continue;
> > > > > + }
> > > > > + if (cmp(&afu_dev->device, data) == 0)
> > > > > + return &afu_dev->device;
> > > > > + }
> > > > > + }
> > > > > + return NULL;
> > > > > +}
> > > > > +static int
> > > > > +ifpga_parse(const char *name, void *addr) {
> > > > > + int *out = addr;
> > > > > + struct rte_rawdev *rawdev = NULL;
> > > > > + char rawdev_name[RTE_RAWDEV_NAME_MAX_LEN];
> > > > > + char *c1 = NULL, *c2 = NULL;
> > > > > + int port = IFPGA_BUS_DEV_PORT_MAX;
> > > > > + char str_port[8];
> > > > > + int str_port_len = 0;
> > > > > + int ret;
> > > > > +
> > > > > + memset(str_port, 0, 8);
> > > > > + c1 = strchr(name, '|');
> > > > > + if (c1 != NULL) {
> > > > > + str_port_len = c1-name;
> > > > > + c2 = c1+1;
> > > > > + }
> > > > > +
> > > > > + if (str_port_len < 8 &&
> > > > > + str_port_len > 0) {
> > > > > + memcpy(str_port, name, str_port_len);
> > > > > + ret = sscanf(str_port, "%d", &port);
> > > > > + if (ret == -1)
> > > > > + return 0;
> > > > > + }
> > > > > +
> > > > > + memset(rawdev_name, 0, sizeof(rawdev_name));
> > > > > + snprintf(rawdev_name, RTE_RAWDEV_NAME_MAX_LEN,
> "IFPGA:%s",
> > > > c2);
> > > > > + rawdev = rte_rawdev_pmd_get_named_dev(rawdev_name);
> > > > > +
> > > > > + if ((port < IFPGA_BUS_DEV_PORT_MAX) &&
> > > > > + rawdev &&
> > > > > + (addr != NULL))
> > > > > + *out = port;
> > > > > +
> > > > > + if ((port < IFPGA_BUS_DEV_PORT_MAX) &&
> > > > > + rawdev)
> > > > > + return 0;
> > > > > + else
> > > > > + return 1;
> > > > > +}
> > > > > +
> > > > > +static struct rte_bus rte_ifpga_bus = {
> > > > > + .scan = ifpga_scan,
> > > > > + .probe = ifpga_probe,
> > > > > + .find_device = ifpga_find_device,
> > > > > + .plug = ifpga_plug,
> > > > > + .unplug = ifpga_unplug,
> > > > > + .parse = ifpga_parse,
> > > > > +};
> > > > > +
> > > > > +RTE_REGISTER_BUS(IFPGA_BUS_NAME, rte_ifpga_bus);
> > > > > +
> > > > > +RTE_INIT(ifpga_init_log)
> > > > > +{
> > > > > + ifpga_bus_logtype = rte_log_register("bus.ifpga");
> > > > > + if (ifpga_bus_logtype >= 0)
> > > > > + rte_log_set_level(ifpga_bus_logtype, RTE_LOG_NOTICE); }
> > > > > diff --git a/drivers/bus/ifpga/ifpga_common.c
> > > > > b/drivers/bus/ifpga/ifpga_common.c
> > > > > new file mode 100644
> > > > > index 0000000..78e2eae
> > > > > --- /dev/null
> > > > > +++ b/drivers/bus/ifpga/ifpga_common.c
> > > > > @@ -0,0 +1,88 @@
> > > > > +/* SPDX-License-Identifier: BSD-3-Clause
> > > > > + * Copyright(c) 2010-2018 Intel Corporation */
> > > > > +
> > > > > +#include <string.h>
> > > > > +#include <inttypes.h>
> > > > > +#include <stdint.h>
> > > > > +#include <stdlib.h>
> > > > > +#include <stdio.h>
> > > > > +#include <sys/queue.h>
> > > > > +#include <sys/mman.h>
> > > > > +#include <sys/types.h>
> > > > > +#include <unistd.h>
> > > > > +#include <fcntl.h>
> > > > > +
> > > > > +#include <rte_errno.h>
> > > > > +#include <rte_bus.h>
> > > > > +#include <rte_per_lcore.h>
> > > > > +#include <rte_memory.h>
> > > > > +#include <rte_memzone.h>
> > > > > +#include <rte_eal.h>
> > > > > +#include <rte_common.h>
> > > > > +
> > > > > +#include <rte_devargs.h>
> > > > > +#include <rte_kvargs.h>
> > > > > +#include <rte_alarm.h>
> > > > > +
> > > > > +#include "rte_bus_ifpga.h"
> > > > > +#include "ifpga_logs.h"
> > > > > +#include "ifpga_common.h"
> > > > > +
> > > > > +int rte_ifpga_get_string_arg(const char *key __rte_unused,
> > > > > + const char *value, void *extra_args) {
> > > > > + if (!value || !extra_args)
> > > > > + return -EINVAL;
> > > > > +
> > > > > + *(char **)extra_args = strdup(value);
> > > > > +
> > > > > + if (!*(char **)extra_args)
> > > > > + return -ENOMEM;
> > > > > +
> > > > > + return 0;
> > > > > +}
> > > > > +int rte_ifpga_get_integer32_arg(const char *key __rte_unused,
> > > > > + const char *value, void *extra_args) {
> > > > > + if (!value || !extra_args)
> > > > > + return -EINVAL;
> > > > > +
> > > > > + *(int *)extra_args = strtoull(value, NULL, 0);
> > > > > +
> > > > > + return 0;
> > > > > +}
> > > > > +int ifpga_get_integer64_arg(const char *key __rte_unused,
> > > > > + const char *value, void *extra_args) {
> > > > > + if (!value || !extra_args)
> > > > > + return -EINVAL;
> > > > > +
> > > > > + *(uint64_t *)extra_args = strtoull(value, NULL, 0);
> > > > > +
> > > > > + return 0;
> > > > > +}
> > > > > +int ifpga_get_unsigned_long(const char *str, int base) {
> > > > > + unsigned long num;
> > > > > + char *end = NULL;
> > > > > +
> > > > > + errno = 0;
> > > > > +
> > > > > + num = strtoul(str, &end, base);
> > > > > + if ((str[0] == '\0') || (end == NULL) || (*end != '\0') || (errno != 0))
> > > > > + return -1;
> > > > > +
> > > > > + return num;
> > > > > +}
> > > > > +
> > > > > +int ifpga_afu_id_cmp(const struct rte_afu_id *afu_id0,
> > > > > + const struct rte_afu_id *afu_id1) {
> > > > > + if ((afu_id0->uuid.uuid_low == afu_id1->uuid.uuid_low) &&
> > > > > + (afu_id0->uuid.uuid_high == afu_id1->uuid.uuid_high) &&
> > > > > + (afu_id0->port == afu_id1->port)) {
> > > > > + return 0;
> > > > > + } else
> > > > > + return 1;
> > > > > +}
> > > > > diff --git a/drivers/bus/ifpga/ifpga_common.h
> > > > > b/drivers/bus/ifpga/ifpga_common.h
> > > > > new file mode 100644
> > > > > index 0000000..f9254b9
> > > > > --- /dev/null
> > > > > +++ b/drivers/bus/ifpga/ifpga_common.h
> > > > > @@ -0,0 +1,18 @@
> > > > > +/* SPDX-License-Identifier: BSD-3-Clause
> > > > > + * Copyright(c) 2010-2018 Intel Corporation */
> > > > > +
> > > > > +#ifndef _IFPGA_COMMON_H_
> > > > > +#define _IFPGA_COMMON_H_
> > > > > +
> > > > > +int rte_ifpga_get_string_arg(const char *key __rte_unused,
> > > > > + const char *value, void *extra_args); int
> > > > > +rte_ifpga_get_integer32_arg(const char *key __rte_unused,
> > > > > + const char *value, void *extra_args); int
> > > > > +ifpga_get_integer64_arg(const char *key __rte_unused,
> > > > > + const char *value, void *extra_args); int
> > > > > +ifpga_get_unsigned_long(const char *str, int base); int
> > > > > +ifpga_afu_id_cmp(const struct rte_afu_id *afu_id0,
> > > > > + const struct rte_afu_id *afu_id1);
> > > > > +
> > > > > +#endif /* _IFPGA_COMMON_H_ */
> > > > > diff --git a/drivers/bus/ifpga/ifpga_logs.h
> > > > > b/drivers/bus/ifpga/ifpga_logs.h new file mode 100644 index
> > > > > 0000000..873e0a4
> > > > > --- /dev/null
> > > > > +++ b/drivers/bus/ifpga/ifpga_logs.h
> > > > > @@ -0,0 +1,31 @@
> > > > > +/* SPDX-License-Identifier: BSD-3-Clause
> > > > > + * Copyright(c) 2010-2018 Intel Corporation */
> > > > > +
> > > > > +#ifndef _IFPGA_LOGS_H_
> > > > > +#define _IFPGA_LOGS_H_
> > > > > +
> > > > > +#include <rte_log.h>
> > > > > +
> > > > > +extern int ifpga_bus_logtype;
> > > > > +
> > > > > +#define IFPGA_LOG(level, fmt, args...) \
> > > > > + rte_log(RTE_LOG_ ## level, ifpga_bus_logtype, "%s(): " fmt "\n", \
> > > > > + __func__, ##args)
> > > > > +
> > > > > +#define IFPGA_BUS_LOG(level, fmt, args...) \
> > > > > + rte_log(RTE_LOG_ ## level, ifpga_bus_logtype, "%s(): " fmt "\n", \
> > > > > + __func__, ##args)
> > > > > +
> > > > > +#define IFPGA_BUS_FUNC_TRACE() IFPGA_BUS_LOG(DEBUG, ">>")
> > > > > +
> > > > > +#define IFPGA_BUS_DEBUG(fmt, args...) \
> > > > > + IFPGA_BUS_LOG(DEBUG, fmt, ## args) #define IFPGA_BUS_INFO(fmt,
> > > > > +args...) \
> > > > > + IFPGA_BUS_LOG(INFO, fmt, ## args) #define IFPGA_BUS_ERR(fmt,
> > > > > +args...) \
> > > > > + IFPGA_BUS_LOG(ERR, fmt, ## args) #define IFPGA_BUS_WARN(fmt,
> > > > > +args...) \
> > > > > + IFPGA_BUS_LOG(WARNING, fmt, ## args)
> > > > > +
> > > > > +#endif /* _IFPGA_BUS_LOGS_H_ */
> > > > > diff --git a/drivers/bus/ifpga/meson.build
> > > > > b/drivers/bus/ifpga/meson.build new file mode 100644 index
> > > > > 0000000..c9b08c8
> > > > > --- /dev/null
> > > > > +++ b/drivers/bus/ifpga/meson.build
> > > > > @@ -0,0 +1,8 @@
> > > > > +# SPDX-License-Identifier: BSD-3-Clause # Copyright(c)
> > > > > +2010-2018 Intel Corporation
> > > > > +
> > > > > +deps += ['pci', 'kvargs', 'rawdev']
> > > > > +install_headers('rte_bus_ifpga.h')
> > > > > +sources = files('ifpga_common.c', 'ifpga_bus.c')
> > > > > +
> > > > > +allow_experimental_apis = true
> > > > > diff --git a/drivers/bus/ifpga/rte_bus_ifpga.h
> > > > > b/drivers/bus/ifpga/rte_bus_ifpga.h
> > > > > new file mode 100644
> > > > > index 0000000..5c559e1
> > > > > --- /dev/null
> > > > > +++ b/drivers/bus/ifpga/rte_bus_ifpga.h
> > > > > @@ -0,0 +1,160 @@
> > > > > +/* SPDX-License-Identifier: BSD-3-Clause
> > > > > + * Copyright(c) 2010-2018 Intel Corporation */
> > > > > +
> > > > > +#ifndef _RTE_BUS_IFPGA_H_
> > > > > +#define _RTE_BUS_IFPGA_H_
> > > > > +
> > > > > +/**
> > > > > + * @file
> > > > > + *
> > > > > + * RTE Intel FPGA Bus Interface */
> > > > > +
> > > > > +#ifdef __cplusplus
> > > > > +extern "C" {
> > > > > +#endif
> > > > > +
> > > > > +#include <rte_bus.h>
> > > > > +#include <rte_pci.h>
> > > > > +
> > > > > +/** Name of Intel FPGA Bus */
> > > > > +#define IFPGA_BUS_NAME ifpga
> > > > > +
> > > > > +/* Forward declarations */
> > > > > +struct rte_afu_device;
> > > > > +
> > > > > +/** List of Intel AFU devices */ TAILQ_HEAD(afu_device_list,
> > > > > +rte_afu_device);
> > > > > +/** Double linked list of AFU device drivers. */
> > > > > +TAILQ_HEAD(afu_driver_list, rte_afu_driver);
> > > > > +
> > > > > +#define IFPGA_BUS_BITSTREAM_PATH_MAX_LEN 256
> > > > > +
> > > > > +struct rte_afu_uuid {
> > > > > + uint64_t uuid_low;
> > > > > + uint64_t uuid_high;
> > > > > +} __attribute__ ((packed));
> > > > > +
> > > > > +#define IFPGA_BUS_DEV_PORT_MAX 4
> > > > > +
> > > > > +/**
> > > > > + * A structure describing an ID for a AFU driver. Each driver
> > > > > +provides a
> > > > > + * table of these IDs for each device that it supports.
> > > > > + */
> > > > > +struct rte_afu_id {
> > > > > + struct rte_afu_uuid uuid;
> > > > > + int port; /**< port number */
> > > > > +} __attribute__ ((packed));
> > > > > +
> > > > > +/**
> > > > > + * A structure PR (Partial Reconfiguration) configuration AFU driver.
> > > > > + */
> > > > > +
> > > > > +struct rte_afu_pr_conf {
> > > > > + struct rte_afu_id afu_id;
> > > > > + int pr_enable;
> > > > > + char bs_path[IFPGA_BUS_BITSTREAM_PATH_MAX_LEN];
> > > > > +};
> > > > > +
> > > > > +#define AFU_PRI_STR_SIZE (PCI_PRI_STR_SIZE + 8)
> > > > > +
> > > > > +/**
> > > > > + * A structure describing a fpga device.
> > > > > + */
> > > > > +struct rte_ifpga_device {
> > > > > + TAILQ_ENTRY(rte_ifpga_device) next; /**< Next in device list.
> */
> > > > > + struct rte_rawdev *rdev;
> > > > > + struct afu_device_list afu_list; /**< List of AFU devices */
> > > > > +};
> > > > > +
> > > > > +/**
> > > > > + * A structure describing a AFU device.
> > > > > + */
> > > > > +struct rte_afu_device {
> > > > > + TAILQ_ENTRY(rte_afu_device) next; /**< Next in device list.
> */
> > > > > + struct rte_device device; /**< Inherit core device */
> > > > > + struct rte_rawdev *rawdev; /**< Point Rawdev */
> > > > > + struct rte_ifpga_device *ifpga_dev; /**< Point ifpga device */
> > > > > + struct rte_afu_id id; /**< AFU id within FPGA.
> */
> > > > > + uint32_t num_region; /**< number of regions found */
> > > > > + struct rte_mem_resource mem_resource[PCI_MAX_RESOURCE];
> > > > > + /**< AFU Memory Resource
> > > > */
> > > > > + struct rte_intr_handle intr_handle; /**< Interrupt handle */
> > > > > + struct rte_afu_driver *driver; /**< Associated driver */
> > > > > + char path[IFPGA_BUS_BITSTREAM_PATH_MAX_LEN];
> > > > > +} __attribute__ ((packed));
> > > > > +
> > > > > +/**
> > > > > + * @internal
> > > > > + * Helper macro for drivers that need to convert to struct
> > rte_afu_device.
> > > > > + */
> > > > > +#define RTE_DEV_TO_AFU(ptr) \
> > > > > + container_of(ptr, struct rte_afu_device, device)
> > > > > +
> > > > > +#define RTE_DRV_TO_AFU_CONST(ptr) \
> > > > > + container_of(ptr, const struct rte_afu_driver, driver)
> > > > > +
> > > > > +/**
> > > > > + * Initialisation function for the driver called during FPGA BUS probing.
> > > >
> > > >
> > > > typo: Initialization
> > >
> > > Fixed.
> > >
> > > > > + */
> > > > > +typedef int (afu_probe_t)(struct rte_afu_device *);
> > > > > +
> > > > > +/**
> > > > > + * Uninitialisation function for the driver called during hotplugging.
> > > >
> > > > typo: Unintialization
> > >
> > > Fixed.
> > >
> > > > > + */
> > > > > +typedef int (afu_remove_t)(struct rte_afu_device *);
> > > > > +
> > > > > +/**
> > > > > + * A structure describing a AFU device.
> > > > > + */
> > > > > +struct rte_afu_driver {
> > > > > + TAILQ_ENTRY(rte_afu_driver) next; /**< Next afu driver. */
> > > > > + struct rte_driver driver; /**< Inherit core driver. */
> > > > > + afu_probe_t *probe; /**< Device Probe
> function.
> > > > > */
> > > > > + afu_remove_t *remove; /**< Device Remove
> > > > > function. */
> > > > > + const struct rte_afu_uuid *id_table; /**< AFU uuid within FPGA.
> */
> > > > > + uint32_t drv_flags; /**< Flags contolling handling of device.
> */
> > > >
> > > > Typo Controlling
> > > > And what is "flags controlling handling of device?"
> > >
> > > This variable is not used, so I have removed it.
> > >
> > > > > +};
> > > > > +
> > > > > +static inline const char *
> > > > > +rte_ifpga_device_name(const struct rte_afu_device *afu) {
> > > > > + if (afu && afu->device.name)
> > > > > + return afu->device.name;
> > > > > + return NULL;
> > > > > +}
> > > > > +
> > > > > +/**
> > > > > + * Register a ifpga afu device driver.
> > > > > + *
> > > > > + * @param driver
> > > > > + * A pointer to a rte_afu_driver structure describing the driver
> > > > > + * to be registered.
> > > > > + */
> > > > > +void rte_ifpga_driver_register(struct rte_afu_driver *driver);
> > > > > +
> > > > > +/**
> > > > > + * Unregister a ifpga afu device driver.
> > > > > + *
> > > > > + * @param driver
> > > > > + * A pointer to a rte_afu_driver structure describing the driver
> > > > > + * to be unregistered.
> > > > > + */
> > > > > +void rte_ifpga_driver_unregister(struct rte_afu_driver
> > > > > +*driver);
> > > > > +
> > > > > +#define RTE_PMD_REGISTER_AFU(nm, afudrv)\
> > > > > +RTE_INIT(afudrvinitfn_ ##afudrv);\ static const char
> > > > > +*afudrvinit_ ## nm ## _alias;\ static void afudrvinitfn_ ##afudrv(void)\
> {\
> > > > > + (afudrv).driver.name = RTE_STR(nm);\
> > > > > + (afudrv).driver.alias = afudrvinit_ ## nm ## _alias;\
> > > > > + rte_ifpga_driver_register(&afudrv);\
> > > > > +} \
> > > > > +RTE_PMD_EXPORT_NAME(nm, __COUNTER__)
> > > > > +
> > > > > +#define RTE_PMD_REGISTER_AFU_ALIAS(nm, alias)\ static const
> > > > > +char *afudrvinit_ ## nm ## _alias = RTE_STR(alias)
> > > > > +
> > > > > +#endif /* _RTE_BUS_IFPGA_H_ */
> > > > > diff --git a/drivers/bus/ifpga/rte_bus_ifpga_version.map
> > > > > b/drivers/bus/ifpga/rte_bus_ifpga_version.map
> > > > > new file mode 100644
> > > > > index 0000000..a027979
> > > > > --- /dev/null
> > > > > +++ b/drivers/bus/ifpga/rte_bus_ifpga_version.map
> > > > > @@ -0,0 +1,10 @@
> > > > > +DPDK_18.05 {
> > > > > + global:
> > > > > +
> > > > > + rte_ifpga_get_integer32_arg;
> > > > > + rte_ifpga_get_string_arg;
> > > > > + rte_ifpga_driver_register;
> > > > > + rte_ifpga_driver_unregister;
> > > > > +
> > > > > + local: *;
> > > > > +};
> > > > > diff --git a/drivers/bus/meson.build b/drivers/bus/meson.build
> > > > > index 58dfbe2..52c755d 100644
> > > > > --- a/drivers/bus/meson.build
> > > > > +++ b/drivers/bus/meson.build
> > > > > @@ -1,7 +1,7 @@
> > > > > # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2017
> > > > > Intel Corporation
> > > > >
> > > > > -drivers = ['dpaa', 'fslmc', 'pci', 'vdev']
> > > > > +drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'vdev']
> > > > > std_deps = ['eal']
> > > > > config_flag_fmt = 'RTE_LIBRTE_@0@_BUS'
> > > > > driver_name_fmt = 'rte_bus_@0@'
> > > > > diff --git a/mk/rte.app.mk b/mk/rte.app.mk index
> > > > > 26f3563..3861e1a
> > > > > 100644
> > > > > --- a/mk/rte.app.mk
> > > > > +++ b/mk/rte.app.mk
> > > > > @@ -255,6 +255,7 @@ ifeq
> > > > > ($(CONFIG_RTE_EAL_VFIO)$(CONFIG_RTE_LIBRTE_FSLMC_BUS),yy)
> > > > > _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_CMDIF_RAWDEV) +=
> > > > > -lrte_pmd_dpaa2_cmdif
> > > > > _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV) +=
> > > > > -lrte_pmd_dpaa2_qdma endif # CONFIG_RTE_LIBRTE_FSLMC_BUS
> > > > > +_LDLIBS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += -lrte_bus_ifpga
> > > > > endif # CONFIG_RTE_LIBRTE_RAWDEV
> > > > >
> > > > >
> > > > > --
> > > > > 1.8.3.1
> > > >
> > > > Regards
> > > > Qi
10/05/2018 15:51, Xu, Rosen:
> Hi Qi,
>
> I miss one comment, so I fix it in this email.
In order to avoid missing comments, and to ease reading for others,
please remove the useless context lines when replying.
> -----Original Message-----
> From: Xu, Rosen
> Sent: Thursday, May 10, 2018 8:21 PM
> To: Wu, Jingjing <jingjing.wu@intel.com>; dev@dpdk.org; thomas@monjalon.net
> Cc: Zhang, Roy Fan <roy.fan.zhang@intel.com>; Doherty, Declan
> <declan.doherty@intel.com>; Richardson, Bruce <bruce.richardson@intel.com>;
> shreyansh.jain@nxp.com; Yigit, Ferruh <ferruh.yigit@intel.com>; Ananyev, Konstantin
> <konstantin.ananyev@intel.com>; Zhang, Tianfei <tianfei.zhang@intel.com>; Liu, Song
> <song.liu@intel.com>; Wu, Hao <hao.wu@intel.com>; gaetan.rivet@6wind.com
> Subject: RE: [dpdk-dev] [PATCH v10 1/3] bus/ifpga: Add Intel FPGA BUS Library
>
> Hi Jingjing,
>
> > -----Original Message-----
> > From: Wu, Jingjing
> > Sent: Thursday, May 10, 2018 16:44
> > To: Xu, Rosen <rosen.xu@intel.com>; dev@dpdk.org; thomas@monjalon.net
> > Cc: Xu, Rosen <rosen.xu@intel.com>; Zhang, Roy Fan
> > <roy.fan.zhang@intel.com>; Doherty, Declan <declan.doherty@intel.com>;
> > Richardson, Bruce <bruce.richardson@intel.com>; shreyansh.jain@nxp.com;
> > Yigit, Ferruh <ferruh.yigit@intel.com>; Ananyev, Konstantin
> > <konstantin.ananyev@intel.com>; Zhang, Tianfei <tianfei.zhang@intel.com>;
> > Liu, Song <song.liu@intel.com>; Wu, Hao <hao.wu@intel.com>;
> > gaetan.rivet@6wind.com
> > Subject: RE: [dpdk-dev] [PATCH v10 1/3] bus/ifpga: Add Intel FPGA BUS
> > Library
> >
> > Hi, Rosen
> >
> > Few comments below.
>
> Thanks a lot Jingjing.
>
> >
> > Thanks
> > Jingjing
> >
> > [......]
> > > +static struct rte_ifpga_device *
> > > +ifpga_find_ifpga_dev(const struct rte_rawdev *rdev) {
> > > + struct rte_ifpga_device *ifpga_dev = NULL;
> > > +
> > > + TAILQ_FOREACH(ifpga_dev, &ifpga_device_list, next) {
> > > + if (rdev &&
> > rdev -> ifpage_dev ??
>
> Rdev doesn't has this variable.
I mean "if (ifpag_dev" & to replace "rdev"
>
> > > + ifpga_dev->rdev &&
> > > + ifpga_dev->rdev == rdev)
> > > + return ifpga_dev;
> > > + }
> > > + return NULL;
> > > +}
> > > +
> > > +static struct rte_afu_device *
> > > +ifpga_find_afu_dev(const struct rte_ifpga_device *ifpga_dev,
> > > + const struct rte_afu_id *afu_id)
> > > +{
> > > + struct rte_afu_device *afu_dev = NULL;
> > > +
> > > + TAILQ_FOREACH(afu_dev, &ifpga_dev->afu_list, next) {
> > > + if (!ifpga_afu_id_cmp(&afu_dev->id, afu_id))
> > Add checking afu_dev?
>
> Fixed.
>
> > [...]
> >
> > > +static int
> > > +ifpga_parse(const char *name, void *addr) {
> > > + int *out = addr;
> > > + struct rte_rawdev *rawdev = NULL;
> > > + char rawdev_name[RTE_RAWDEV_NAME_MAX_LEN];
> > > + char *c1 = NULL, *c2 = NULL;
> > According to coding style, we need to two lines for the definition like:
> > char *c1 = NULL;
> > char *c2 = NULL;
>
> Fixed
>
> > > + int port = IFPGA_BUS_DEV_PORT_MAX;
> > > + char str_port[8];
> > > + int str_port_len = 0;
> > > + int ret;
> > > +
> > > + memset(str_port, 0, 8);
> > > + c1 = strchr(name, '|');
> > > + if (c1 != NULL) {
> > > + str_port_len = c1-name;
> > According to coding style, spaces are required around opreations.
>
> Fixed.
>
> > > + c2 = c1+1;
> > > + }
Hi Jingjing,
> -----Original Message-----
> From: Wu, Jingjing
> Sent: Friday, May 11, 2018 6:40
> To: Xu, Rosen <rosen.xu@intel.com>; dev@dpdk.org; thomas@monjalon.net
> Cc: Zhang, Roy Fan <roy.fan.zhang@intel.com>; Doherty, Declan
> <declan.doherty@intel.com>; Richardson, Bruce
> <bruce.richardson@intel.com>; shreyansh.jain@nxp.com; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Ananyev, Konstantin
> <konstantin.ananyev@intel.com>; Zhang, Tianfei <tianfei.zhang@intel.com>;
> Liu, Song <song.liu@intel.com>; Wu, Hao <hao.wu@intel.com>;
> gaetan.rivet@6wind.com
> Subject: RE: [dpdk-dev] [PATCH v10 1/3] bus/ifpga: Add Intel FPGA BUS
> Library
>
>
>
> > -----Original Message-----
> > From: Xu, Rosen
> > Sent: Thursday, May 10, 2018 8:21 PM
> > To: Wu, Jingjing <jingjing.wu@intel.com>; dev@dpdk.org;
> > thomas@monjalon.net
> > Cc: Zhang, Roy Fan <roy.fan.zhang@intel.com>; Doherty, Declan
> > <declan.doherty@intel.com>; Richardson, Bruce
> > <bruce.richardson@intel.com>; shreyansh.jain@nxp.com; Yigit, Ferruh
> > <ferruh.yigit@intel.com>; Ananyev, Konstantin
> > <konstantin.ananyev@intel.com>; Zhang, Tianfei
> > <tianfei.zhang@intel.com>; Liu, Song <song.liu@intel.com>; Wu, Hao
> > <hao.wu@intel.com>; gaetan.rivet@6wind.com
> > Subject: RE: [dpdk-dev] [PATCH v10 1/3] bus/ifpga: Add Intel FPGA BUS
> > Library
> >
> > Hi Jingjing,
> >
> > > -----Original Message-----
> > > From: Wu, Jingjing
> > > Sent: Thursday, May 10, 2018 16:44
> > > To: Xu, Rosen <rosen.xu@intel.com>; dev@dpdk.org;
> > > thomas@monjalon.net
> > > Cc: Xu, Rosen <rosen.xu@intel.com>; Zhang, Roy Fan
> > > <roy.fan.zhang@intel.com>; Doherty, Declan
> > > <declan.doherty@intel.com>; Richardson, Bruce
> > > <bruce.richardson@intel.com>; shreyansh.jain@nxp.com; Yigit, Ferruh
> > > <ferruh.yigit@intel.com>; Ananyev, Konstantin
> > > <konstantin.ananyev@intel.com>; Zhang, Tianfei
> > > <tianfei.zhang@intel.com>; Liu, Song <song.liu@intel.com>; Wu, Hao
> > > <hao.wu@intel.com>; gaetan.rivet@6wind.com
> > > Subject: RE: [dpdk-dev] [PATCH v10 1/3] bus/ifpga: Add Intel FPGA
> > > BUS Library
> > >
> > > Hi, Rosen
> > >
> > > Few comments below.
> >
> > Thanks a lot Jingjing.
> >
> > >
> > > Thanks
> > > Jingjing
> > >
> > > [......]
> > > > +static struct rte_ifpga_device *
> > > > +ifpga_find_ifpga_dev(const struct rte_rawdev *rdev) {
> > > > + struct rte_ifpga_device *ifpga_dev = NULL;
> > > > +
> > > > + TAILQ_FOREACH(ifpga_dev, &ifpga_device_list, next) {
> > > > + if (rdev &&
> > > rdev -> ifpage_dev ??
> >
> > Rdev doesn't has this variable.
> I mean "if (ifpag_dev" & to replace "rdev"
I will fixed it in patch v12.
> >
> > > > + ifpga_dev->rdev &&
> > > > + ifpga_dev->rdev == rdev)
> > > > + return ifpga_dev;
> > > > + }
> > > > + return NULL;
> > > > +}
> > > > +
> > > > +static struct rte_afu_device *
> > > > +ifpga_find_afu_dev(const struct rte_ifpga_device *ifpga_dev,
> > > > + const struct rte_afu_id *afu_id) {
> > > > + struct rte_afu_device *afu_dev = NULL;
> > > > +
> > > > + TAILQ_FOREACH(afu_dev, &ifpga_dev->afu_list, next) {
> > > > + if (!ifpga_afu_id_cmp(&afu_dev->id, afu_id))
> > > Add checking afu_dev?
> >
> > Fixed.
> >
> > > [...]
> > >
> > > > +static int
> > > > +ifpga_parse(const char *name, void *addr) {
> > > > + int *out = addr;
> > > > + struct rte_rawdev *rawdev = NULL;
> > > > + char rawdev_name[RTE_RAWDEV_NAME_MAX_LEN];
> > > > + char *c1 = NULL, *c2 = NULL;
> > > According to coding style, we need to two lines for the definition like:
> > > char *c1 = NULL;
> > > char *c2 = NULL;
> >
> > Fixed
> >
> > > > + int port = IFPGA_BUS_DEV_PORT_MAX;
> > > > + char str_port[8];
> > > > + int str_port_len = 0;
> > > > + int ret;
> > > > +
> > > > + memset(str_port, 0, 8);
> > > > + c1 = strchr(name, '|');
> > > > + if (c1 != NULL) {
> > > > + str_port_len = c1-name;
> > > According to coding style, spaces are required around opreations.
> >
> > Fixed.
> >
> > > > + c2 = c1+1;
> > > > + }
@@ -383,6 +383,10 @@ F: drivers/mempool/bucket/
Bus Drivers
-----------
+Intel FPGA buses
+M: Rosen Xu <rosen.xu@intel.com>
+F: drivers/bus/ifpga/
+
NXP buses
M: Hemant Agrawal <hemant.agrawal@nxp.com>
M: Shreyansh Jain <shreyansh.jain@nxp.com>
@@ -139,6 +139,11 @@ CONFIG_RTE_ETHDEV_PROFILE_ITT_WASTED_RX_ITERATIONS=n
CONFIG_RTE_ETHDEV_TX_PREPARE_NOOP=n
#
+# Compile the Intel FPGA bus
+#
+CONFIG_RTE_LIBRTE_IFPGA_BUS=y
+
+#
# Compile PCI bus driver
#
CONFIG_RTE_LIBRTE_PCI_BUS=y
@@ -183,6 +183,11 @@ New Features
stats/xstats on shared memory from secondary process, and also pdump packets on
those virtual devices.
+* **Added Ifpga Bus, a generic Intel FPGA Bus library.**
+
+ The Ifpga Bus library provides support for integrating any Intel FPGA device with
+ the DPDK framework. It provides Intel FPGA Partial Bit Stream AFU(Accelerated
+ Function Unit) scan and drivers prove.
API Changes
-----------
@@ -7,6 +7,7 @@ DIRS-$(CONFIG_RTE_LIBRTE_DPAA_BUS) += dpaa
ifeq ($(CONFIG_RTE_EAL_VFIO),y)
DIRS-$(CONFIG_RTE_LIBRTE_FSLMC_BUS) += fslmc
endif
+DIRS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += ifpga
DIRS-$(CONFIG_RTE_LIBRTE_PCI_BUS) += pci
DIRS-$(CONFIG_RTE_LIBRTE_VDEV_BUS) += vdev
new file mode 100644
@@ -0,0 +1,32 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2018 Intel Corporation
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+#
+# library name
+#
+LIB = librte_bus_ifpga.a
+
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+CFLAGS += -O3
+CFLAGS += $(WERROR_FLAGS)
+LDLIBS += -lrte_eal
+LDLIBS += -lrte_rawdev
+LDLIBS += -lrte_kvargs
+
+# versioning export map
+EXPORT_MAP := rte_bus_ifpga_version.map
+
+# library version
+LIBABIVER := 1
+
+SRCS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += ifpga_bus.c
+SRCS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += ifpga_common.c
+
+#
+# Export include files
+#
+SYMLINK-$(CONFIG_RTE_LIBRTE_IFPGA_BUS)-include += rte_bus_ifpga.h
+
+include $(RTE_SDK)/mk/rte.lib.mk
new file mode 100644
@@ -0,0 +1,501 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2018 Intel Corporation
+ */
+
+#include <string.h>
+#include <inttypes.h>
+#include <stdint.h>
+#include <stdlib.h>
+#include <stdio.h>
+#include <sys/queue.h>
+#include <sys/mman.h>
+#include <sys/types.h>
+#include <unistd.h>
+#include <fcntl.h>
+
+#include <rte_errno.h>
+#include <rte_bus.h>
+#include <rte_per_lcore.h>
+#include <rte_memory.h>
+#include <rte_memzone.h>
+#include <rte_eal.h>
+#include <rte_common.h>
+
+#include <rte_devargs.h>
+#include <rte_kvargs.h>
+#include <rte_alarm.h>
+
+#include "rte_rawdev.h"
+#include "rte_rawdev_pmd.h"
+#include "rte_bus_ifpga.h"
+#include "ifpga_logs.h"
+#include "ifpga_common.h"
+
+int ifpga_bus_logtype;
+
+/* Forward declaration to access Intel FPGA bus
+ * on which iFPGA devices are connected
+ */
+static struct rte_bus rte_ifpga_bus;
+
+/** Double linked list of IFPGA device. */
+TAILQ_HEAD(ifpga_device_list, rte_ifpga_device);
+
+static struct ifpga_device_list ifpga_device_list =
+ TAILQ_HEAD_INITIALIZER(ifpga_device_list);
+static struct afu_driver_list afu_driver_list =
+ TAILQ_HEAD_INITIALIZER(afu_driver_list);
+
+
+/* register a ifpga bus based driver */
+void rte_ifpga_driver_register(struct rte_afu_driver *driver)
+{
+ RTE_VERIFY(driver);
+
+ TAILQ_INSERT_TAIL(&afu_driver_list, driver, next);
+}
+
+/* un-register a fpga bus based driver */
+void rte_ifpga_driver_unregister(struct rte_afu_driver *driver)
+{
+ TAILQ_REMOVE(&afu_driver_list, driver, next);
+}
+
+static struct rte_ifpga_device *
+ifpga_find_ifpga_dev(const struct rte_rawdev *rdev)
+{
+ struct rte_ifpga_device *ifpga_dev = NULL;
+
+ TAILQ_FOREACH(ifpga_dev, &ifpga_device_list, next) {
+ if (rdev &&
+ ifpga_dev->rdev &&
+ ifpga_dev->rdev == rdev)
+ return ifpga_dev;
+ }
+ return NULL;
+}
+
+static struct rte_afu_device *
+ifpga_find_afu_dev(const struct rte_ifpga_device *ifpga_dev,
+ const struct rte_afu_id *afu_id)
+{
+ struct rte_afu_device *afu_dev = NULL;
+
+ TAILQ_FOREACH(afu_dev, &ifpga_dev->afu_list, next) {
+ if (!ifpga_afu_id_cmp(&afu_dev->id, afu_id))
+ return afu_dev;
+ }
+ return NULL;
+}
+
+static const char * const valid_args[] = {
+#define IFPGA_ARG_NAME "ifpga"
+ IFPGA_ARG_NAME,
+#define IFPGA_ARG_PORT "port"
+ IFPGA_ARG_PORT,
+#define IFPGA_AFU_BTS "afu_bts"
+ IFPGA_AFU_BTS,
+ NULL
+};
+
+/*
+ * Scan the content of the FPGA bus, and the devices in the devices
+ * list
+ */
+static struct rte_afu_device *
+ifpga_scan_one(struct rte_devargs *devargs,
+ struct rte_ifpga_device *ifpga_dev)
+{
+ struct rte_kvargs *kvlist = NULL;
+ struct rte_rawdev *rawdev = NULL;
+ struct rte_afu_device *afu_dev = NULL;
+ struct rte_afu_pr_conf afu_pr_conf;
+ int ret = 0;
+ char *path = NULL;
+
+ memset(&afu_pr_conf, 0, sizeof(struct rte_afu_pr_conf));
+
+ kvlist = rte_kvargs_parse(devargs->args, valid_args);
+ if (!kvlist) {
+ IFPGA_BUS_ERR("error when parsing param");
+ goto end;
+ }
+
+ if (rte_kvargs_count(kvlist, IFPGA_ARG_PORT) == 1) {
+ if (rte_kvargs_process(kvlist, IFPGA_ARG_PORT,
+ &rte_ifpga_get_integer32_arg, &afu_pr_conf.afu_id.port) < 0) {
+ IFPGA_BUS_ERR("error to parse %s",
+ IFPGA_ARG_PORT);
+ goto end;
+ }
+ } else {
+ IFPGA_BUS_ERR("arg %s is mandatory for ifpga bus",
+ IFPGA_ARG_PORT);
+ goto end;
+ }
+
+ if (rte_kvargs_count(kvlist, IFPGA_AFU_BTS) == 1) {
+ if (rte_kvargs_process(kvlist, IFPGA_AFU_BTS,
+ &rte_ifpga_get_string_arg, &path) < 0) {
+ IFPGA_BUS_ERR("Failed to parse %s",
+ IFPGA_AFU_BTS);
+ goto end;
+ }
+ } else {
+ IFPGA_BUS_ERR("arg %s is mandatory for ifpga bus",
+ IFPGA_AFU_BTS);
+ goto end;
+ }
+
+ afu_pr_conf.afu_id.uuid.uuid_low = 0;
+ afu_pr_conf.afu_id.uuid.uuid_high = 0;
+ afu_pr_conf.pr_enable = path?1:0;
+
+ rawdev = ifpga_dev->rdev;
+ if (ifpga_find_afu_dev(ifpga_dev, &afu_pr_conf.afu_id))
+ goto end;
+
+ afu_dev = calloc(1, sizeof(*afu_dev));
+ if (!afu_dev)
+ goto end;
+
+ afu_dev->device.devargs = devargs;
+ afu_dev->device.numa_node = SOCKET_ID_ANY;
+ afu_dev->device.name = devargs->name;
+ afu_dev->rawdev = rawdev;
+ afu_dev->id.uuid.uuid_low = 0;
+ afu_dev->id.uuid.uuid_high = 0;
+ afu_dev->id.port = afu_pr_conf.afu_id.port;
+ afu_dev->ifpga_dev = ifpga_dev;
+
+ if (rawdev->dev_ops && rawdev->dev_ops->dev_info_get)
+ rawdev->dev_ops->dev_info_get(rawdev, afu_dev);
+
+ if (rawdev->dev_ops &&
+ rawdev->dev_ops->dev_start &&
+ rawdev->dev_ops->dev_start(rawdev))
+ goto free_dev;
+
+ strncpy(afu_pr_conf.bs_path, path, sizeof(afu_pr_conf.bs_path));
+ if (rawdev->dev_ops->firmware_load &&
+ rawdev->dev_ops->firmware_load(rawdev,
+ &afu_pr_conf)){
+ IFPGA_BUS_ERR("firmware load error %d\n", ret);
+ goto free_dev;
+ }
+ afu_dev->id.uuid.uuid_low = afu_pr_conf.afu_id.uuid.uuid_low;
+ afu_dev->id.uuid.uuid_high = afu_pr_conf.afu_id.uuid.uuid_high;
+
+ return afu_dev;
+
+free_dev:
+ free(afu_dev);
+end:
+ if (kvlist)
+ rte_kvargs_free(kvlist);
+ if (path)
+ free(path);
+
+ return NULL;
+}
+
+/*
+ * Scan the content of the FPGA bus, and the devices in the devices
+ * list
+ */
+static int
+ifpga_scan(void)
+{
+ struct rte_ifpga_device *ifpga_dev;
+ struct rte_devargs *devargs;
+ struct rte_kvargs *kvlist = NULL;
+ struct rte_rawdev *rawdev = NULL;
+ char *name = NULL;
+ char name1[RTE_RAWDEV_NAME_MAX_LEN];
+ struct rte_afu_device *afu_dev = NULL;
+
+ /* for FPGA devices we scan the devargs_list populated via cmdline */
+ RTE_EAL_DEVARGS_FOREACH("ifpga", devargs) {
+ if (devargs->bus != &rte_ifpga_bus)
+ continue;
+
+ kvlist = rte_kvargs_parse(devargs->args, valid_args);
+ if (!kvlist) {
+ IFPGA_BUS_ERR("error when parsing param");
+ goto end;
+ }
+
+ if (rte_kvargs_count(kvlist, IFPGA_ARG_NAME) == 1) {
+ if (rte_kvargs_process(kvlist, IFPGA_ARG_NAME,
+ &rte_ifpga_get_string_arg, &name) < 0) {
+ IFPGA_BUS_ERR("error to parse %s",
+ IFPGA_ARG_NAME);
+ goto end;
+ }
+ } else {
+ IFPGA_BUS_ERR("arg %s is mandatory for ifpga bus",
+ IFPGA_ARG_NAME);
+ goto end;
+ }
+
+ memset(name1, 0, sizeof(name1));
+ snprintf(name1, RTE_RAWDEV_NAME_MAX_LEN, "IFPGA:%s", name);
+
+ rawdev = rte_rawdev_pmd_get_named_dev(name1);
+ if (!rawdev)
+ goto end;
+
+ if (ifpga_find_ifpga_dev(rawdev))
+ continue;
+
+ ifpga_dev = calloc(1, sizeof(*ifpga_dev));
+ if (!ifpga_dev)
+ goto end;
+
+ ifpga_dev->rdev = rawdev;
+ TAILQ_INIT(&ifpga_dev->afu_list);
+
+ TAILQ_INSERT_TAIL(&ifpga_device_list, ifpga_dev, next);
+ afu_dev = ifpga_scan_one(devargs, ifpga_dev);
+ if (afu_dev != NULL)
+ TAILQ_INSERT_TAIL(&ifpga_dev->afu_list, afu_dev, next);
+ }
+
+end:
+ if (kvlist)
+ rte_kvargs_free(kvlist);
+ if (name)
+ free(name);
+
+ return 0;
+}
+
+/*
+ * Match the AFU Driver and AFU Device using the ID Table
+ */
+static int
+rte_afu_match(const struct rte_afu_driver *afu_drv,
+ const struct rte_afu_device *afu_dev)
+{
+ const struct rte_afu_uuid *id_table;
+
+ for (id_table = afu_drv->id_table;
+ ((id_table->uuid_low != 0) && (id_table->uuid_high != 0));
+ id_table++) {
+ /* check if device's identifiers match the driver's ones */
+ if ((id_table->uuid_low != afu_dev->id.uuid.uuid_low) ||
+ id_table->uuid_high !=
+ afu_dev->id.uuid.uuid_high)
+ continue;
+
+ return 1;
+ }
+
+ return 0;
+}
+
+static int
+ifpga_probe_one_driver(struct rte_afu_driver *drv,
+ struct rte_afu_device *afu_dev)
+{
+ int ret;
+
+ if (!rte_afu_match(drv, afu_dev))
+ /* Match of device and driver failed */
+ return 1;
+
+ /* reference driver structure */
+ afu_dev->driver = drv;
+ afu_dev->device.driver = &drv->driver;
+
+ /* call the driver probe() function */
+ ret = drv->probe(afu_dev);
+ if (ret) {
+ afu_dev->driver = NULL;
+ afu_dev->device.driver = NULL;
+ }
+
+ return ret;
+}
+
+static int
+ifpga_probe_all_drivers(struct rte_afu_device *afu_dev)
+{
+ struct rte_afu_driver *drv = NULL;
+ int ret = 0;
+
+ if (afu_dev == NULL)
+ return -1;
+
+ /* Check if a driver is already loaded */
+ if (afu_dev->driver != NULL)
+ return 0;
+
+ TAILQ_FOREACH(drv, &afu_driver_list, next) {
+ if (ifpga_probe_one_driver(drv, afu_dev)) {
+ ret = -1;
+ break;
+ }
+ }
+ return ret;
+}
+
+/*
+ * Scan the content of the Intel FPGA bus, and call the probe() function for
+ * all registered drivers that have a matching entry in its id_table
+ * for discovered devices.
+ */
+static int
+ifpga_probe(void)
+{
+ struct rte_ifpga_device *ifpga_dev;
+ struct rte_afu_device *afu_dev = NULL;
+ int ret = 0;
+
+ TAILQ_FOREACH(ifpga_dev, &ifpga_device_list, next) {
+ TAILQ_FOREACH(afu_dev, &ifpga_dev->afu_list, next) {
+
+ if (afu_dev->device.driver)
+ continue;
+
+ ret = ifpga_probe_all_drivers(afu_dev);
+ if (ret < 0)
+ IFPGA_BUS_ERR("failed to initialize %s device\n",
+ rte_ifpga_device_name(afu_dev));
+ }
+ }
+
+ return ret;
+}
+
+static int
+ifpga_plug(struct rte_device *dev)
+{
+ return ifpga_probe_all_drivers(RTE_DEV_TO_AFU(dev));
+}
+
+static int
+ifpga_remove_driver(struct rte_afu_device *afu_dev)
+{
+ const char *name;
+ const struct rte_afu_driver *driver;
+
+ name = rte_ifpga_device_name(afu_dev);
+ if (!afu_dev->device.driver) {
+ IFPGA_BUS_DEBUG("no driver attach to device %s\n", name);
+ return 1;
+ }
+
+ driver = RTE_DRV_TO_AFU_CONST(afu_dev->device.driver);
+ return driver->remove(afu_dev);
+}
+
+static int
+ifpga_unplug(struct rte_device *dev)
+{
+ struct rte_ifpga_device *ifpga_dev = NULL;
+ struct rte_afu_device *afu_dev = NULL;
+ struct rte_devargs *devargs = NULL;
+ int ret;
+
+ if (dev == NULL)
+ return -EINVAL;
+
+ afu_dev = RTE_DEV_TO_AFU(dev);
+ if (!dev)
+ return -ENOENT;
+
+ ifpga_dev = afu_dev->ifpga_dev;
+ devargs = dev->devargs;
+
+ ret = ifpga_remove_driver(afu_dev);
+ if (ret)
+ return ret;
+
+ TAILQ_REMOVE(&ifpga_dev->afu_list, afu_dev, next);
+
+ rte_devargs_remove(devargs->bus->name, devargs->name);
+ free(afu_dev);
+ return 0;
+
+}
+
+static struct rte_device *
+ifpga_find_device(const struct rte_device *start,
+ rte_dev_cmp_t cmp, const void *data)
+{
+ struct rte_ifpga_device *ifpga_dev;
+ struct rte_afu_device *afu_dev;
+
+ TAILQ_FOREACH(ifpga_dev, &ifpga_device_list, next) {
+ TAILQ_FOREACH(afu_dev, &ifpga_dev->afu_list, next) {
+ if (start && &afu_dev->device == start) {
+ start = NULL;
+ continue;
+ }
+ if (cmp(&afu_dev->device, data) == 0)
+ return &afu_dev->device;
+ }
+ }
+ return NULL;
+}
+static int
+ifpga_parse(const char *name, void *addr)
+{
+ int *out = addr;
+ struct rte_rawdev *rawdev = NULL;
+ char rawdev_name[RTE_RAWDEV_NAME_MAX_LEN];
+ char *c1 = NULL, *c2 = NULL;
+ int port = IFPGA_BUS_DEV_PORT_MAX;
+ char str_port[8];
+ int str_port_len = 0;
+ int ret;
+
+ memset(str_port, 0, 8);
+ c1 = strchr(name, '|');
+ if (c1 != NULL) {
+ str_port_len = c1-name;
+ c2 = c1+1;
+ }
+
+ if (str_port_len < 8 &&
+ str_port_len > 0) {
+ memcpy(str_port, name, str_port_len);
+ ret = sscanf(str_port, "%d", &port);
+ if (ret == -1)
+ return 0;
+ }
+
+ memset(rawdev_name, 0, sizeof(rawdev_name));
+ snprintf(rawdev_name, RTE_RAWDEV_NAME_MAX_LEN, "IFPGA:%s", c2);
+ rawdev = rte_rawdev_pmd_get_named_dev(rawdev_name);
+
+ if ((port < IFPGA_BUS_DEV_PORT_MAX) &&
+ rawdev &&
+ (addr != NULL))
+ *out = port;
+
+ if ((port < IFPGA_BUS_DEV_PORT_MAX) &&
+ rawdev)
+ return 0;
+ else
+ return 1;
+}
+
+static struct rte_bus rte_ifpga_bus = {
+ .scan = ifpga_scan,
+ .probe = ifpga_probe,
+ .find_device = ifpga_find_device,
+ .plug = ifpga_plug,
+ .unplug = ifpga_unplug,
+ .parse = ifpga_parse,
+};
+
+RTE_REGISTER_BUS(IFPGA_BUS_NAME, rte_ifpga_bus);
+
+RTE_INIT(ifpga_init_log)
+{
+ ifpga_bus_logtype = rte_log_register("bus.ifpga");
+ if (ifpga_bus_logtype >= 0)
+ rte_log_set_level(ifpga_bus_logtype, RTE_LOG_NOTICE);
+}
new file mode 100644
@@ -0,0 +1,88 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2018 Intel Corporation
+ */
+
+#include <string.h>
+#include <inttypes.h>
+#include <stdint.h>
+#include <stdlib.h>
+#include <stdio.h>
+#include <sys/queue.h>
+#include <sys/mman.h>
+#include <sys/types.h>
+#include <unistd.h>
+#include <fcntl.h>
+
+#include <rte_errno.h>
+#include <rte_bus.h>
+#include <rte_per_lcore.h>
+#include <rte_memory.h>
+#include <rte_memzone.h>
+#include <rte_eal.h>
+#include <rte_common.h>
+
+#include <rte_devargs.h>
+#include <rte_kvargs.h>
+#include <rte_alarm.h>
+
+#include "rte_bus_ifpga.h"
+#include "ifpga_logs.h"
+#include "ifpga_common.h"
+
+int rte_ifpga_get_string_arg(const char *key __rte_unused,
+ const char *value, void *extra_args)
+{
+ if (!value || !extra_args)
+ return -EINVAL;
+
+ *(char **)extra_args = strdup(value);
+
+ if (!*(char **)extra_args)
+ return -ENOMEM;
+
+ return 0;
+}
+int rte_ifpga_get_integer32_arg(const char *key __rte_unused,
+ const char *value, void *extra_args)
+{
+ if (!value || !extra_args)
+ return -EINVAL;
+
+ *(int *)extra_args = strtoull(value, NULL, 0);
+
+ return 0;
+}
+int ifpga_get_integer64_arg(const char *key __rte_unused,
+ const char *value, void *extra_args)
+{
+ if (!value || !extra_args)
+ return -EINVAL;
+
+ *(uint64_t *)extra_args = strtoull(value, NULL, 0);
+
+ return 0;
+}
+int ifpga_get_unsigned_long(const char *str, int base)
+{
+ unsigned long num;
+ char *end = NULL;
+
+ errno = 0;
+
+ num = strtoul(str, &end, base);
+ if ((str[0] == '\0') || (end == NULL) || (*end != '\0') || (errno != 0))
+ return -1;
+
+ return num;
+}
+
+int ifpga_afu_id_cmp(const struct rte_afu_id *afu_id0,
+ const struct rte_afu_id *afu_id1)
+{
+ if ((afu_id0->uuid.uuid_low == afu_id1->uuid.uuid_low) &&
+ (afu_id0->uuid.uuid_high == afu_id1->uuid.uuid_high) &&
+ (afu_id0->port == afu_id1->port)) {
+ return 0;
+ } else
+ return 1;
+}
new file mode 100644
@@ -0,0 +1,18 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2018 Intel Corporation
+ */
+
+#ifndef _IFPGA_COMMON_H_
+#define _IFPGA_COMMON_H_
+
+int rte_ifpga_get_string_arg(const char *key __rte_unused,
+ const char *value, void *extra_args);
+int rte_ifpga_get_integer32_arg(const char *key __rte_unused,
+ const char *value, void *extra_args);
+int ifpga_get_integer64_arg(const char *key __rte_unused,
+ const char *value, void *extra_args);
+int ifpga_get_unsigned_long(const char *str, int base);
+int ifpga_afu_id_cmp(const struct rte_afu_id *afu_id0,
+ const struct rte_afu_id *afu_id1);
+
+#endif /* _IFPGA_COMMON_H_ */
new file mode 100644
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2018 Intel Corporation
+ */
+
+#ifndef _IFPGA_LOGS_H_
+#define _IFPGA_LOGS_H_
+
+#include <rte_log.h>
+
+extern int ifpga_bus_logtype;
+
+#define IFPGA_LOG(level, fmt, args...) \
+ rte_log(RTE_LOG_ ## level, ifpga_bus_logtype, "%s(): " fmt "\n", \
+ __func__, ##args)
+
+#define IFPGA_BUS_LOG(level, fmt, args...) \
+ rte_log(RTE_LOG_ ## level, ifpga_bus_logtype, "%s(): " fmt "\n", \
+ __func__, ##args)
+
+#define IFPGA_BUS_FUNC_TRACE() IFPGA_BUS_LOG(DEBUG, ">>")
+
+#define IFPGA_BUS_DEBUG(fmt, args...) \
+ IFPGA_BUS_LOG(DEBUG, fmt, ## args)
+#define IFPGA_BUS_INFO(fmt, args...) \
+ IFPGA_BUS_LOG(INFO, fmt, ## args)
+#define IFPGA_BUS_ERR(fmt, args...) \
+ IFPGA_BUS_LOG(ERR, fmt, ## args)
+#define IFPGA_BUS_WARN(fmt, args...) \
+ IFPGA_BUS_LOG(WARNING, fmt, ## args)
+
+#endif /* _IFPGA_BUS_LOGS_H_ */
new file mode 100644
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2010-2018 Intel Corporation
+
+deps += ['pci', 'kvargs', 'rawdev']
+install_headers('rte_bus_ifpga.h')
+sources = files('ifpga_common.c', 'ifpga_bus.c')
+
+allow_experimental_apis = true
new file mode 100644
@@ -0,0 +1,160 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2018 Intel Corporation
+ */
+
+#ifndef _RTE_BUS_IFPGA_H_
+#define _RTE_BUS_IFPGA_H_
+
+/**
+ * @file
+ *
+ * RTE Intel FPGA Bus Interface
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_bus.h>
+#include <rte_pci.h>
+
+/** Name of Intel FPGA Bus */
+#define IFPGA_BUS_NAME ifpga
+
+/* Forward declarations */
+struct rte_afu_device;
+
+/** List of Intel AFU devices */
+TAILQ_HEAD(afu_device_list, rte_afu_device);
+/** Double linked list of AFU device drivers. */
+TAILQ_HEAD(afu_driver_list, rte_afu_driver);
+
+#define IFPGA_BUS_BITSTREAM_PATH_MAX_LEN 256
+
+struct rte_afu_uuid {
+ uint64_t uuid_low;
+ uint64_t uuid_high;
+} __attribute__ ((packed));
+
+#define IFPGA_BUS_DEV_PORT_MAX 4
+
+/**
+ * A structure describing an ID for a AFU driver. Each driver provides a
+ * table of these IDs for each device that it supports.
+ */
+struct rte_afu_id {
+ struct rte_afu_uuid uuid;
+ int port; /**< port number */
+} __attribute__ ((packed));
+
+/**
+ * A structure PR (Partial Reconfiguration) configuration AFU driver.
+ */
+
+struct rte_afu_pr_conf {
+ struct rte_afu_id afu_id;
+ int pr_enable;
+ char bs_path[IFPGA_BUS_BITSTREAM_PATH_MAX_LEN];
+};
+
+#define AFU_PRI_STR_SIZE (PCI_PRI_STR_SIZE + 8)
+
+/**
+ * A structure describing a fpga device.
+ */
+struct rte_ifpga_device {
+ TAILQ_ENTRY(rte_ifpga_device) next; /**< Next in device list. */
+ struct rte_rawdev *rdev;
+ struct afu_device_list afu_list; /**< List of AFU devices */
+};
+
+/**
+ * A structure describing a AFU device.
+ */
+struct rte_afu_device {
+ TAILQ_ENTRY(rte_afu_device) next; /**< Next in device list. */
+ struct rte_device device; /**< Inherit core device */
+ struct rte_rawdev *rawdev; /**< Point Rawdev */
+ struct rte_ifpga_device *ifpga_dev; /**< Point ifpga device */
+ struct rte_afu_id id; /**< AFU id within FPGA. */
+ uint32_t num_region; /**< number of regions found */
+ struct rte_mem_resource mem_resource[PCI_MAX_RESOURCE];
+ /**< AFU Memory Resource */
+ struct rte_intr_handle intr_handle; /**< Interrupt handle */
+ struct rte_afu_driver *driver; /**< Associated driver */
+ char path[IFPGA_BUS_BITSTREAM_PATH_MAX_LEN];
+} __attribute__ ((packed));
+
+/**
+ * @internal
+ * Helper macro for drivers that need to convert to struct rte_afu_device.
+ */
+#define RTE_DEV_TO_AFU(ptr) \
+ container_of(ptr, struct rte_afu_device, device)
+
+#define RTE_DRV_TO_AFU_CONST(ptr) \
+ container_of(ptr, const struct rte_afu_driver, driver)
+
+/**
+ * Initialisation function for the driver called during FPGA BUS probing.
+ */
+typedef int (afu_probe_t)(struct rte_afu_device *);
+
+/**
+ * Uninitialisation function for the driver called during hotplugging.
+ */
+typedef int (afu_remove_t)(struct rte_afu_device *);
+
+/**
+ * A structure describing a AFU device.
+ */
+struct rte_afu_driver {
+ TAILQ_ENTRY(rte_afu_driver) next; /**< Next afu driver. */
+ struct rte_driver driver; /**< Inherit core driver. */
+ afu_probe_t *probe; /**< Device Probe function. */
+ afu_remove_t *remove; /**< Device Remove function. */
+ const struct rte_afu_uuid *id_table; /**< AFU uuid within FPGA. */
+ uint32_t drv_flags; /**< Flags contolling handling of device. */
+};
+
+static inline const char *
+rte_ifpga_device_name(const struct rte_afu_device *afu)
+{
+ if (afu && afu->device.name)
+ return afu->device.name;
+ return NULL;
+}
+
+/**
+ * Register a ifpga afu device driver.
+ *
+ * @param driver
+ * A pointer to a rte_afu_driver structure describing the driver
+ * to be registered.
+ */
+void rte_ifpga_driver_register(struct rte_afu_driver *driver);
+
+/**
+ * Unregister a ifpga afu device driver.
+ *
+ * @param driver
+ * A pointer to a rte_afu_driver structure describing the driver
+ * to be unregistered.
+ */
+void rte_ifpga_driver_unregister(struct rte_afu_driver *driver);
+
+#define RTE_PMD_REGISTER_AFU(nm, afudrv)\
+RTE_INIT(afudrvinitfn_ ##afudrv);\
+static const char *afudrvinit_ ## nm ## _alias;\
+static void afudrvinitfn_ ##afudrv(void)\
+{\
+ (afudrv).driver.name = RTE_STR(nm);\
+ (afudrv).driver.alias = afudrvinit_ ## nm ## _alias;\
+ rte_ifpga_driver_register(&afudrv);\
+} \
+RTE_PMD_EXPORT_NAME(nm, __COUNTER__)
+
+#define RTE_PMD_REGISTER_AFU_ALIAS(nm, alias)\
+static const char *afudrvinit_ ## nm ## _alias = RTE_STR(alias)
+
+#endif /* _RTE_BUS_IFPGA_H_ */
new file mode 100644
@@ -0,0 +1,10 @@
+DPDK_18.05 {
+ global:
+
+ rte_ifpga_get_integer32_arg;
+ rte_ifpga_get_string_arg;
+ rte_ifpga_driver_register;
+ rte_ifpga_driver_unregister;
+
+ local: *;
+};
@@ -1,7 +1,7 @@
# SPDX-License-Identifier: BSD-3-Clause
# Copyright(c) 2017 Intel Corporation
-drivers = ['dpaa', 'fslmc', 'pci', 'vdev']
+drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'vdev']
std_deps = ['eal']
config_flag_fmt = 'RTE_LIBRTE_@0@_BUS'
driver_name_fmt = 'rte_bus_@0@'
@@ -255,6 +255,7 @@ ifeq ($(CONFIG_RTE_EAL_VFIO)$(CONFIG_RTE_LIBRTE_FSLMC_BUS),yy)
_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_CMDIF_RAWDEV) += -lrte_pmd_dpaa2_cmdif
_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV) += -lrte_pmd_dpaa2_qdma
endif # CONFIG_RTE_LIBRTE_FSLMC_BUS
+_LDLIBS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += -lrte_bus_ifpga
endif # CONFIG_RTE_LIBRTE_RAWDEV