[v5,1/1] bus/pci: optimise scanning with whitelist/blacklist
Checks
Commit Message
rte_bus_scan API scans all the available PCI devices irrespective of white
or black listing parameters then further devices are probed based on white
or black listing parameters. So unnecessary CPU cycles are wasted during
rte_pci_scan.
For Octeontx2 platform with core frequency 2.4 Ghz, rte_bus_scan consumes
around 26ms to scan around 90 PCI devices but all may not be used by the
application. So for the application which uses 2 NICs, rte_bus_scan
consumes few microseconds and rest time is saved with this patch.
Patch restricts devices to be scanned as per below mentioned conditions:
- All devices will be scanned if no parameters are passed.
- Only white listed devices will be scanned if white list is available.
- All devices, except black listed, will be scanned if black list is
available.
Signed-off-by: Sunil Kumar Kori <skori@marvell.com>
---
v5:
- revert devargs->bus->parse changes.
v4:
- Review comments incorporated (Gaeten and David).
- Rebased on top of tree.
v3:
- Remove __rte_experimental from private function.
- Remove entry from map file too.
v2:
- Added function to validate ignorance of device based on PCI address.
- Marked device validation function as experimental.
drivers/bus/pci/bsd/pci.c | 12 +++++++++++-
drivers/bus/pci/linux/pci.c | 3 +++
drivers/bus/pci/pci_common.c | 29 +++++++++--------------------
drivers/bus/pci/private.h | 11 +++++++++++
4 files changed, 34 insertions(+), 21 deletions(-)
Comments
On 02/05/20 13:12 +0530, Sunil Kumar Kori wrote:
> rte_bus_scan API scans all the available PCI devices irrespective of white
> or black listing parameters then further devices are probed based on white
> or black listing parameters. So unnecessary CPU cycles are wasted during
> rte_pci_scan.
>
> For Octeontx2 platform with core frequency 2.4 Ghz, rte_bus_scan consumes
> around 26ms to scan around 90 PCI devices but all may not be used by the
> application. So for the application which uses 2 NICs, rte_bus_scan
> consumes few microseconds and rest time is saved with this patch.
>
> Patch restricts devices to be scanned as per below mentioned conditions:
> - All devices will be scanned if no parameters are passed.
> - Only white listed devices will be scanned if white list is available.
> - All devices, except black listed, will be scanned if black list is
> available.
>
> Signed-off-by: Sunil Kumar Kori <skori@marvell.com>
LGTM,
Acked-by: Gaetan Rivet <grive@u256.net>
On Sat, May 2, 2020 at 9:42 AM Sunil Kumar Kori <skori@marvell.com> wrote:
> diff --git a/drivers/bus/pci/private.h b/drivers/bus/pci/private.h
> index a205d4d9f..3a9da7322 100644
> --- a/drivers/bus/pci/private.h
> +++ b/drivers/bus/pci/private.h
> @@ -42,6 +42,17 @@ int rte_pci_scan(void);
> void
> pci_name_set(struct rte_pci_device *dev);
>
> +/**
> + * Validate whether a device with given pci address should be ignored or not.
> + *
> + * @param pci_addr
> + * PCI address of device to be validated
> + * @return
> + * 1: if device is to be ignored,
> + * 0: if device is to be scanned,
true/false
> + */
> +bool rte_pci_ignore_device(const struct rte_pci_addr *pci_addr);
Gaetan, private API must not be prefixed with rte_, this is for public
APIs only.
I noticed inconsistencies in the pci bus some time ago, I will fix
this patch for now and send some followup patches I had in store.
> +
> /**
> * Add a PCI device to the PCI Bus (append to PCI Device list). This function
> * also updates the bus references of the PCI Device (and the generic device
Sunil, no need to send a new version, I will fix this when applying.
Thanks.
>-----Original Message-----
>From: David Marchand <david.marchand@redhat.com>
>Sent: Monday, May 4, 2020 7:47 PM
>To: Sunil Kumar Kori <skori@marvell.com>; Gaetan Rivet <grive@u256.net>
>Cc: Stephen Hemminger <stephen@networkplumber.org>; Jerin Jacob
>Kollanukkaran <jerinj@marvell.com>; dev <dev@dpdk.org>
>Subject: [EXT] Re: [PATCH v5 1/1] bus/pci: optimise scanning with
>whitelist/blacklist
>
>External Email
>
>----------------------------------------------------------------------
>On Sat, May 2, 2020 at 9:42 AM Sunil Kumar Kori <skori@marvell.com> wrote:
>> diff --git a/drivers/bus/pci/private.h b/drivers/bus/pci/private.h
>> index a205d4d9f..3a9da7322 100644
>> --- a/drivers/bus/pci/private.h
>> +++ b/drivers/bus/pci/private.h
>> @@ -42,6 +42,17 @@ int rte_pci_scan(void); void pci_name_set(struct
>> rte_pci_device *dev);
>>
>> +/**
>> + * Validate whether a device with given pci address should be ignored or
>not.
>> + *
>> + * @param pci_addr
>> + * PCI address of device to be validated
>> + * @return
>> + * 1: if device is to be ignored,
>> + * 0: if device is to be scanned,
>
>true/false
>
>
>> + */
>> +bool rte_pci_ignore_device(const struct rte_pci_addr *pci_addr);
>
>Gaetan, private API must not be prefixed with rte_, this is for public APIs only.
>I noticed inconsistencies in the pci bus some time ago, I will fix this patch for
>now and send some followup patches I had in store.
>
>
>
>> +
>> /**
>> * Add a PCI device to the PCI Bus (append to PCI Device list). This function
>> * also updates the bus references of the PCI Device (and the generic
>> device
>
>Sunil, no need to send a new version, I will fix this when applying.
>
>
Ack.
>Thanks.
>
>--
>David Marchand
On Mon, May 4, 2020 at 4:17 PM David Marchand <david.marchand@redhat.com> wrote:
>
> On Sat, May 2, 2020 at 9:42 AM Sunil Kumar Kori <skori@marvell.com> wrote:
> I noticed inconsistencies in the pci bus some time ago, I will fix
> this patch for now and send some followup patches I had in store.
I only sent the minimum, there are other changes like hiding internal
structures that could go later.
https://patchwork.dpdk.org/project/dpdk/list/?series=9874
On Sat, May 2, 2020 at 9:42 AM Sunil Kumar Kori <skori@marvell.com> wrote:
>
> rte_bus_scan API scans all the available PCI devices irrespective of white
> or black listing parameters then further devices are probed based on white
> or black listing parameters. So unnecessary CPU cycles are wasted during
> rte_pci_scan.
>
> For Octeontx2 platform with core frequency 2.4 Ghz, rte_bus_scan consumes
> around 26ms to scan around 90 PCI devices but all may not be used by the
> application. So for the application which uses 2 NICs, rte_bus_scan
> consumes few microseconds and rest time is saved with this patch.
>
> Patch restricts devices to be scanned as per below mentioned conditions:
> - All devices will be scanned if no parameters are passed.
> - Only white listed devices will be scanned if white list is available.
> - All devices, except black listed, will be scanned if black list is
> available.
>
> Signed-off-by: Sunil Kumar Kori <skori@marvell.com>
Acked-by: Gaetan Rivet <grive@u256.net>
> diff --git a/drivers/bus/pci/private.h b/drivers/bus/pci/private.h
> index a205d4d9f..3a9da7322 100644
> --- a/drivers/bus/pci/private.h
> +++ b/drivers/bus/pci/private.h
> @@ -42,6 +42,17 @@ int rte_pci_scan(void);
> void
> pci_name_set(struct rte_pci_device *dev);
>
> +/**
> + * Validate whether a device with given pci address should be ignored or not.
PCI address
> + *
> + * @param pci_addr
> + * PCI address of device to be validated
> + * @return
> + * 1: if device is to be ignored,
> + * 0: if device is to be scanned,
On Sat, May 2, 2020 at 9:42 AM Sunil Kumar Kori <skori@marvell.com> wrote:
> diff --git a/drivers/bus/pci/private.h b/drivers/bus/pci/private.h
> index a205d4d9f..3a9da7322 100644
> --- a/drivers/bus/pci/private.h
> +++ b/drivers/bus/pci/private.h
> @@ -42,6 +42,17 @@ int rte_pci_scan(void);
> void
> pci_name_set(struct rte_pci_device *dev);
>
> +/**
> + * Validate whether a device with given pci address should be ignored or not.
> + *
> + * @param pci_addr
> + * PCI address of device to be validated
> + * @return
> + * 1: if device is to be ignored,
> + * 0: if device is to be scanned,
true/false
> + */
> +bool rte_pci_ignore_device(const struct rte_pci_addr *pci_addr);
> +
> /**
> * Add a PCI device to the PCI Bus (append to PCI Device list). This function
> * also updates the bus references of the PCI Device (and the generic device
Applied, thanks.
@@ -338,6 +338,7 @@ rte_pci_scan(void)
.match_buf_len = sizeof(matches),
.matches = &matches[0],
};
+ struct rte_pci_addr pci_addr;
/* for debug purposes, PCI can be disabled */
if (!rte_eal_has_pci())
@@ -357,9 +358,18 @@ rte_pci_scan(void)
goto error;
}
- for (i = 0; i < conf_io.num_matches; i++)
+ for (i = 0; i < conf_io.num_matches; i++) {
+ pci_addr.domain = matches[i].pc_sel.pc_domain;
+ pci_addr.bus = matches[i].pc_sel.pc_bus;
+ pci_addr.devid = matches[i].pc_sel.pc_dev;
+ pci_addr.function = matches[i].pc_sel.pc_func;
+
+ if (rte_pci_ignore_device(&pci_addr))
+ continue;
+
if (pci_scan_one(fd, &matches[i]) < 0)
goto error;
+ }
dev_count += conf_io.num_matches;
} while(conf_io.status == PCI_GETCONF_MORE_DEVS);
@@ -487,6 +487,9 @@ rte_pci_scan(void)
if (parse_pci_addr_format(e->d_name, sizeof(e->d_name), &addr) != 0)
continue;
+ if (rte_pci_ignore_device(&addr))
+ continue;
+
snprintf(dirname, sizeof(dirname), "%s/%s",
rte_pci_get_sysfs_path(), e->d_name);
@@ -42,14 +42,15 @@ const char *rte_pci_get_sysfs_path(void)
return path;
}
-static struct rte_devargs *pci_devargs_lookup(struct rte_pci_device *dev)
+static struct rte_devargs *
+pci_devargs_lookup(const struct rte_pci_addr *pci_addr)
{
struct rte_devargs *devargs;
struct rte_pci_addr addr;
RTE_EAL_DEVARGS_FOREACH("pci", devargs) {
devargs->bus->parse(devargs->name, &addr);
- if (!rte_pci_addr_cmp(&dev->addr, &addr))
+ if (!rte_pci_addr_cmp(pci_addr, &addr))
return devargs;
}
return NULL;
@@ -63,7 +64,7 @@ pci_name_set(struct rte_pci_device *dev)
/* Each device has its internal, canonical name set. */
rte_pci_device_name(&dev->addr,
dev->name, sizeof(dev->name));
- devargs = pci_devargs_lookup(dev);
+ devargs = pci_devargs_lookup(&dev->addr);
dev->device.devargs = devargs;
/* In blacklist mode, if the device is not blacklisted, no
* rte_devargs exists for it.
@@ -293,23 +294,12 @@ rte_pci_probe(void)
{
struct rte_pci_device *dev = NULL;
size_t probed = 0, failed = 0;
- struct rte_devargs *devargs;
- int probe_all = 0;
int ret = 0;
- if (rte_pci_bus.bus.conf.scan_mode != RTE_BUS_SCAN_WHITELIST)
- probe_all = 1;
-
FOREACH_DEVICE_ON_PCIBUS(dev) {
probed++;
- devargs = dev->device.devargs;
- /* probe all or only whitelisted devices */
- if (probe_all)
- ret = pci_probe_all_drivers(dev);
- else if (devargs != NULL &&
- devargs->policy == RTE_DEV_WHITELISTED)
- ret = pci_probe_all_drivers(dev);
+ ret = pci_probe_all_drivers(dev);
if (ret < 0) {
if (ret != -EEXIST) {
RTE_LOG(ERR, EAL, "Requested device "
@@ -589,10 +579,10 @@ pci_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
return -1;
}
-static bool
-pci_ignore_device(const struct rte_pci_device *dev)
+bool
+rte_pci_ignore_device(const struct rte_pci_addr *pci_addr)
{
- struct rte_devargs *devargs = dev->device.devargs;
+ struct rte_devargs *devargs = pci_devargs_lookup(pci_addr);
switch (rte_pci_bus.bus.conf.scan_mode) {
case RTE_BUS_SCAN_WHITELIST:
@@ -627,8 +617,7 @@ rte_pci_get_iommu_class(void)
if (iommu_no_va == -1)
iommu_no_va = pci_device_iommu_support_va(dev)
? 0 : 1;
- if (pci_ignore_device(dev))
- continue;
+
if (dev->kdrv == RTE_KDRV_UNKNOWN ||
dev->kdrv == RTE_KDRV_NONE)
continue;
@@ -42,6 +42,17 @@ int rte_pci_scan(void);
void
pci_name_set(struct rte_pci_device *dev);
+/**
+ * Validate whether a device with given pci address should be ignored or not.
+ *
+ * @param pci_addr
+ * PCI address of device to be validated
+ * @return
+ * 1: if device is to be ignored,
+ * 0: if device is to be scanned,
+ */
+bool rte_pci_ignore_device(const struct rte_pci_addr *pci_addr);
+
/**
* Add a PCI device to the PCI Bus (append to PCI Device list). This function
* also updates the bus references of the PCI Device (and the generic device