[dpdk-dev,v5,2/2] app/testpmd: use uevent to monitor hot removal
Checks
Commit Message
use testpmd for example, to show app how to request and use
uevent monitoring to handle the hot removal event and the
hot insertion event.
Signed-off-by: Jeff Guo <jia.guo@intel.com>
---
v5->v4:
add new callback to process hot insertion
---
app/test-pmd/testpmd.c | 90 ++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 90 insertions(+)
Comments
So far, about hot plug in dpdk, we already have hot plug add/remove
api and fail-safe driver to offload the fail-safe work from the app
user. But there are still lack of a general event api, since the interrupt
event, which hot plug related with, is diversity between each device and
driver, such as mlx4, pci driver and others.
Use the hot removal event for example, pci drivers not all exposure the
remove interrupt, so in order to make user to easy use the hot plug feature
for pci driver, something must be done to detect the remove event at the
kernel level and offer a new line of interrupt to the user land.
Base on the uevent of kobject mechanism in kernel, we could use it to
benefit for monitoring the hot plug status of the device which not only
uio/vfio of pci bus devices, but also other, such as cpu/usb/pci-express
bus devices.
The idea is comming as bellow.
a.The uevent message form FD monitoring which will be useful.
remove@/devices/pci0000:80/0000:80:02.2/0000:82:00.0/0000:83:03.0/0000:84:00.2/uio/uio2
ACTION=remove
DEVPATH=/devices/pci0000:80/0000:80:02.2/0000:82:00.0/0000:83:03.0/0000:84:00.2/uio/uio2
SUBSYSTEM=uio
MAJOR=243
MINOR=2
DEVNAME=uio2
SEQNUM=11366
b.add uevent monitoring machanism:
add several general api to enable uevent monitoring.
c.add common uevent handler and uevent failure handler
uevent of device should be handler at bus or device layer, and the memory read
and write failure when hot removal should be handle correctly before detach behaviors.
d.show example how to use uevent monitor
enable uevent monitoring in testpmd or fail-safe to show usage.
patchset history:
v6->v5:
1.add hot plug policy, in eal, default handle to prepare hot plug work for
all pci device, then let app to manage to deside which device need to
hot plug.
2.modify to manage event callback in each device.
3.fix some system hung issue when igb_uio release.
4.modify the pci part to the bus-pci base on the bus rework.
5.add hot plug policy in app, show example to use hotplug list to manage
to deside which device need to hot plug.
v5->v4:
1.Move uevent monitor epolling from eal interrupt to eal device layer.
2.Redefine the eal device API for common, and distinguish between linux and bsd
3.Add failure handler helper api in bus layer.Add function of find device by name.
4.Replace of individual fd bind with single device, use a common fd to polling all device.
5.Add to register hot insertion monitoring and process, add function to auto bind driver befor user add device
6.Refine some coding style and typos issue
7.add new callback to process hot insertion
v4->v3:
1.move uevent monitor api from eal interrupt to eal device layer.
2.create uevent type and struct in eal device.
3.move uevent handler for each driver to eal layer.
4.add uevent failure handler to process signal fault issue.
5.add example for request and use uevent monitoring in testpmd.
v3->v2:
1.refine some return error
2.refine the string searching logic to avoid memory issue
v2->v1:
1.remove global variables of hotplug_fd, add uevent_fd
in rte_intr_handle to let each pci device self maintain it fd,
to fix dual device fd issue.
2.refine some typo error.
Jeff Guo (2):
eal: add uevent monitor for hot plug
app/testpmd: use uevent to monitor hotplug
app/test-pmd/testpmd.c | 172 ++++++++++
app/test-pmd/testpmd.h | 9 +
drivers/bus/pci/bsd/pci.c | 23 ++
drivers/bus/pci/linux/pci.c | 34 ++
drivers/bus/pci/linux/pci_init.h | 1 +
drivers/bus/pci/pci_common.c | 42 +++
drivers/bus/pci/pci_common_uio.c | 28 ++
drivers/bus/pci/private.h | 12 +
drivers/bus/pci/rte_bus_pci.h | 9 +
lib/librte_eal/bsdapp/eal/eal_dev.c | 64 ++++
.../bsdapp/eal/include/exec-env/rte_dev.h | 105 ++++++
lib/librte_eal/common/eal_common_bus.c | 29 ++
lib/librte_eal/common/eal_common_dev.c | 222 +++++++++++++
lib/librte_eal/common/eal_common_vdev.c | 27 ++
lib/librte_eal/common/include/rte_bus.h | 51 +++
lib/librte_eal/common/include/rte_dev.h | 107 ++++++-
lib/librte_eal/linuxapp/eal/Makefile | 3 +-
lib/librte_eal/linuxapp/eal/eal_dev.c | 353 +++++++++++++++++++++
.../linuxapp/eal/include/exec-env/rte_dev.h | 105 ++++++
lib/librte_eal/linuxapp/igb_uio/igb_uio.c | 6 +
lib/librte_pci/rte_pci.c | 20 ++
lib/librte_pci/rte_pci.h | 17 +
22 files changed, 1437 insertions(+), 2 deletions(-)
create mode 100644 lib/librte_eal/bsdapp/eal/eal_dev.c
create mode 100644 lib/librte_eal/bsdapp/eal/include/exec-env/rte_dev.h
create mode 100644 lib/librte_eal/linuxapp/eal/eal_dev.c
create mode 100644 lib/librte_eal/linuxapp/eal/include/exec-env/rte_dev.h
Hello,
I would like to apply this patch in order to review it.
Trying to apply it on 17.11 (and latest) fails due to missing lib/librte_eal/common/eal_common_vdev.c
Trying to apply it on 17.08.1 fails on missing drivers/bus/pci/bsd/pci.c file
So, on what DPDK version should I apply it ?
Or maybe there is a bunch of other patches I have to apply in order to use this patch ?
Thanks
Moti H.
> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Jeff Guo
> Sent: Wednesday, November 1, 2017 10:17 PM
> To: stephen@networkplumber.org; bruce.richardson@intel.com;
> ferruh.yigit@intel.com; gaetan.rivet@6wind.com; Thomas Monjalon
> <thomas@monjalon.net>
> Cc: konstantin.ananyev@intel.com; jblunck@infradead.org;
> shreyansh.jain@nxp.com; jingjing.wu@intel.com; dev@dpdk.org;
> jia.guo@intel.com; helin.zhang@intel.com
> Subject: [dpdk-dev] [PATCH v6 0/2] add uevent monitor for hot plug
>
> So far, about hot plug in dpdk, we already have hot plug add/remove api and
> fail-safe driver to offload the fail-safe work from the app user. But there are
> still lack of a general event api, since the interrupt event, which hot plug
> related with, is diversity between each device and driver, such as mlx4, pci
> driver and others.
>
> Use the hot removal event for example, pci drivers not all exposure the
> remove interrupt, so in order to make user to easy use the hot plug feature
> for pci driver, something must be done to detect the remove event at the
> kernel level and offer a new line of interrupt to the user land.
>
> Base on the uevent of kobject mechanism in kernel, we could use it to
> benefit for monitoring the hot plug status of the device which not only
> uio/vfio of pci bus devices, but also other, such as cpu/usb/pci-express bus
> devices.
>
> The idea is comming as bellow.
>
> a.The uevent message form FD monitoring which will be useful.
> remove@/devices/pci0000:80/0000:80:02.2/0000:82:00.0/0000:83:03.0/0000:8
> 4:00.2/uio/uio2
> ACTION=remove
> DEVPATH=/devices/pci0000:80/0000:80:02.2/0000:82:00.0/0000:83:03.0/0000:
> 84:00.2/uio/uio2
> SUBSYSTEM=uio
> MAJOR=243
> MINOR=2
> DEVNAME=uio2
> SEQNUM=11366
>
> b.add uevent monitoring machanism:
> add several general api to enable uevent monitoring.
>
> c.add common uevent handler and uevent failure handler uevent of device
> should be handler at bus or device layer, and the memory read and write
> failure when hot removal should be handle correctly before detach
> behaviors.
>
> d.show example how to use uevent monitor enable uevent monitoring in
> testpmd or fail-safe to show usage.
>
> patchset history:
> v6->v5:
> 1.add hot plug policy, in eal, default handle to prepare hot plug work for all
> pci device, then let app to manage to deside which device need to hot plug.
> 2.modify to manage event callback in each device.
> 3.fix some system hung issue when igb_uio release.
> 4.modify the pci part to the bus-pci base on the bus rework.
> 5.add hot plug policy in app, show example to use hotplug list to manage to
> deside which device need to hot plug.
>
> v5->v4:
> 1.Move uevent monitor epolling from eal interrupt to eal device layer.
> 2.Redefine the eal device API for common, and distinguish between linux
> and bsd 3.Add failure handler helper api in bus layer.Add function of find
> device by name.
> 4.Replace of individual fd bind with single device, use a common fd to polling
> all device.
> 5.Add to register hot insertion monitoring and process, add function to auto
> bind driver befor user add device 6.Refine some coding style and typos issue
> 7.add new callback to process hot insertion
>
> v4->v3:
> 1.move uevent monitor api from eal interrupt to eal device layer.
> 2.create uevent type and struct in eal device.
> 3.move uevent handler for each driver to eal layer.
> 4.add uevent failure handler to process signal fault issue.
> 5.add example for request and use uevent monitoring in testpmd.
>
> v3->v2:
> 1.refine some return error
> 2.refine the string searching logic to avoid memory issue
>
> v2->v1:
> 1.remove global variables of hotplug_fd, add uevent_fd in rte_intr_handle to
> let each pci device self maintain it fd, to fix dual device fd issue.
> 2.refine some typo error.
>
> Jeff Guo (2):
> eal: add uevent monitor for hot plug
> app/testpmd: use uevent to monitor hotplug
>
> app/test-pmd/testpmd.c | 172 ++++++++++
> app/test-pmd/testpmd.h | 9 +
> drivers/bus/pci/bsd/pci.c | 23 ++
> drivers/bus/pci/linux/pci.c | 34 ++
> drivers/bus/pci/linux/pci_init.h | 1 +
> drivers/bus/pci/pci_common.c | 42 +++
> drivers/bus/pci/pci_common_uio.c | 28 ++
> drivers/bus/pci/private.h | 12 +
> drivers/bus/pci/rte_bus_pci.h | 9 +
> lib/librte_eal/bsdapp/eal/eal_dev.c | 64 ++++
> .../bsdapp/eal/include/exec-env/rte_dev.h | 105 ++++++
> lib/librte_eal/common/eal_common_bus.c | 29 ++
> lib/librte_eal/common/eal_common_dev.c | 222 +++++++++++++
> lib/librte_eal/common/eal_common_vdev.c | 27 ++
> lib/librte_eal/common/include/rte_bus.h | 51 +++
> lib/librte_eal/common/include/rte_dev.h | 107 ++++++-
> lib/librte_eal/linuxapp/eal/Makefile | 3 +-
> lib/librte_eal/linuxapp/eal/eal_dev.c | 353
> +++++++++++++++++++++
> .../linuxapp/eal/include/exec-env/rte_dev.h | 105 ++++++
> lib/librte_eal/linuxapp/igb_uio/igb_uio.c | 6 +
> lib/librte_pci/rte_pci.c | 20 ++
> lib/librte_pci/rte_pci.h | 17 +
> 22 files changed, 1437 insertions(+), 2 deletions(-) create mode 100644
> lib/librte_eal/bsdapp/eal/eal_dev.c
> create mode 100644 lib/librte_eal/bsdapp/eal/include/exec-env/rte_dev.h
> create mode 100644 lib/librte_eal/linuxapp/eal/eal_dev.c
> create mode 100644 lib/librte_eal/linuxapp/eal/include/exec-env/rte_dev.h
>
> --
> 2.7.4
Hello Moti,
On Thu, Dec 14, 2017 at 09:48:23AM +0000, Mordechay Haimovsky wrote:
> Hello,
> I would like to apply this patch in order to review it.
>
In absence of answer from Jeff,
Those two paths were modified during the 17.08 release: both pci and
vdev buses have been moved to drivers/bus.
> Trying to apply it on 17.11 (and latest) fails due to missing lib/librte_eal/common/eal_common_vdev.c
> Trying to apply it on 17.08.1 fails on missing drivers/bus/pci/bsd/pci.c file
>
Only the pci bus move was integrated by Jeff to this version of the udev
monitor. The vdev bus move however came later and should be rebased
upon.
> So, on what DPDK version should I apply it ?
> Or maybe there is a bunch of other patches I have to apply in order to use this patch ?
>
You should apply it on 17.11 IMO.
Either you take upon yourself to make it work with the new tree, or
wait for Jeff to send a new version.
Moti, Hello and sorry for be reply late until now, definitely as gaetan said that there might be some change after the version, anyway I will create a new version to benefit you all to review and further test.
Best regards,
Jeff Guo
-----Original Message-----
From: Gaëtan Rivet [mailto:gaetan.rivet@6wind.com]
Sent: Thursday, December 14, 2017 6:21 PM
To: Mordechay Haimovsky <motih@mellanox.com>
Cc: Guo, Jia <jia.guo@intel.com>; dev@dpdk.org
Subject: Re: [dpdk-dev] [PATCH v6 0/2] add uevent monitor for hot plug
Hello Moti,
On Thu, Dec 14, 2017 at 09:48:23AM +0000, Mordechay Haimovsky wrote:
> Hello,
> I would like to apply this patch in order to review it.
>
In absence of answer from Jeff,
Those two paths were modified during the 17.08 release: both pci and vdev buses have been moved to drivers/bus.
> Trying to apply it on 17.11 (and latest) fails due to missing lib/librte_eal/common/eal_common_vdev.c
> Trying to apply it on 17.08.1 fails on missing
> drivers/bus/pci/bsd/pci.c file
>
Only the pci bus move was integrated by Jeff to this version of the udev monitor. The vdev bus move however came later and should be rebased upon.
> So, on what DPDK version should I apply it ?
> Or maybe there is a bunch of other patches I have to apply in order to use this patch ?
>
You should apply it on 17.11 IMO.
Either you take upon yourself to make it work with the new tree, or wait for Jeff to send a new version.
--
Gaëtan Rivet
6WIND
Thanks Jeff,
Do you have an estimation on when will these patches be ready ?
Moti H.
> -----Original Message-----
> From: Guo, Jia [mailto:jia.guo@intel.com]
> Sent: Friday, December 22, 2017 2:16 AM
> To: Gaëtan Rivet <gaetan.rivet@6wind.com>; Mordechay Haimovsky
> <motih@mellanox.com>
> Cc: dev@dpdk.org
> Subject: RE: [dpdk-dev] [PATCH v6 0/2] add uevent monitor for hot plug
>
> Moti, Hello and sorry for be reply late until now, definitely as gaetan said that
> there might be some change after the version, anyway I will create a new
> version to benefit you all to review and further test.
>
> Best regards,
> Jeff Guo
>
> -----Original Message-----
> From: Gaëtan Rivet [mailto:gaetan.rivet@6wind.com]
> Sent: Thursday, December 14, 2017 6:21 PM
> To: Mordechay Haimovsky <motih@mellanox.com>
> Cc: Guo, Jia <jia.guo@intel.com>; dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH v6 0/2] add uevent monitor for hot plug
>
> Hello Moti,
>
> On Thu, Dec 14, 2017 at 09:48:23AM +0000, Mordechay Haimovsky wrote:
> > Hello,
> > I would like to apply this patch in order to review it.
> >
>
> In absence of answer from Jeff,
>
> Those two paths were modified during the 17.08 release: both pci and vdev
> buses have been moved to drivers/bus.
>
> > Trying to apply it on 17.11 (and latest) fails due to missing
> lib/librte_eal/common/eal_common_vdev.c
> > Trying to apply it on 17.08.1 fails on missing
> > drivers/bus/pci/bsd/pci.c file
> >
>
> Only the pci bus move was integrated by Jeff to this version of the udev
> monitor. The vdev bus move however came later and should be rebased
> upon.
>
> > So, on what DPDK version should I apply it ?
> > Or maybe there is a bunch of other patches I have to apply in order to use
> this patch ?
> >
>
> You should apply it on 17.11 IMO.
> Either you take upon yourself to make it work with the new tree, or wait for
> Jeff to send a new version.
>
> --
> Gaëtan Rivet
> 6WIND
hi,moti
please see v7 patch set , thanks.
On 12/24/2017 11:12 PM, Mordechay Haimovsky wrote:
> Thanks Jeff,
> Do you have an estimation on when will these patches be ready ?
>
> Moti H.
>
>> -----Original Message-----
>> From: Guo, Jia [mailto:jia.guo@intel.com]
>> Sent: Friday, December 22, 2017 2:16 AM
>> To: Gaëtan Rivet <gaetan.rivet@6wind.com>; Mordechay Haimovsky
>> <motih@mellanox.com>
>> Cc: dev@dpdk.org
>> Subject: RE: [dpdk-dev] [PATCH v6 0/2] add uevent monitor for hot plug
>>
>> Moti, Hello and sorry for be reply late until now, definitely as gaetan said that
>> there might be some change after the version, anyway I will create a new
>> version to benefit you all to review and further test.
>>
>> Best regards,
>> Jeff Guo
>>
>> -----Original Message-----
>> From: Gaëtan Rivet [mailto:gaetan.rivet@6wind.com]
>> Sent: Thursday, December 14, 2017 6:21 PM
>> To: Mordechay Haimovsky <motih@mellanox.com>
>> Cc: Guo, Jia <jia.guo@intel.com>; dev@dpdk.org
>> Subject: Re: [dpdk-dev] [PATCH v6 0/2] add uevent monitor for hot plug
>>
>> Hello Moti,
>>
>> On Thu, Dec 14, 2017 at 09:48:23AM +0000, Mordechay Haimovsky wrote:
>>> Hello,
>>> I would like to apply this patch in order to review it.
>>>
>> In absence of answer from Jeff,
>>
>> Those two paths were modified during the 17.08 release: both pci and vdev
>> buses have been moved to drivers/bus.
>>
>>> Trying to apply it on 17.11 (and latest) fails due to missing
>> lib/librte_eal/common/eal_common_vdev.c
>>> Trying to apply it on 17.08.1 fails on missing
>>> drivers/bus/pci/bsd/pci.c file
>>>
>> Only the pci bus move was integrated by Jeff to this version of the udev
>> monitor. The vdev bus move however came later and should be rebased
>> upon.
>>
>>> So, on what DPDK version should I apply it ?
>>> Or maybe there is a bunch of other patches I have to apply in order to use
>> this patch ?
>> You should apply it on 17.11 IMO.
>> Either you take upon yourself to make it work with the new tree, or wait for
>> Jeff to send a new version.
>>
>> --
>> Gaëtan Rivet
>> 6WIND
@@ -393,6 +393,9 @@ static void check_all_ports_link_status(uint32_t port_mask);
static int eth_event_callback(uint8_t port_id,
enum rte_eth_event_type type,
void *param, void *ret_param);
+static int eth_uevent_callback(enum rte_eal_dev_event_type type,
+ void *param, void *ret_param);
+
/*
* Check if all the ports are started.
@@ -1413,6 +1416,7 @@ start_port(portid_t pid)
struct rte_port *port;
struct ether_addr mac_addr;
enum rte_eth_event_type event_type;
+ enum rte_eal_dev_event_type dev_event_type;
if (port_id_is_invalid(pid, ENABLED_WARN))
return 0;
@@ -1547,6 +1551,21 @@ start_port(portid_t pid)
return -1;
}
}
+ rte_eal_dev_monitor_enable();
+
+ for (dev_event_type = RTE_EAL_DEV_EVENT_UNKNOWN;
+ dev_event_type < RTE_EAL_DEV_EVENT_MAX;
+ dev_event_type++) {
+ diag = rte_dev_callback_register(dev_event_type,
+ eth_uevent_callback,
+ &pi);
+ if (diag) {
+ printf("Failed to setup uevent callback for"
+ " device event %d\n",
+ dev_event_type);
+ return -1;
+ }
+ }
/* start port */
if (rte_eth_dev_start(pi) < 0) {
@@ -1883,6 +1902,35 @@ rmv_event_callback(void *arg)
dev->device->name);
}
+static void
+rmv_uevent_callback(void *arg)
+{
+ char name[RTE_ETH_NAME_MAX_LEN];
+
+ uint8_t port_id = *(uint8_t *)arg;
+
+ printf("removing device port_id:%u\n", port_id);
+
+ if (rte_eth_dev_detach(port_id, name)) {
+ RTE_LOG(ERR, USER1, "Failed to detach port '%s'\n", name);
+ return;
+ }
+
+ nb_ports = rte_eth_dev_count();
+
+ printf("Port '%s' is detached. Now total ports is %d\n",
+ name, nb_ports);
+}
+
+static void
+add_uevent_callback(void *arg)
+{
+ char *dev_name = (char *)arg;
+
+ printf("adding device %s\n", dev_name);
+ attach_port(dev_name);
+}
+
/* This function is used by the interrupt thread */
static int
eth_event_callback(uint8_t port_id, enum rte_eth_event_type type, void *param,
@@ -1924,6 +1972,48 @@ eth_event_callback(uint8_t port_id, enum rte_eth_event_type type, void *param,
return 0;
}
+/* This function is used by the interrupt thread */
+static int
+eth_uevent_callback(enum rte_eal_dev_event_type type, void *arg,
+ void *ret_param)
+{
+ static const char * const event_desc[] = {
+ [RTE_EAL_DEV_EVENT_UNKNOWN] = "Unknown",
+ [RTE_EAL_DEV_EVENT_ADD] = "add",
+ [RTE_EAL_DEV_EVENT_REMOVE] = "remove",
+ };
+
+ RTE_SET_USED(ret_param);
+
+ if (type >= RTE_EAL_DEV_EVENT_MAX) {
+ fprintf(stderr, "%s called upon invalid event %d\n",
+ __func__, type);
+ fflush(stderr);
+ } else if (event_print_mask & (UINT32_C(1) << type)) {
+ printf("%s event\n",
+ event_desc[type]);
+ fflush(stdout);
+ }
+
+ switch (type) {
+ case RTE_EAL_DEV_EVENT_ADD:
+ if (rte_eal_alarm_set(2000000,
+ add_uevent_callback, (void *)arg))
+ fprintf(stderr, "Could not set up deferred "
+ "device removal\n");
+ break;
+ case RTE_EAL_DEV_EVENT_REMOVE:
+ if (rte_eal_alarm_set(100000,
+ rmv_uevent_callback, (void *)arg))
+ fprintf(stderr, "Could not set up deferred "
+ "device removal\n");
+ break;
+ default:
+ break;
+ }
+ return 0;
+}
+
static int
set_tx_queue_stats_mapping_registers(uint8_t port_id, struct rte_port *port)
{