[dpdk-dev,v6,1/4] ethdev: add support of NIC reset
Checks
Commit Message
This patch adds a new eth_dev layer API function rte_eth_dev_reset().
A DPDK application can call this function to reset a NIC and keep
its port id afterwards.
It means that all SW resources allocated in ethdev layer should be
kept and SW and HW resources of the NIC in PMD need to be reset in
similar way that it runs PCI dev_uninit() and then dev_init().
The sequence of dev_uninit() and dev_init() can be packed into a
single interface API rte_eth_dev_reset().
Please See the comments before the declaration of rte_eht_dev_reset()
in lib/librte_ether/rte_ethdev.h to get more details on why this
function is needed, what this function does and what an application
should do after calling this function.
Signed-off-by: Wei Dai <wei.dai@intel.com>
---
lib/librte_ether/rte_ethdev.c | 16 ++++++++++++
lib/librte_ether/rte_ethdev.h | 45 ++++++++++++++++++++++++++++++++++
lib/librte_ether/rte_ether_version.map | 2 +-
3 files changed, 62 insertions(+), 1 deletion(-)
Comments
-----Original Message-----
> Date: Mon, 10 Jul 2017 18:05:41 +0800
> From: Wei Dai <wei.dai@intel.com>
> To: thomas@monjalon.net, wenzhuo.lu@intel.com,
> konstantin.ananyev@intel.com, jingjing.wu@intel.com, beilei.xing@intel.com
> CC: dev@dpdk.org, Wei Dai <wei.dai@intel.com>
> Subject: [dpdk-dev] [PATCH v6 1/4] ethdev: add support of NIC reset
> X-Mailer: git-send-email 2.7.5
>
> This patch adds a new eth_dev layer API function rte_eth_dev_reset().
> A DPDK application can call this function to reset a NIC and keep
> its port id afterwards.
> It means that all SW resources allocated in ethdev layer should be
> kept and SW and HW resources of the NIC in PMD need to be reset in
> similar way that it runs PCI dev_uninit() and then dev_init().
> The sequence of dev_uninit() and dev_init() can be packed into a
> single interface API rte_eth_dev_reset().
> Please See the comments before the declaration of rte_eht_dev_reset()
> in lib/librte_ether/rte_ethdev.h to get more details on why this
> function is needed, what this function does and what an application
> should do after calling this function.
>
> Signed-off-by: Wei Dai <wei.dai@intel.com>
> ---
> /**
> + * Reset a Ethernet device and keep its port id.
> + *
> + * A DPDK application calls this function to do an initiative or passive
> + * reset of a port.
> + *
> + * Sometimes a port have to be reset passively. For example a PF is reset,
> + * all its VFs should also be reset by application itself to align with the
> + * PF.
May be I didn't understood this use case properly,But, PF can always send mailbox
message to VF on PF reset. Right?
On such message from PF, VF can transparently reset without application
knowledge(i.e rx and/or tx burst return zero)
> + * A DPDK application also can call this function to trigger an initative
> + * port reset.
When apart from the above use case? Even if it is above case, we should have
some event to notify application to initiate the reset.Right? Without the
event, When or on what basics application needs to initiate reset?
> + *
-----Original Message-----
From: Jerin Jacob [mailto:jerin.jacob@caviumnetworks.com]
Sent: Monday, July 10, 2017 7:35 PM
To: Dai, Wei <wei.dai@intel.com>
Cc: thomas@monjalon.net; Lu, Wenzhuo <wenzhuo.lu@intel.com>; Ananyev, Konstantin <konstantin.ananyev@intel.com>; Wu, Jingjing <jingjing.wu@intel.com>; Xing, Beilei <beilei.xing@intel.com>; dev@dpdk.org
Subject: Re: [dpdk-dev] [PATCH v6 1/4] ethdev: add support of NIC reset
-----Original Message-----
> Date: Mon, 10 Jul 2017 18:05:41 +0800
> From: Wei Dai <wei.dai@intel.com>
> To: thomas@monjalon.net, wenzhuo.lu@intel.com,
> konstantin.ananyev@intel.com, jingjing.wu@intel.com,
> beilei.xing@intel.com
> CC: dev@dpdk.org, Wei Dai <wei.dai@intel.com>
> Subject: [dpdk-dev] [PATCH v6 1/4] ethdev: add support of NIC reset
> X-Mailer: git-send-email 2.7.5
>
> This patch adds a new eth_dev layer API function rte_eth_dev_reset().
> A DPDK application can call this function to reset a NIC and keep its
> port id afterwards.
> It means that all SW resources allocated in ethdev layer should be
> kept and SW and HW resources of the NIC in PMD need to be reset in
> similar way that it runs PCI dev_uninit() and then dev_init().
> The sequence of dev_uninit() and dev_init() can be packed into a
> single interface API rte_eth_dev_reset().
> Please See the comments before the declaration of rte_eht_dev_reset()
> in lib/librte_ether/rte_ethdev.h to get more details on why this
> function is needed, what this function does and what an application
> should do after calling this function.
>
> Signed-off-by: Wei Dai <wei.dai@intel.com>
> ---
> /**
> + * Reset a Ethernet device and keep its port id.
> + *
> + * A DPDK application calls this function to do an initiative or
> + passive
> + * reset of a port.
> + *
> + * Sometimes a port have to be reset passively. For example a PF is
> + reset,
> + * all its VFs should also be reset by application itself to align
> + with the
> + * PF.
May be I didn't understood this use case properly,But, PF can always send mailbox message to VF on PF reset. Right?
[Wei: As to ixgbe, PF kernel driver always send mailbox message to VF when PF is reset. For other NICs, there are maybe
other mechanism to notify VF of PF reset.]
On such message from PF, VF can transparently reset without application knowledge(i.e rx and/or tx burst return zero)
[Wei: VF reset is handling many HW registers which may have effect on working Rx/Tx path and may cause some unexpected
behavior like crash. So application should make some operations like stopping Rx/Tx path before it begin VF reset. This is why
application should handle VF reset itself.]
> + * A DPDK application also can call this function to trigger an
> + initative
> + * port reset.
When apart from the above use case? Even if it is above case, we should have some event to notify application to initiate the reset.Right? Without the event, When or on what basics application needs to initiate reset?
[Wei: Indeed, until now we didn't see any use case which DPDK application initiative port reset itself.
The most usual case is that PF is working with kernel driver and VFs are working with DPDK PMD.
Some operations on kernel PF lead to a PF reset which causes its VF reset.
Anyway this new function provides a possibility for application to trigger an initiative port reset.]
> + *
-----Original Message-----
> Date: Tue, 11 Jul 2017 01:57:15 +0000
> From: "Dai, Wei" <wei.dai@intel.com>
> To: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> CC: "thomas@monjalon.net" <thomas@monjalon.net>, "Lu, Wenzhuo"
> <wenzhuo.lu@intel.com>, "Ananyev, Konstantin"
> <konstantin.ananyev@intel.com>, "Wu, Jingjing" <jingjing.wu@intel.com>,
> "Xing, Beilei" <beilei.xing@intel.com>, "dev@dpdk.org" <dev@dpdk.org>
> Subject: RE: [dpdk-dev] [PATCH v6 1/4] ethdev: add support of NIC reset
>
>
> > + * A DPDK application also can call this function to trigger an
> > + initative
> > + * port reset.
>
> When apart from the above use case? Even if it is above case, we should have some event to notify application to initiate the reset.Right? Without the event, When or on what basics application needs to initiate reset?
> [Wei: Indeed, until now we didn't see any use case which DPDK application initiative port reset itself.
> The most usual case is that PF is working with kernel driver and VFs are working with DPDK PMD.
> Some operations on kernel PF lead to a PF reset which causes its VF reset.
> Anyway this new function provides a possibility for application to trigger an initiative port reset.]
That's fine. The only concern part is when to call reset API from
application. Can we say on RTE_ETH_EVENT_INTR_RESET event, application
needs to call the reset API? I think, it is important to specify when
application need to call this API, otherwise this api behavior will be
tightly coupled with underneath PMD. Side effect is, a new, non portable PMD specific API.
If a PMD wishes to fixup some overflow case(as an example), by invoking the reset API from
the application BUT same case may not valid for another PMD hence it
will create unexpected behavior from application based on the underneath PMD.
if RTE_ETH_EVENT_INTR_RESET event is not expected event to call the
reset API then create a new event or if it needs to be called in
RTE_ETH_EVENT_INTR_RESET then update the API documentation.
>
> > + *
> -----Original Message-----
> From: Jerin Jacob [mailto:jerin.jacob@caviumnetworks.com]
> Sent: Tuesday, July 11, 2017 1:17 PM
> To: Dai, Wei <wei.dai@intel.com>
> Cc: thomas@monjalon.net; Lu, Wenzhuo <wenzhuo.lu@intel.com>;
> Ananyev, Konstantin <konstantin.ananyev@intel.com>; Wu, Jingjing
> <jingjing.wu@intel.com>; Xing, Beilei <beilei.xing@intel.com>;
> dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH v6 1/4] ethdev: add support of NIC reset
>
> -----Original Message-----
> > Date: Tue, 11 Jul 2017 01:57:15 +0000
> > From: "Dai, Wei" <wei.dai@intel.com>
> > To: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> > CC: "thomas@monjalon.net" <thomas@monjalon.net>, "Lu, Wenzhuo"
> > <wenzhuo.lu@intel.com>, "Ananyev, Konstantin"
> > <konstantin.ananyev@intel.com>, "Wu, Jingjing"
> > <jingjing.wu@intel.com>, "Xing, Beilei" <beilei.xing@intel.com>,
> > "dev@dpdk.org" <dev@dpdk.org>
> > Subject: RE: [dpdk-dev] [PATCH v6 1/4] ethdev: add support of NIC
> > reset
> >
> >
> > > + * A DPDK application also can call this function to trigger an
> > > + initative
> > > + * port reset.
> >
> > When apart from the above use case? Even if it is above case, we should
> have some event to notify application to initiate the reset.Right? Without
> the event, When or on what basics application needs to initiate reset?
> > [Wei: Indeed, until now we didn't see any use case which DPDK application
> initiative port reset itself.
> > The most usual case is that PF is working with kernel driver and VFs are
> working with DPDK PMD.
> > Some operations on kernel PF lead to a PF reset which causes its VF reset.
> > Anyway this new function provides a possibility for application to
> > trigger an initiative port reset.]
>
> That's fine. The only concern part is when to call reset API from application.
> Can we say on RTE_ETH_EVENT_INTR_RESET event, application needs to
> call the reset API? I think, it is important to specify when application need to
> call this API, otherwise this api behavior will be tightly coupled with
> underneath PMD. Side effect is, a new, non portable PMD specific API.
> If a PMD wishes to fixup some overflow case(as an example), by invoking the
> reset API from the application BUT same case may not valid for another
> PMD hence it will create unexpected behavior from application based on the
> underneath PMD.
It is duty of PMD to trigger RTE_ETH_EVENT_INTR_RESET event and application
should also register some callback function to handle this event.
When PMD wants to trigger a reset, it can trigger RTE_ETH_EVENT_INTR_RESET.
On the received event of RTE_ETH_EVENT_INTR_RESET, application can begin to
handle it: stop working queues, make rx and tx function not be called
and then call rte_eth_dev_reset( ).
For thread safety, all these controlling operations had better be made in same thread.
For example, when ixgbe PF is reset, PF send a message to notify VF this event and
also trigger an interrupt to VF. And then in the interrupt service routine DPDK VF
detect this notification message and calls
_rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_INTR_RESET, NULL, NULL).
So it means that PF reset trigger RTE_ETH_EVENT_INTR_RESET event in VF.
The function _rte_eth_dev_callback_process( ) will call the registered callback function.
The callback function can trigger application to handle all operations of VF reset including
something like stopping working Rx/Tx queues and call this rte_eth_dev_reset().
The rte_eth_dev_reset( ) itself is generic function which only does some HW reset operations
through calling dev_unint() and dev_init(). And itself doesn't handle above synchronization which
is handled by application.
PMD itself should not call rte_eth_dev_reset( ). PMD can trigger application to handle reset event.
It is duty of application to handle all synchronization befort it calls rte_eth_dev_reset( ).
>
> if RTE_ETH_EVENT_INTR_RESET event is not expected event to call the reset
> API then create a new event or if it needs to be called in
> RTE_ETH_EVENT_INTR_RESET then update the API documentation.
>
Of course, when PMD wants to trigger a reset event, it can trigger other event other than
RTE_ETH_EVENT_INTR_RESET. So the application should know which the alternate event is.
This make application more complex. So it is suggested that only RTE_ETH_EVENT_INTR_RESET
can be used to trigger a port reset.
> >
> > > + *
-----Original Message-----
> Date: Tue, 11 Jul 2017 14:36:57 +0000
> From: "Dai, Wei" <wei.dai@intel.com>
> To: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> CC: "thomas@monjalon.net" <thomas@monjalon.net>, "Lu, Wenzhuo"
> <wenzhuo.lu@intel.com>, "Ananyev, Konstantin"
> <konstantin.ananyev@intel.com>, "Wu, Jingjing" <jingjing.wu@intel.com>,
> "Xing, Beilei" <beilei.xing@intel.com>, "dev@dpdk.org" <dev@dpdk.org>
> Subject: RE: [dpdk-dev] [PATCH v6 1/4] ethdev: add support of NIC reset
>
> > -----Original Message-----
> > From: Jerin Jacob [mailto:jerin.jacob@caviumnetworks.com]
> > Sent: Tuesday, July 11, 2017 1:17 PM
> > To: Dai, Wei <wei.dai@intel.com>
> > Cc: thomas@monjalon.net; Lu, Wenzhuo <wenzhuo.lu@intel.com>;
> > Ananyev, Konstantin <konstantin.ananyev@intel.com>; Wu, Jingjing
> > <jingjing.wu@intel.com>; Xing, Beilei <beilei.xing@intel.com>;
> > dev@dpdk.org
> > Subject: Re: [dpdk-dev] [PATCH v6 1/4] ethdev: add support of NIC reset
> >
> > -----Original Message-----
> > > Date: Tue, 11 Jul 2017 01:57:15 +0000
> > > From: "Dai, Wei" <wei.dai@intel.com>
> > > To: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> > > CC: "thomas@monjalon.net" <thomas@monjalon.net>, "Lu, Wenzhuo"
> > > <wenzhuo.lu@intel.com>, "Ananyev, Konstantin"
> > > <konstantin.ananyev@intel.com>, "Wu, Jingjing"
> > > <jingjing.wu@intel.com>, "Xing, Beilei" <beilei.xing@intel.com>,
> > > "dev@dpdk.org" <dev@dpdk.org>
> > > Subject: RE: [dpdk-dev] [PATCH v6 1/4] ethdev: add support of NIC
> > > reset
> > >
> > >
> > > > + * A DPDK application also can call this function to trigger an
> > > > + initative
> > > > + * port reset.
> > >
> > > When apart from the above use case? Even if it is above case, we should
> > have some event to notify application to initiate the reset.Right? Without
> > the event, When or on what basics application needs to initiate reset?
> > > [Wei: Indeed, until now we didn't see any use case which DPDK application
> > initiative port reset itself.
> > > The most usual case is that PF is working with kernel driver and VFs are
> > working with DPDK PMD.
> > > Some operations on kernel PF lead to a PF reset which causes its VF reset.
> > > Anyway this new function provides a possibility for application to
> > > trigger an initiative port reset.]
> >
> > That's fine. The only concern part is when to call reset API from application.
> > Can we say on RTE_ETH_EVENT_INTR_RESET event, application needs to
> > call the reset API? I think, it is important to specify when application need to
> > call this API, otherwise this api behavior will be tightly coupled with
> > underneath PMD. Side effect is, a new, non portable PMD specific API.
> > If a PMD wishes to fixup some overflow case(as an example), by invoking the
> > reset API from the application BUT same case may not valid for another
> > PMD hence it will create unexpected behavior from application based on the
> > underneath PMD.
> It is duty of PMD to trigger RTE_ETH_EVENT_INTR_RESET event and application
> should also register some callback function to handle this event.
> When PMD wants to trigger a reset, it can trigger RTE_ETH_EVENT_INTR_RESET.
> On the received event of RTE_ETH_EVENT_INTR_RESET, application can begin to
> handle it: stop working queues, make rx and tx function not be called
> and then call rte_eth_dev_reset( ).
> For thread safety, all these controlling operations had better be made in same thread.
> For example, when ixgbe PF is reset, PF send a message to notify VF this event and
> also trigger an interrupt to VF. And then in the interrupt service routine DPDK VF
> detect this notification message and calls
> _rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_INTR_RESET, NULL, NULL).
> So it means that PF reset trigger RTE_ETH_EVENT_INTR_RESET event in VF.
> The function _rte_eth_dev_callback_process( ) will call the registered callback function.
> The callback function can trigger application to handle all operations of VF reset including
> something like stopping working Rx/Tx queues and call this rte_eth_dev_reset().
> The rte_eth_dev_reset( ) itself is generic function which only does some HW reset operations
> through calling dev_unint() and dev_init(). And itself doesn't handle above synchronization which
> is handled by application.
> PMD itself should not call rte_eth_dev_reset( ). PMD can trigger application to handle reset event.
> It is duty of application to handle all synchronization befort it calls rte_eth_dev_reset( ).
No disagreement on the expected behavior.
>
> >
> > if RTE_ETH_EVENT_INTR_RESET event is not expected event to call the reset
> > API then create a new event or if it needs to be called in
> > RTE_ETH_EVENT_INTR_RESET then update the API documentation.
> >
> Of course, when PMD wants to trigger a reset event, it can trigger other event other than
> RTE_ETH_EVENT_INTR_RESET. So the application should know which the alternate event is.
> This make application more complex. So it is suggested that only RTE_ETH_EVENT_INTR_RESET
> can be used to trigger a port reset.
Yes. I suggest to add this info on documentation. ie "application invokes the
reset API on RTE_ETH_EVENT_INTR_RESET event". That will answer "when"
application need to invoke this API.
>
> > >
> > > > + *
> > Subject: RE: [dpdk-dev] [PATCH v6 1/4] ethdev: add support of NIC
> > reset
> >
> >
> > > + * A DPDK application also can call this function to trigger an
> > > + initative
> > > + * port reset.
> >
> > When apart from the above use case? Even if it is above case, we should
> have some event to notify application to initiate the reset.Right? Without
> the event, When or on what basics application needs to initiate reset?
> > [Wei: Indeed, until now we didn't see any use case which DPDK application
> initiative port reset itself.
> > The most usual case is that PF is working with kernel driver and VFs are
> working with DPDK PMD.
> > Some operations on kernel PF lead to a PF reset which causes its VF reset.
> > Anyway this new function provides a possibility for application to
> > trigger an initiative port reset.]
>
> That's fine. The only concern part is when to call reset API from application.
> Can we say on RTE_ETH_EVENT_INTR_RESET event, application needs to
> call the reset API? I think, it is important to specify when application need to
> call this API, otherwise this api behavior will be tightly coupled with
> underneath PMD. Side effect is, a new, non portable PMD specific API.
> If a PMD wishes to fixup some overflow case(as an example), by invoking the
> reset API from the application BUT same case may not valid for another
> PMD hence it will create unexpected behavior from application based on the
> underneath PMD.
>
> if RTE_ETH_EVENT_INTR_RESET event is not expected event to call the reset
> API then create a new event or if it needs to be called in
> RTE_ETH_EVENT_INTR_RESET then update the API documentation.
Thanks for your feedback.
I'll add description of this function in API doc in my v7 patch set.
>
> >
> > > + *
@@ -3414,3 +3414,19 @@ rte_eth_dev_adjust_nb_rx_tx_desc(uint8_t port_id,
return 0;
}
+
+int
+rte_eth_dev_reset(uint8_t port_id)
+{
+ struct rte_eth_dev *dev;
+ int ret;
+
+ RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);
+ dev = &rte_eth_devices[port_id];
+
+ RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_reset, -ENOTSUP);
+
+ rte_eth_dev_stop(port_id);
+ ret = dev->dev_ops->dev_reset(dev);
+ return ret;
+}
@@ -1113,6 +1113,9 @@ typedef int (*eth_dev_set_link_down_t)(struct rte_eth_dev *dev);
typedef void (*eth_dev_close_t)(struct rte_eth_dev *dev);
/**< @internal Function used to close a configured Ethernet device. */
+typedef int (*eth_dev_reset_t)(struct rte_eth_dev *dev);
+/** <@internal Function used to reset a configured Ethernet device. */
+
typedef void (*eth_promiscuous_enable_t)(struct rte_eth_dev *dev);
/**< @internal Function used to enable the RX promiscuous mode of an Ethernet device. */
@@ -1430,6 +1433,7 @@ struct eth_dev_ops {
eth_dev_set_link_up_t dev_set_link_up; /**< Device link up. */
eth_dev_set_link_down_t dev_set_link_down; /**< Device link down. */
eth_dev_close_t dev_close; /**< Close device. */
+ eth_dev_reset_t dev_reset; /**< Reset device. */
eth_link_update_t link_update; /**< Get device link state. */
eth_promiscuous_enable_t promiscuous_enable; /**< Promiscuous ON. */
@@ -2132,6 +2136,47 @@ int rte_eth_dev_set_link_down(uint8_t port_id);
void rte_eth_dev_close(uint8_t port_id);
/**
+ * Reset a Ethernet device and keep its port id.
+ *
+ * A DPDK application calls this function to do an initiative or passive
+ * reset of a port.
+ *
+ * Sometimes a port have to be reset passively. For example a PF is reset,
+ * all its VFs should also be reset by application itself to align with the
+ * PF.
+ * A DPDK application also can call this function to trigger an initative
+ * port reset.
+ *
+ * When processing reset, if the port goes through PCI remove() and then
+ * PCI probe() for restoration, its port id may be changed and this is not
+ * expected by some DPDK application.
+ *
+ * Normally, PCI probe() includes two parts: one is in rte_ethdev layer
+ * to allocate resource in rte_ethdev layer and the other is calling PMD
+ * specific dev_init() to allocate and initialize resource in PMD layer.
+ * PCI remove( ) releases all resource allocated from rte_ethdev layer
+ * in PCI probe( ) and calls PMD specific dev_uninit( ) to releaase
+ * resource allocated by dev_init( ) in PMD layer.
+ *
+ * To keep same port id and reset the port, only dev_uninit() and
+ * dev_init( ) in PMD can be called and keep all resources allocated
+ * from rte_ethdev layer poart in PCI probe( ). All these are what
+ * rte_eth_dev_reset() does.
+ *
+ * The rte_eth_dev_reset( ) calls rte_eth_dev_stop( ), PMD dev_uninit( )
+ * and then PMD dev_init( ) to reset a port and keep same port id.
+ *
+ * After calling rte_eth_dev_reset( ), the application can go through
+ * rte_eth_dev_configure( ), rte_eth_rx_queue_setup( ),
+ * rte_eth_tx_queue_setup( ) and rte_eth_dev_start( ) again to restore
+ * its previous settings or to reconfigure itself with different settings.
+ *
+ * @param port_id
+ * The port identifier of the Ethernet device.
+ */
+int rte_eth_dev_reset(uint8_t port_id);
+
+/**
* Enable receipt in promiscuous mode for an Ethernet device.
*
* @param port_id
@@ -155,5 +155,5 @@ DPDK_17.08 {
rte_eth_dev_adjust_nb_rx_tx_desc;
rte_flow_copy;
rte_flow_isolate;
-
+ rte_eth_dev_reset;
} DPDK_17.05;