[dpdk-dev] eventdev: remove experimental label

  The eventdev API was introduced in DPDK 17.05 release.
Since then it
- has been reviewed and iterated for 17.08, 17.11 releases
- three drivers were implemented using the API.
- introduced another subsystem like service core and ethdev-eventdev Rx
adapter APIs to abstract the difference between HW and SW
eventdev implementations in a transparent way.
- had extensive use by the app/test-eventdev/ and
examples/eventdev_pipeline_sw_pmd/

I believe the API is now stable and the EXPERIMENTAL label
should be removed.

CC: Bruce Richardson <bruce.richardson@intel.com>
CC: Harry van Haaren <harry.van.haaren@intel.com>
CC: Gage Eads <gage.eads@intel.com>
CC: Hemant Agrawal <hemant.agrawal@nxp.com>
CC: Nipun Gupta <nipun.gupta@nxp.com>
CC: Nikhil Rao <nikhil.rao@intel.com>
CC: Pavan Nikhilesh <pbhagavatula@caviumnetworks.com>
CC: Thomas Monjalon <thomas@monjalon.net>
Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
---

There are two more outstanding eventdev API changes. Please find below.
Please express if you have any concern in changing those APIs. I would         
like to fix this API issue and remove experimental tag in v17.11,
if we all agree.            
                                                                                
- evendev: fix inconsistency in event queue config                              
http://dpdk.org/dev/patchwork/patch/30293/                                      
- remove rte_event_schedule() API and use service core infrastructure 
http://dpdk.org/dev/patchwork/patch/30375/  

---
 MAINTAINERS | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
  

Hi Jerin,

I have one concern with the API that may delay changing the label.

The implicit release that in rte_event_dequeue_burst() is a problem when using asynchronous/look-aside hardware, like a cryptodev. For instance, let's say in pipeline stage N the worker takes the event's mbuf and places it in a per-worker crypto request queue. When the worker next calls rte_event_dequeue_burst(), that function will release the previous event which could cause the flow to migrate to another worker, and this could result in packet reordering.

To prevent this, the worker can't call dequeue until the look-aside operation completes...in effect treating the asynchronous/look-aside hardware as synchronous. Another option is to feed stage N's queue to a single port to avoid the flow migration, but that port may become a bottleneck.

We could remove the implicit release functionality or add a port configuration flag to disable it, so the default behavior is unchanged. Removing it will completely will likely require changes in existing code, but it simplifies the usage model (all dequeued events must be either forwarded or released) and the PMD's dequeue code. This functionality could be removed from the software eventdev fairly easily, but I haven't looked into the hardware PMDs.

Thanks,
Gage

> -----Original Message-----
> From: Jerin Jacob [mailto:jerin.jacob@caviumnetworks.com]
> Sent: Monday, October 16, 2017 5:33 AM
> To: dev@dpdk.org
> Cc: Jerin Jacob <jerin.jacob@caviumnetworks.com>; Richardson, Bruce
> <bruce.richardson@intel.com>; Van Haaren, Harry
> <harry.van.haaren@intel.com>; Eads, Gage <gage.eads@intel.com>; Hemant
> Agrawal <hemant.agrawal@nxp.com>; Nipun Gupta <nipun.gupta@nxp.com>;
> Rao, Nikhil <nikhil.rao@intel.com>; Pavan Nikhilesh
> <pbhagavatula@caviumnetworks.com>; Thomas Monjalon
> <thomas@monjalon.net>
> Subject: [dpdk-dev] [PATCH] eventdev: remove experimental label
> 
> The eventdev API was introduced in DPDK 17.05 release.
> Since then it
> - has been reviewed and iterated for 17.08, 17.11 releases
> - three drivers were implemented using the API.
> - introduced another subsystem like service core and ethdev-eventdev Rx
> adapter APIs to abstract the difference between HW and SW eventdev
> implementations in a transparent way.
> - had extensive use by the app/test-eventdev/ and
> examples/eventdev_pipeline_sw_pmd/
> 
> I believe the API is now stable and the EXPERIMENTAL label should be removed.
> 
> CC: Bruce Richardson <bruce.richardson@intel.com>
> CC: Harry van Haaren <harry.van.haaren@intel.com>
> CC: Gage Eads <gage.eads@intel.com>
> CC: Hemant Agrawal <hemant.agrawal@nxp.com>
> CC: Nipun Gupta <nipun.gupta@nxp.com>
> CC: Nikhil Rao <nikhil.rao@intel.com>
> CC: Pavan Nikhilesh <pbhagavatula@caviumnetworks.com>
> CC: Thomas Monjalon <thomas@monjalon.net>
> Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> ---
> 
> There are two more outstanding eventdev API changes. Please find below.
> Please express if you have any concern in changing those APIs. I would
> like to fix this API issue and remove experimental tag in v17.11,
> if we all agree.
> 
> - evendev: fix inconsistency in event queue config
> http://dpdk.org/dev/patchwork/patch/30293/
> - remove rte_event_schedule() API and use service core infrastructure
> http://dpdk.org/dev/patchwork/patch/30375/
> 
> ---
>  MAINTAINERS | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 2a58378b7..4a4be3a54 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -262,7 +262,7 @@ F: lib/librte_cryptodev/
>  F: test/test/test_cryptodev*
>  F: examples/l2fwd-crypto/
> 
> -Eventdev API - EXPERIMENTAL
> +Eventdev API
>  M: Jerin Jacob <jerin.jacob@caviumnetworks.com>
>  T: git://dpdk.org/next/dpdk-next-eventdev
>  F: lib/librte_eventdev/
> --
> 2.14.2
  

-----Original Message-----
> Date: Mon, 23 Oct 2017 18:27:52 +0000
> From: "Eads, Gage" <gage.eads@intel.com>
> To: Jerin Jacob <jerin.jacob@caviumnetworks.com>, "dev@dpdk.org"
>  <dev@dpdk.org>
> CC: "Richardson, Bruce" <bruce.richardson@intel.com>, "Van Haaren, Harry"
>  <harry.van.haaren@intel.com>, Hemant Agrawal <hemant.agrawal@nxp.com>,
>  Nipun Gupta <nipun.gupta@nxp.com>, "Rao, Nikhil" <nikhil.rao@intel.com>,
>  Pavan Nikhilesh <pbhagavatula@caviumnetworks.com>, Thomas Monjalon
>  <thomas@monjalon.net>
> Subject: RE: [dpdk-dev] [PATCH] eventdev: remove experimental label
> 
> Hi Jerin,

Hi Gage,

> 
> I have one concern with the API that may delay changing the label.
> 
> The implicit release that in rte_event_dequeue_burst() is a problem when using asynchronous/look-aside hardware, like a cryptodev. For instance, let's say in pipeline stage N the worker takes the event's mbuf and places it in a per-worker crypto request queue. When the worker next calls rte_event_dequeue_burst(), that function will release the previous event which could cause the flow to migrate to another worker, and this could result in packet reordering.
> 
> To prevent this, the worker can't call dequeue until the look-aside operation completes...in effect treating the asynchronous/look-aside hardware as synchronous. Another option is to feed stage N's queue to a single port to avoid the flow migration, but that port may become a bottleneck.
> 
> We could remove the implicit release functionality or add a port configuration flag to disable it, so the default behavior is unchanged. Removing it will completely will likely require changes in existing code, but it simplifies the usage model (all dequeued events must be either forwarded or released) and the PMD's dequeue code. This functionality could be removed from the software eventdev fairly easily, but I haven't looked into the hardware PMDs.



The HW implementations, I know, it does the implicit release. Otherwise it      
will result in deadlock because it cannot hold reordering metadata for
the longtime(SRAM is limited etc)                                             
                                                                                
Coming back to cryptodev use case, if I understand it correctly, before
application enqueues to crypto queue, the application will change the tag and
submit to ATOMIC queue. So as long as crypto queue competes for the 
crypto work in order then the order will be maintained.                        
                                                                                
In typical outbound IPSec use case,                                             
- Stage 1 will be processed in ORDERED where application does the SA        
  lookup                                                                        
- Once SA found, application enqueue to ATOMIC stage with SA as flow_id.        
- When the event comes from the ATOMIC queue, it in ingress order and
  then it submits to the crypto queue                                          
- Crypto queue maintains the FIFO order.                                        
- On IPSec crypto work competition, packets will come in Stage 3.              
- So at Stage 3, packets are in ingress order for the given SA flow id.   
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^         
                                                                                
Having said that, If SW implementation needs to do differently for
performance reasons then we will end up in capability as HW
implementation works in the implicit release. May we can sort out
through capability or separate adapter for crypto case. But I think, those will
be new additions to the API.So removing the experimental tags may be OK.
But if you have strong opinion on keeping the experimental tag till we address
the crypto use case then I am fine with that.
                                                                                
Thoughts? 

Jerin


> 
> Thanks,
> Gage
> 
> > -----Original Message-----
> > From: Jerin Jacob [mailto:jerin.jacob@caviumnetworks.com]
> > Sent: Monday, October 16, 2017 5:33 AM
> > To: dev@dpdk.org
> > Cc: Jerin Jacob <jerin.jacob@caviumnetworks.com>; Richardson, Bruce
> > <bruce.richardson@intel.com>; Van Haaren, Harry
> > <harry.van.haaren@intel.com>; Eads, Gage <gage.eads@intel.com>; Hemant
> > Agrawal <hemant.agrawal@nxp.com>; Nipun Gupta <nipun.gupta@nxp.com>;
> > Rao, Nikhil <nikhil.rao@intel.com>; Pavan Nikhilesh
> > <pbhagavatula@caviumnetworks.com>; Thomas Monjalon
> > <thomas@monjalon.net>
> > Subject: [dpdk-dev] [PATCH] eventdev: remove experimental label
> > 
> > The eventdev API was introduced in DPDK 17.05 release.
> > Since then it
> > - has been reviewed and iterated for 17.08, 17.11 releases
> > - three drivers were implemented using the API.
> > - introduced another subsystem like service core and ethdev-eventdev Rx
> > adapter APIs to abstract the difference between HW and SW eventdev
> > implementations in a transparent way.
> > - had extensive use by the app/test-eventdev/ and
> > examples/eventdev_pipeline_sw_pmd/
> > 
> > I believe the API is now stable and the EXPERIMENTAL label should be removed.
> > 
> > CC: Bruce Richardson <bruce.richardson@intel.com>
> > CC: Harry van Haaren <harry.van.haaren@intel.com>
> > CC: Gage Eads <gage.eads@intel.com>
> > CC: Hemant Agrawal <hemant.agrawal@nxp.com>
> > CC: Nipun Gupta <nipun.gupta@nxp.com>
> > CC: Nikhil Rao <nikhil.rao@intel.com>
> > CC: Pavan Nikhilesh <pbhagavatula@caviumnetworks.com>
> > CC: Thomas Monjalon <thomas@monjalon.net>
> > Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> > ---
> > 
> > There are two more outstanding eventdev API changes. Please find below.
> > Please express if you have any concern in changing those APIs. I would
> > like to fix this API issue and remove experimental tag in v17.11,
> > if we all agree.
> > 
> > - evendev: fix inconsistency in event queue config
> > http://dpdk.org/dev/patchwork/patch/30293/
> > - remove rte_event_schedule() API and use service core infrastructure
> > http://dpdk.org/dev/patchwork/patch/30375/
> > 
> > ---
> >  MAINTAINERS | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/MAINTAINERS b/MAINTAINERS
> > index 2a58378b7..4a4be3a54 100644
> > --- a/MAINTAINERS
> > +++ b/MAINTAINERS
> > @@ -262,7 +262,7 @@ F: lib/librte_cryptodev/
> >  F: test/test/test_cryptodev*
> >  F: examples/l2fwd-crypto/
> > 
> > -Eventdev API - EXPERIMENTAL
> > +Eventdev API
> >  M: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> >  T: git://dpdk.org/next/dpdk-next-eventdev
> >  F: lib/librte_eventdev/
> > --
> > 2.14.2
>
  

> -----Original Message-----
> From: Jerin Jacob [mailto:jerin.jacob@caviumnetworks.com]
> Sent: Monday, October 30, 2017 12:38 PM
> To: Eads, Gage <gage.eads@intel.com>
> Cc: dev@dpdk.org; Richardson, Bruce <bruce.richardson@intel.com>; Van
> Haaren, Harry <harry.van.haaren@intel.com>; Hemant Agrawal
> <hemant.agrawal@nxp.com>; Nipun Gupta <nipun.gupta@nxp.com>; Rao,
> Nikhil <nikhil.rao@intel.com>; Pavan Nikhilesh
> <pbhagavatula@caviumnetworks.com>; Thomas Monjalon
> <thomas@monjalon.net>
> Subject: Re: [dpdk-dev] [PATCH] eventdev: remove experimental label
> 
> -----Original Message-----
> > Date: Mon, 23 Oct 2017 18:27:52 +0000
> > From: "Eads, Gage" <gage.eads@intel.com>
> > To: Jerin Jacob <jerin.jacob@caviumnetworks.com>, "dev@dpdk.org"
> >  <dev@dpdk.org>
> > CC: "Richardson, Bruce" <bruce.richardson@intel.com>, "Van Haaren, Harry"
> >  <harry.van.haaren@intel.com>, Hemant Agrawal
> > <hemant.agrawal@nxp.com>,  Nipun Gupta <nipun.gupta@nxp.com>, "Rao,
> > Nikhil" <nikhil.rao@intel.com>,  Pavan Nikhilesh
> > <pbhagavatula@caviumnetworks.com>, Thomas Monjalon
> > <thomas@monjalon.net>
> > Subject: RE: [dpdk-dev] [PATCH] eventdev: remove experimental label
> >
> > Hi Jerin,
> 
> Hi Gage,
> 
> >
> > I have one concern with the API that may delay changing the label.
> >
> > The implicit release that in rte_event_dequeue_burst() is a problem when using
> asynchronous/look-aside hardware, like a cryptodev. For instance, let's say in
> pipeline stage N the worker takes the event's mbuf and places it in a per-worker
> crypto request queue. When the worker next calls rte_event_dequeue_burst(),
> that function will release the previous event which could cause the flow to
> migrate to another worker, and this could result in packet reordering.
> >
> > To prevent this, the worker can't call dequeue until the look-aside operation
> completes...in effect treating the asynchronous/look-aside hardware as
> synchronous. Another option is to feed stage N's queue to a single port to avoid
> the flow migration, but that port may become a bottleneck.
> >
> > We could remove the implicit release functionality or add a port configuration
> flag to disable it, so the default behavior is unchanged. Removing it will
> completely will likely require changes in existing code, but it simplifies the usage
> model (all dequeued events must be either forwarded or released) and the
> PMD's dequeue code. This functionality could be removed from the software
> eventdev fairly easily, but I haven't looked into the hardware PMDs.
> 
> 
> 
> The HW implementations, I know, it does the implicit release. Otherwise it
> will result in deadlock because it cannot hold reordering metadata for
> the longtime(SRAM is limited etc)
> 
> Coming back to cryptodev use case, if I understand it correctly, before
> application enqueues to crypto queue, the application will change the tag and
> submit to ATOMIC queue. So as long as crypto queue competes for the
> crypto work in order then the order will be maintained.
> 
> In typical outbound IPSec use case,
> - Stage 1 will be processed in ORDERED where application does the SA
>   lookup
> - Once SA found, application enqueue to ATOMIC stage with SA as flow_id.
> - When the event comes from the ATOMIC queue, it in ingress order and
>   then it submits to the crypto queue
> - Crypto queue maintains the FIFO order.
> - On IPSec crypto work competition, packets will come in Stage 3.
> - So at Stage 3, packets are in ingress order for the given SA flow id.
>                                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
> 
> Having said that, If SW implementation needs to do differently for performance
> reasons then we will end up in capability as HW implementation works in the
> implicit release. May we can sort out through capability or separate adapter for
> crypto case. But I think, those will be new additions to the API.So removing the
> experimental tags may be OK.
> But if you have strong opinion on keeping the experimental tag till we address
> the crypto use case then I am fine with that.
> 
> Thoughts?

Ok, agreed, no need to keep the tag for this concern. The capability idea is intriguing -- I'll chew on this and we can tackle it at a later point.

Thanks,
Gage

> 
> Jerin
> 
> 
> >
> > Thanks,
> > Gage
> >
  

-----Original Message-----
> Date: Wed, 1 Nov 2017 14:12:59 +0000
> From: "Eads, Gage" <gage.eads@intel.com>
> To: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> CC: "dev@dpdk.org" <dev@dpdk.org>, "Richardson, Bruce"
>  <bruce.richardson@intel.com>, "Van Haaren, Harry"
>  <harry.van.haaren@intel.com>, Hemant Agrawal <hemant.agrawal@nxp.com>,
>  Nipun Gupta <nipun.gupta@nxp.com>, "Rao, Nikhil" <nikhil.rao@intel.com>,
>  Pavan Nikhilesh <pbhagavatula@caviumnetworks.com>, Thomas Monjalon
>  <thomas@monjalon.net>
> Subject: RE: [dpdk-dev] [PATCH] eventdev: remove experimental label
> 
> 
> 
> > -----Original Message-----
> > From: Jerin Jacob [mailto:jerin.jacob@caviumnetworks.com]
> > Sent: Monday, October 30, 2017 12:38 PM
> > To: Eads, Gage <gage.eads@intel.com>
> > Cc: dev@dpdk.org; Richardson, Bruce <bruce.richardson@intel.com>; Van
> > Haaren, Harry <harry.van.haaren@intel.com>; Hemant Agrawal
> > <hemant.agrawal@nxp.com>; Nipun Gupta <nipun.gupta@nxp.com>; Rao,
> > Nikhil <nikhil.rao@intel.com>; Pavan Nikhilesh
> > <pbhagavatula@caviumnetworks.com>; Thomas Monjalon
> > <thomas@monjalon.net>
> > Subject: Re: [dpdk-dev] [PATCH] eventdev: remove experimental label
> > 
> > -----Original Message-----
> > > Date: Mon, 23 Oct 2017 18:27:52 +0000
> > > From: "Eads, Gage" <gage.eads@intel.com>
> > > To: Jerin Jacob <jerin.jacob@caviumnetworks.com>, "dev@dpdk.org"
> > >  <dev@dpdk.org>
> > > CC: "Richardson, Bruce" <bruce.richardson@intel.com>, "Van Haaren, Harry"
> > >  <harry.van.haaren@intel.com>, Hemant Agrawal
> > > <hemant.agrawal@nxp.com>,  Nipun Gupta <nipun.gupta@nxp.com>, "Rao,
> > > Nikhil" <nikhil.rao@intel.com>,  Pavan Nikhilesh
> > > <pbhagavatula@caviumnetworks.com>, Thomas Monjalon
> > > <thomas@monjalon.net>
> > > Subject: RE: [dpdk-dev] [PATCH] eventdev: remove experimental label
> > >
> > > Hi Jerin,
> > 
> > Hi Gage,
> > 
> > >
> > > I have one concern with the API that may delay changing the label.
> > >
> > > The implicit release that in rte_event_dequeue_burst() is a problem when using
> > asynchronous/look-aside hardware, like a cryptodev. For instance, let's say in
> > pipeline stage N the worker takes the event's mbuf and places it in a per-worker
> > crypto request queue. When the worker next calls rte_event_dequeue_burst(),
> > that function will release the previous event which could cause the flow to
> > migrate to another worker, and this could result in packet reordering.
> > >
> > > To prevent this, the worker can't call dequeue until the look-aside operation
> > completes...in effect treating the asynchronous/look-aside hardware as
> > synchronous. Another option is to feed stage N's queue to a single port to avoid
> > the flow migration, but that port may become a bottleneck.
> > >
> > > We could remove the implicit release functionality or add a port configuration
> > flag to disable it, so the default behavior is unchanged. Removing it will
> > completely will likely require changes in existing code, but it simplifies the usage
> > model (all dequeued events must be either forwarded or released) and the
> > PMD's dequeue code. This functionality could be removed from the software
> > eventdev fairly easily, but I haven't looked into the hardware PMDs.
> > 
> > 
> > 
> > The HW implementations, I know, it does the implicit release. Otherwise it
> > will result in deadlock because it cannot hold reordering metadata for
> > the longtime(SRAM is limited etc)
> > 
> > Coming back to cryptodev use case, if I understand it correctly, before
> > application enqueues to crypto queue, the application will change the tag and
> > submit to ATOMIC queue. So as long as crypto queue competes for the
> > crypto work in order then the order will be maintained.
> > 
> > In typical outbound IPSec use case,
> > - Stage 1 will be processed in ORDERED where application does the SA
> >   lookup
> > - Once SA found, application enqueue to ATOMIC stage with SA as flow_id.
> > - When the event comes from the ATOMIC queue, it in ingress order and
> >   then it submits to the crypto queue
> > - Crypto queue maintains the FIFO order.
> > - On IPSec crypto work competition, packets will come in Stage 3.
> > - So at Stage 3, packets are in ingress order for the given SA flow id.
> >                                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > 
> > Having said that, If SW implementation needs to do differently for performance
> > reasons then we will end up in capability as HW implementation works in the
> > implicit release. May we can sort out through capability or separate adapter for
> > crypto case. But I think, those will be new additions to the API.So removing the
> > experimental tags may be OK.
> > But if you have strong opinion on keeping the experimental tag till we address
> > the crypto use case then I am fine with that.
> > 
> > Thoughts?
> 
> Ok, agreed, no need to keep the tag for this concern. The capability idea is intriguing -- I'll chew on this and we can tackle it at a later point.

OK. Please add Acked-by:

> 
> Thanks,
> Gage
> 
> > 
> > Jerin
> > 
> > 
> > >
> > > Thanks,
> > > Gage
> > >
  

> -----Original Message-----
> From: Jerin Jacob [mailto:jerin.jacob@caviumnetworks.com]
> Sent: Wednesday, November 1, 2017 11:12 PM
> To: Eads, Gage <gage.eads@intel.com>
> Cc: dev@dpdk.org; Richardson, Bruce <bruce.richardson@intel.com>; Van
> Haaren, Harry <harry.van.haaren@intel.com>; Hemant Agrawal
> <hemant.agrawal@nxp.com>; Nipun Gupta <nipun.gupta@nxp.com>; Rao,
> Nikhil <nikhil.rao@intel.com>; Pavan Nikhilesh
> <pbhagavatula@caviumnetworks.com>; Thomas Monjalon
> <thomas@monjalon.net>
> Subject: Re: [dpdk-dev] [PATCH] eventdev: remove experimental label
> 
> -----Original Message-----
> > Date: Wed, 1 Nov 2017 14:12:59 +0000
> > From: "Eads, Gage" <gage.eads@intel.com>
> > To: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> > CC: "dev@dpdk.org" <dev@dpdk.org>, "Richardson, Bruce"
> >  <bruce.richardson@intel.com>, "Van Haaren, Harry"
> >  <harry.van.haaren@intel.com>, Hemant Agrawal
> > <hemant.agrawal@nxp.com>,  Nipun Gupta <nipun.gupta@nxp.com>, "Rao,
> > Nikhil" <nikhil.rao@intel.com>,  Pavan Nikhilesh
> > <pbhagavatula@caviumnetworks.com>, Thomas Monjalon
> > <thomas@monjalon.net>
> > Subject: RE: [dpdk-dev] [PATCH] eventdev: remove experimental label
> >
> >
> >
> > > -----Original Message-----
> > > From: Jerin Jacob [mailto:jerin.jacob@caviumnetworks.com]
> > > Sent: Monday, October 30, 2017 12:38 PM
> > > To: Eads, Gage <gage.eads@intel.com>
> > > Cc: dev@dpdk.org; Richardson, Bruce <bruce.richardson@intel.com>;
> > > Van Haaren, Harry <harry.van.haaren@intel.com>; Hemant Agrawal
> > > <hemant.agrawal@nxp.com>; Nipun Gupta <nipun.gupta@nxp.com>; Rao,
> > > Nikhil <nikhil.rao@intel.com>; Pavan Nikhilesh
> > > <pbhagavatula@caviumnetworks.com>; Thomas Monjalon
> > > <thomas@monjalon.net>
> > > Subject: Re: [dpdk-dev] [PATCH] eventdev: remove experimental label
> > >
> > > -----Original Message-----
> > > > Date: Mon, 23 Oct 2017 18:27:52 +0000
> > > > From: "Eads, Gage" <gage.eads@intel.com>
> > > > To: Jerin Jacob <jerin.jacob@caviumnetworks.com>, "dev@dpdk.org"
> > > >  <dev@dpdk.org>
> > > > CC: "Richardson, Bruce" <bruce.richardson@intel.com>, "Van Haaren,
> Harry"
> > > >  <harry.van.haaren@intel.com>, Hemant Agrawal
> > > > <hemant.agrawal@nxp.com>,  Nipun Gupta <nipun.gupta@nxp.com>,
> > > > "Rao, Nikhil" <nikhil.rao@intel.com>,  Pavan Nikhilesh
> > > > <pbhagavatula@caviumnetworks.com>, Thomas Monjalon
> > > > <thomas@monjalon.net>
> > > > Subject: RE: [dpdk-dev] [PATCH] eventdev: remove experimental
> > > > label
> > > >
> > > > Hi Jerin,
> > >
> > > Hi Gage,
> > >
> > > >
> > > > I have one concern with the API that may delay changing the label.
> > > >
> > > > The implicit release that in rte_event_dequeue_burst() is a
> > > > problem when using
> > > asynchronous/look-aside hardware, like a cryptodev. For instance,
> > > let's say in pipeline stage N the worker takes the event's mbuf and
> > > places it in a per-worker crypto request queue. When the worker next
> > > calls rte_event_dequeue_burst(), that function will release the
> > > previous event which could cause the flow to migrate to another worker,
> and this could result in packet reordering.
> > > >
> > > > To prevent this, the worker can't call dequeue until the
> > > > look-aside operation
> > > completes...in effect treating the asynchronous/look-aside hardware
> > > as synchronous. Another option is to feed stage N's queue to a
> > > single port to avoid the flow migration, but that port may become a
> bottleneck.
> > > >
> > > > We could remove the implicit release functionality or add a port
> > > > configuration
> > > flag to disable it, so the default behavior is unchanged. Removing
> > > it will completely will likely require changes in existing code, but
> > > it simplifies the usage model (all dequeued events must be either
> > > forwarded or released) and the PMD's dequeue code. This
> > > functionality could be removed from the software eventdev fairly easily, but
> I haven't looked into the hardware PMDs.
> > >
> > >
> > >
> > > The HW implementations, I know, it does the implicit release.
> > > Otherwise it will result in deadlock because it cannot hold
> > > reordering metadata for the longtime(SRAM is limited etc)
> > >
> > > Coming back to cryptodev use case, if I understand it correctly,
> > > before application enqueues to crypto queue, the application will
> > > change the tag and submit to ATOMIC queue. So as long as crypto
> > > queue competes for the crypto work in order then the order will be
> maintained.
> > >
> > > In typical outbound IPSec use case,
> > > - Stage 1 will be processed in ORDERED where application does the SA
> > >   lookup
> > > - Once SA found, application enqueue to ATOMIC stage with SA as flow_id.
> > > - When the event comes from the ATOMIC queue, it in ingress order and
> > >   then it submits to the crypto queue
> > > - Crypto queue maintains the FIFO order.
> > > - On IPSec crypto work competition, packets will come in Stage 3.
> > > - So at Stage 3, packets are in ingress order for the given SA flow id.
> > >
> > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > >
> > > Having said that, If SW implementation needs to do differently for
> > > performance reasons then we will end up in capability as HW
> > > implementation works in the implicit release. May we can sort out
> > > through capability or separate adapter for crypto case. But I think,
> > > those will be new additions to the API.So removing the experimental tags
> may be OK.
> > > But if you have strong opinion on keeping the experimental tag till
> > > we address the crypto use case then I am fine with that.
> > >
> > > Thoughts?
> >
> > Ok, agreed, no need to keep the tag for this concern. The capability idea is
> intriguing -- I'll chew on this and we can tackle it at a later point.
> 
> OK. Please add Acked-by:
> 

Sure.

Acked-by: Gage Eads <gage.eads@intel.com>

> >
> > Thanks,
> > Gage
> >
> > >
> > > Jerin
> > >
> > >
> > > >
> > > > Thanks,
> > > > Gage
> > > >
  

> Acked-by: Gage Eads <gage.eads@intel.com>

Acked-by: Thomas Monjalon <thomas@monjalon.net>

Applied, thanks