[dpdk-dev] Service lcores and Application lcores

[Van Haaren, Harry]

> From: Jerin Jacob [mailto:jerin.jacob at caviumnetworks.com]
> Sent: Friday, June 30, 2017 5:45 AM
> To: Richardson, Bruce <bruce.richardson at intel.com>
> Cc: Van Haaren, Harry <harry.van.haaren at intel.com>; dev at dpdk.org; thomas at monjalon.net;
> Wiles, Keith <keith.wiles at intel.com>
> Subject: Re: Service lcores and Application lcores
> 
> -----Original Message-----
> > Date: Thu, 29 Jun 2017 16:57:08 +0100
> > From: Bruce Richardson <bruce.richardson at intel.com>
> > To: "Van Haaren, Harry" <harry.van.haaren at intel.com>
> > CC: "dev at dpdk.org" <dev at dpdk.org>, 'Jerin Jacob'
> >  <jerin.jacob at caviumnetworks.com>, "thomas at monjalon.net"
> >  <thomas at monjalon.net>, "Wiles, Keith" <keith.wiles at intel.com>
> > Subject: Re: Service lcores and Application lcores
> > User-Agent: Mutt/1.8.1 (2017-04-11)
> >
> > On Thu, Jun 29, 2017 at 03:36:04PM +0100, Van Haaren, Harry wrote:
> > > Hi All,

<snip>

> > > A proposal for Eventdev, to ensure Service lcores and Application lcores play nice;
> > >
> > > 1) Application lcores must not directly call rte_eventdev_schedule()
> > > 2A) Service cores are the proper method to run services
> > > 2B) If an application insists on running a service "manually" on an app lcore, we
> provide a function for that:
> > >      rte_service_run_from_app_lcore(struct service *srv);
> > >
> > > The above function would allow a pesky app to run services on its own (non-service
> core) lcores, but
> > > does so through the service-core framework, allowing the service-library atomic to
> keep access serialized as required for non-multi-thread-safe services.
> > >
> > > The above solution maintains the option of running the eventdev PMD as now (single-
> core dedicated to a single service), while providing correct serialization by using the
> rte_service_run_from_app_lcore() function. Given the atomic is only used when required
> (multiple cores mapped to the service) there should be no performance delta.
> > >
> > > Given that the application should not invoke rte_eventdev_schedule(), we could even
> consider removing it from the Eventdev API. A PMD that requires cycles registers a
> service, and an application can use a service core or the run_from_app_lcore() function if
> it wishes to invoke that service on an application owned lcore.
> > >
> > >
> > > Opinions?
> >
> > I would be in favour of this proposal, except for the proposed name for
> > the new function. It would be useful for an app to be able to "adopt" a
> > service into it's main loop if so desired. If we do this, I think I'd
> 
> +1
> 
> Agree with Harry and Bruce here.
> 
> I think, The adapter function should take "struct service *" and return
> lcore_function_t so that it can run using exiting rte_eal_remote_launch()


I don't think providing a remote-launch API is actually beneficial. Remote-launching a single service
is equivalent to adding that lcore as a service-core, and mapping it to just that single service.
The advantage of adding it as a service core, is future-proofing for if more services need to be added
to that core in future, and statistics of the service core infrastructure. A convenience API could be
provided to perform the core_add(), service_start(), enable_on_service() and core_start() APIs in one.

Also, the remote_launch API doesn't solve the original problem - what if an application lcore wishes
to run one iteration of a service "manually". The remote_launch style API does not solve this problem.


Here a much simpler API to run a service... as a counter-proposal :)

/** Runs one iteration of *service* on the calling lcore */
int rte_service_iterate(struct rte_service_spec *service);


The iterate() function can check that the service is start()-ed, check the number of mapped-lcores and utilize the atomic to prevent concurrent access to multi-thread unsafe services. By exposing the function-pointer/userdata directly, we lose that.

Thinking about it, a function like rte_service_iterate() is the only functionally correct approach. (Exposing the callback directly brings us back to the "application thread without atomic check" problem.)

Thoughts?


> > also support the removal of a dedicated schedule call from the eventdev
> > API, or alternatively, if it is needed by other PMDs, leave it as a
> > no-op in the sw PMD in favour of the service-cores managed function.
> 
> I would be in favor of removing eventdev schedule and
> RTE_EVENT_DEV_CAP_DISTRIBUTED_SCHED capability so that it is completely
> transparent to application whether scheduler runs on HW or SW or "combination
> of both"


Yep this bit sounds good!