[dpdk-dev] Service lcores and Application lcores

[Van Haaren, Harry]

> From: Thomas Monjalon [mailto:thomas at monjalon.net]
> Sent: Thursday, June 29, 2017 4:16 PM
> To: Van Haaren, Harry <harry.van.haaren at intel.com>
> Cc: dev at dpdk.org; 'Jerin Jacob' <jerin.jacob at caviumnetworks.com>; Wiles, Keith
> <keith.wiles at intel.com>; Richardson, Bruce <bruce.richardson at intel.com>
> Subject: Re: Service lcores and Application lcores
> 
> 29/06/2017 16:36, Van Haaren, Harry:
> > The topic of discussion in this thread is how we can ensure
> > that application lcores do not interfere with service cores.
> 
> Please could you give more details on the issue?

Sure, hope I didn't write too much!


> I think you need to clearly explain the 3 types of cores:
> 	- DPDK mainloop
> 	- DPDK services
> 	- not used by DPDK


DPDK cores continue to function as they currently do, with the execption that service-cores are removed from the coremask. Details in 0) below.

DPDK service cores run services - they are not visible to the application. (AKA; the application does not perform any remote_launch() on these cores, it is handled internally in EAL). Service lcores are just normal lcores, only the
lcore_config[lcore_id].core_role == ROLE_SERVICE instead of ROLE_RTE.

Non DPDK cores are not changed.


I'll run through the following scenarios to detail the problem;

0) Explain where service cores come from in relation to non-DPDK cores
1) Describe the current usage of DPDK cores, and how the eventdev scheduler is used
2) Introduce the a service core only usage of eventdev
3) Introduce the problem: service cores and application cores concurrently run a multi-thread unsafe service
4) The proposed solution


0) At application startup, the EAL coremask detects DPDK cores, and "brings them up".
   Service cores are "stolen" from the previous core-mask, so the service-core mask is a subset of the EAL coremask.
   Service cores are marked as ROLE_SERVICE, and the application "FOR_EACH_LCORE" will not use them.
   Non-DPDK cores are not affected - they remain as they were.


1) Currently, a DPDK application directly invokes rte_eventdev_schedule() using an ordinary app lcore.
   The application is responsible for multiplexing work on cores (assuming multiple PMDs are running on one core).
   The app logic *must* be updated if we wish to add more SW workloads (aka, using a SW PMD instead of HW acceleration).
   This change in app logic is the workaround to DPDK failing to abstract HW / SW PMD requirements.
   Service cores provides the abstraction of environment (SW/HW PMD) difference to the application.


2) In a service core "only" situation, the SW PMD registers a service. This service is run on a service core.
   The application logic does not have to change, as the service-core running the service is not under app control.
   Note that the application does NOT call rte_eventdev_schedule() as in 1) above, since the service core now performs this.


3) The problem;
   If a service core runs the SW PMD schedule() function (option 2) *AND*
   the application lcore runs schedule() func (option 1), the result is that
   two threads are concurrently running a multi-thread unsafe function.

   The issue is not that the application is wrong: it correctly called rte_schedule()
   It is also not that service core infra is wrong: it correctly ran the service
   The combination of both (and the un-awareness of eachother) that causes the issue.


4) The proposed solution;
   In order to ensure that multiple threads do not concurrently run a multi-thread unsafe service function,
   the threads must be aware of runtime of the other threads. The service core code handles this
   using an atomic operation per service; multiple service cores operate correctly, no problem.
   The root cause of the issue is that the application cores are not using the service atomic.

   The rte_service_run() function, allows the application to be aware of service-core runtime habits
   due to calling into the service library, and running the service from there. With this additional rule,
   all cores (service and application owned) will be aware of eachother, and can run multi-thread unsafe
   services safely, in a co-operative manner.

   In order to allow the application core still run the eventdev PMD "manually" if it insists,
   I am proposing to allow it to invoke a specific function, which is aware of the service
   atomic. This will result in all cores "playing nice", regardless of if it is app or service owned.

   The rte_service_run() function (which allows an application-lcore to run a service) allows
   much easier porting of applications to the service-core infrastructure. It is easier because
   the threading model of the application does not have to change, it looks up the service it
   would like to run, and can repeatedly call the rte_service_run() function to have the application
   behave in the same way as before the service core addition.


Ok, this got longer than intended - but hopefully clearly articulates the motivation for the rte_service_run() concept.

Regards, -Harry