[dpdk-dev] [RFC] Service Cores concept

[Bruce Richardson]

On Wed, May 17, 2017 at 01:28:14PM +0200, Thomas Monjalon wrote:
> 17/05/2017 12:32, Bruce Richardson:
> > On Wed, May 17, 2017 at 12:11:10AM +0200, Thomas Monjalon wrote:
> > > 03/05/2017 13:29, Harry van Haaren:
> > > > The concept is to allow a software function register itself with EAL as
> > > > a "service", which requires CPU time to perform its duties. Multiple
> > > > services can be registered in an application, if more than one service
> > > > exists. The application can retrieve a list of services, and decide how
> > > > many "service cores" to use. The number of service cores is removed
> > > > from the application usage, and they are mapped to services based on
> > > > an application supplied coremask.
> > > > 
> > > > The application now continues as normal, without having to manually
> > > > schedule and implement arbitration of CPU time for the SW services.
> > > 
> > > I think it should not be the DPDK responsibility to schedule threads.
> > > The mainloops and scheduling are application design choices.
> > > 
> > > If I understand well the idea of your proposal, it is a helper for
> > > the application to configure the thread scheduling of known services.
> > > So I think we could add interrupt processing and other thread creations
> > > in this concept.
> > > Could we also embed the rte_eal_mp_remote_launch() calls in this concept?
> > 
> > 
> > There are a couple of parts of this:
> > 1. Allowing libraries and drivers to register the fact that they require
> > background processing, e.g. as a SW fallback for functionality that
> > would otherwise be implemented in hardware
> > 2. Providing support for easily multi-plexing these independent
> > functions from different libs onto a different core, compared to the
> > normal operation of DPDK of firing a single run-forever function on each
> > core.
> > 3. Providing support for the application to configure the running of
> > these background services on specific cores.
> > 4. Once configured, hiding these services and the cores they run on from
> > the rest of the application, so that the rest of the app logic does not
> > need to change depending on whether service cores are in use or not. For
> > instance, removing the service cores from the core list in foreach-lcore
> > loops, and preventing the EAL from trying to run app functions on the
> > cores when the app calls mp_remote_launch.
> > 
> > Overall, the objective is to provide us a way to have software
> > equivalents of hardware functions in as transparent a manner as
> > possible. There is a certain amount of scheduling being done by the
> > DPDK, but it is still very much under the control of the app.
> > 
> > As for other things being able to use this concept, definite +1 for
> > interrupt threads and similar. I would not see mp_remote_launch as being
> > affected here in any significant way (except from the hiding service
> > cores from it, obviously)
> 
> OK to register CPU needs for services (including interrupts processing).
> 
> Then we could take this opportunity to review how threads are managed.
> We will have three types of cores:
> - not used
> - reserved for services
> - used for polling / application processing
> It is fine to reserve/define CPU from DPDK point of view.
> 
> Then DPDK launch threads on cores. Maybe we should allow the application
> to choose how threads are launched and managed.
> Keith was talking about a plugin approach for thread management I think.

For thread management, I'd view us as extending what we have with some
EAL APIs rather than replacing what is there already. What I think we
could with would be APIs to:
* spawn an additional thread on a core i.e. add a bit to the coremask
* shutdown a thread on a core i.e. remove a bit from the coremask
* register an existing thread with DPDK, i.e. give it an lcore_id
  internally so that it can use DPDK data structures as a first-class
  citizen.

However, while needed, this is separate work from the service cores concept.

Regards,
/Bruce