[dpdk-dev] [RFCv2] service core concept

[Van Haaren, Harry]

> -----Original Message-----
> From: Ananyev, Konstantin
> Sent: Wednesday, June 7, 2017 10:51 AM
> To: Van Haaren, Harry <harry.van.haaren at intel.com>; Richardson, Bruce
> <bruce.richardson at intel.com>
> Cc: dev at dpdk.org; Thomas Monjalon <thomas at monjalon.net>; Jerin Jacob
> <jerin.jacob at caviumnetworks.com>; Wiles, Keith <keith.wiles at intel.com>
> Subject: RE: [dpdk-dev] [RFCv2] service core concept
> 
> 
> 
> > -----Original Message-----
> > From: Van Haaren, Harry
> > Sent: Tuesday, June 6, 2017 4:41 PM
> > To: Ananyev, Konstantin <konstantin.ananyev at intel.com>; Richardson, Bruce
> <bruce.richardson at intel.com>
> > Cc: dev at dpdk.org; Thomas Monjalon <thomas at monjalon.net>; Jerin Jacob
> <jerin.jacob at caviumnetworks.com>; Wiles, Keith
> > <keith.wiles at intel.com>
> > Subject: RE: [dpdk-dev] [RFCv2] service core concept
> >
> > > From: Ananyev, Konstantin
> > > Sent: Tuesday, June 6, 2017 4:29 PM
> > > Subject: RE: [dpdk-dev] [RFCv2] service core concept
> > >
> > >
> > > > From: Richardson, Bruce
> > > > Sent: Tuesday, June 6, 2017 3:54 PM
> > > >
> > > > On Tue, Jun 06, 2017 at 11:25:57AM +0100, Van Haaren, Harry wrote:
> > > > > > From: Ananyev, Konstantin
> > > > > > Sent: Saturday, June 3, 2017 11:23 AM
> > > > >
> > > > > <snip>
> >
> > <snip other discussion>
> >
> > > > >
> > > > > There are a number of options here, each with its own merit:
> > > > >
> > > > > A) Services/cores config in EAL
> > > > > Benefit is that service functionality can be transparent to the application. Negative
> is
> > > that the complexity is in EAL.
> > > > >
> > > > > B) Application configures services/cores
> > > > > Benefit is no added EAL complexity. Negative is that application code has to configure
> > > cores (duplicated per application).
> > > > >
> > > > >
> > > > > To answer this question, I think we need to estimate how many applications would
> benefit
> > > from EAL integration and balance that against
> > > > the "complexity cost" of doing so. I do like the simplicity of option (B), however if
> there
> > > is significant value in total transparency to the
> > > > application I think (A) is the better choice.
> > > > >
> > > > >
> > > > > Input on A) or B) welcomed! -Harry
> > > >
> > > > I'm definitely in favour of having it in EAL. The whole reason for doing
> > > > this work is to make it easy for applications to dedicate cores to
> > > > background tasks - including applications written before this
> > > > functionality was added. By merging this into EAL, we can have
> > > > transparency in the app, as we can have the service cores completely in
> > > > the background, and the app can call rte_eal_mp_remote_launch() exactly
> > > > as before, without unexpected failures. If we move this externally, the
> > > > app needs to be reworked to take account of that fact, and call new,
> > > > service-core aware, launch functions instead.
> > >
> > > Not sure I understood you here:
> > > If the app don' plan to use any cores for services, it for sure will be able to call
> > > rte_eal_mp_remote_launch() as before (no services running case).
> >
> > Correct - EAL behavior remains unchanged if --service-cores=0xf is not passed
> >
> >
> > > From other side, if the app would like to use services - it would need to specify
> > > which service it wants to run, and for each service provide a coremask, even if
> > > EAL already allocates service cores for it.
> >
> > See next paragraph
> >
> >
> > > Or are you talking about the when EAL allocates service cores, and then
> > > PMDs themselves (or EAL again) register their services on that cores?
> >
> > EAL could provide sane default behavior. For example, round-robin services over available
> service-cores. Multithread-capable services can
> > be registered on all service cores. Its not a perfect solution for all service-to-core
> mapping problems, but I'd guess about 80% of cases
> > would be covered: using a single service with a single service core dedicated to it :)
> >
> >
> > > That's probably possible, but how PMD would know which service core(s) it allowed to use?
> >
> > The PMD shouldn't be deciding - EAL for basic sanity config, or Application for advanced
> usage.
> >
> >
> > > Things might get over-complicated here - in theory there could be multiple PMDs,
> > > each of them can have more than one service, running on multiple sets of cores, etc.
> >
> > True - the NxM service:core mapping possibility can be huge - the API allows the application
> the flexibility if that flexibility is really required.
> > If the flexibility is not required, the round-robin 1:1 service:core EAL scheme should cover
> it?
> 
> Ok, so if I understand you right: by default EAL will allow each PMD to register it's services
> on all available service cores?

Close, but I don't see the PMD being involved in core mapping. I think of it like this:

1) A PMD registers its service (unaware of number of service cores available)
2) EAL provides default core to service mappings
2.1) Application configures using API for advanced uses (optional)


Worked examples of EAL default core mapping with two services registered:
- Eventdev SW PMD
- Ethdev to eventdev RX

Example A) With cores >= services, the services get one core assigned each:
./dpdk-app --service-cores=0x3
eventdev_sw0            : lcore 0
ethdev_to_eventdev_rx0  : lcore 1

Example B) With more services than cores, the services share the available cores: 
./dpdk-app --service-cores=0x1
eventdev_sw0            : lcore 0
ethdev_to_eventdev_rx0  : lcore 0


The EAL core-mapping logic round-robins services onto cores. If there are more cores than services, they are not used (and a warning print given). If there are more services than cores, they are wrapped back to the first core, and services share the core (example B).

Keep in mind this is just for simple use-cases. For complex services to cores mappings the application has the service API to configure it precisely as it wishes.