[dpdk-dev] [RFCv2] service core concept

[Ananyev, Konstantin]

> -----Original Message-----
> From: Van Haaren, Harry
> Sent: Wednesday, June 7, 2017 11:30 AM
> To: Ananyev, Konstantin <konstantin.ananyev at intel.com>; Richardson, Bruce <bruce.richardson at intel.com>
> Cc: dev at dpdk.org; Thomas Monjalon <thomas at monjalon.net>; Jerin Jacob <jerin.jacob at caviumnetworks.com>; Wiles, Keith
> <keith.wiles at intel.com>
> Subject: RE: [dpdk-dev] [RFCv2] service core concept
> 
> 
> 
> > -----Original Message-----
> > From: Ananyev, Konstantin
> > Sent: Wednesday, June 7, 2017 10:51 AM
> > To: Van Haaren, Harry <harry.van.haaren at intel.com>; Richardson, Bruce
> > <bruce.richardson at intel.com>
> > Cc: dev at dpdk.org; Thomas Monjalon <thomas at monjalon.net>; Jerin Jacob
> > <jerin.jacob at caviumnetworks.com>; Wiles, Keith <keith.wiles at intel.com>
> > Subject: RE: [dpdk-dev] [RFCv2] service core concept
> >
> >
> >
> > > -----Original Message-----
> > > From: Van Haaren, Harry
> > > Sent: Tuesday, June 6, 2017 4:41 PM
> > > To: Ananyev, Konstantin <konstantin.ananyev at intel.com>; Richardson, Bruce
> > <bruce.richardson at intel.com>
> > > Cc: dev at dpdk.org; Thomas Monjalon <thomas at monjalon.net>; Jerin Jacob
> > <jerin.jacob at caviumnetworks.com>; Wiles, Keith
> > > <keith.wiles at intel.com>
> > > Subject: RE: [dpdk-dev] [RFCv2] service core concept
> > >
> > > > From: Ananyev, Konstantin
> > > > Sent: Tuesday, June 6, 2017 4:29 PM
> > > > Subject: RE: [dpdk-dev] [RFCv2] service core concept
> > > >
> > > >
> > > > > From: Richardson, Bruce
> > > > > Sent: Tuesday, June 6, 2017 3:54 PM
> > > > >
> > > > > On Tue, Jun 06, 2017 at 11:25:57AM +0100, Van Haaren, Harry wrote:
> > > > > > > From: Ananyev, Konstantin
> > > > > > > Sent: Saturday, June 3, 2017 11:23 AM
> > > > > >
> > > > > > <snip>
> > >
> > > <snip other discussion>
> > >
> > > > > >
> > > > > > There are a number of options here, each with its own merit:
> > > > > >
> > > > > > A) Services/cores config in EAL
> > > > > > Benefit is that service functionality can be transparent to the application. Negative
> > is
> > > > that the complexity is in EAL.
> > > > > >
> > > > > > B) Application configures services/cores
> > > > > > Benefit is no added EAL complexity. Negative is that application code has to configure
> > > > cores (duplicated per application).
> > > > > >
> > > > > >
> > > > > > To answer this question, I think we need to estimate how many applications would
> > benefit
> > > > from EAL integration and balance that against
> > > > > the "complexity cost" of doing so. I do like the simplicity of option (B), however if
> > there
> > > > is significant value in total transparency to the
> > > > > application I think (A) is the better choice.
> > > > > >
> > > > > >
> > > > > > Input on A) or B) welcomed! -Harry
> > > > >
> > > > > I'm definitely in favour of having it in EAL. The whole reason for doing
> > > > > this work is to make it easy for applications to dedicate cores to
> > > > > background tasks - including applications written before this
> > > > > functionality was added. By merging this into EAL, we can have
> > > > > transparency in the app, as we can have the service cores completely in
> > > > > the background, and the app can call rte_eal_mp_remote_launch() exactly
> > > > > as before, without unexpected failures. If we move this externally, the
> > > > > app needs to be reworked to take account of that fact, and call new,
> > > > > service-core aware, launch functions instead.
> > > >
> > > > Not sure I understood you here:
> > > > If the app don' plan to use any cores for services, it for sure will be able to call
> > > > rte_eal_mp_remote_launch() as before (no services running case).
> > >
> > > Correct - EAL behavior remains unchanged if --service-cores=0xf is not passed
> > >
> > >
> > > > From other side, if the app would like to use services - it would need to specify
> > > > which service it wants to run, and for each service provide a coremask, even if
> > > > EAL already allocates service cores for it.
> > >
> > > See next paragraph
> > >
> > >
> > > > Or are you talking about the when EAL allocates service cores, and then
> > > > PMDs themselves (or EAL again) register their services on that cores?
> > >
> > > EAL could provide sane default behavior. For example, round-robin services over available
> > service-cores. Multithread-capable services can
> > > be registered on all service cores. Its not a perfect solution for all service-to-core
> > mapping problems, but I'd guess about 80% of cases
> > > would be covered: using a single service with a single service core dedicated to it :)
> > >
> > >
> > > > That's probably possible, but how PMD would know which service core(s) it allowed to use?
> > >
> > > The PMD shouldn't be deciding - EAL for basic sanity config, or Application for advanced
> > usage.
> > >
> > >
> > > > Things might get over-complicated here - in theory there could be multiple PMDs,
> > > > each of them can have more than one service, running on multiple sets of cores, etc.
> > >
> > > True - the NxM service:core mapping possibility can be huge - the API allows the application
> > the flexibility if that flexibility is really required.
> > > If the flexibility is not required, the round-robin 1:1 service:core EAL scheme should cover
> > it?
> >
> > Ok, so if I understand you right: by default EAL will allow each PMD to register it's services
> > on all available service cores?
> 
> Close, but I don't see the PMD being involved in core mapping. I think of it like this:
> 
> 1) A PMD registers its service (unaware of number of service cores available)
> 2) EAL provides default core to service mappings
> 2.1) Application configures using API for advanced uses (optional)

Ok, thanks for explanation.
I am still not quite happy with the fact that EAL will have a dependency on service lib,
but I see your point and indeed it might be usefull for many cases.
So wouldn't object here.
Konstantin

> 
> 
> Worked examples of EAL default core mapping with two services registered:
> - Eventdev SW PMD
> - Ethdev to eventdev RX
> 
> Example A) With cores >= services, the services get one core assigned each:
> ./dpdk-app --service-cores=0x3
> eventdev_sw0            : lcore 0
> ethdev_to_eventdev_rx0  : lcore 1
> 
> Example B) With more services than cores, the services share the available cores:
> ./dpdk-app --service-cores=0x1
> eventdev_sw0            : lcore 0
> ethdev_to_eventdev_rx0  : lcore 0
> 
> 
> The EAL core-mapping logic round-robins services onto cores. If there are more cores than services, they are not used (and a warning print
> given). If there are more services than cores, they are wrapped back to the first core, and services share the core (example B).
> 
> Keep in mind this is just for simple use-cases. For complex services to cores mappings the application has the service API to configure it
> precisely as it wishes.