[dpdk-dev] [PATCH] eal: support using 0 as coremask for no-affinitization

[Van Haaren, Harry]

> -----Original Message-----
> From: Burakov, Anatoly <anatoly.burakov at intel.com>
> Sent: Wednesday, February 17, 2021 12:09 PM
> To: Van Haaren, Harry <harry.van.haaren at intel.com>; Richardson, Bruce
> <bruce.richardson at intel.com>
> Cc: dev at dpdk.org
> Subject: Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no-
> affinitization
> 
> On 16-Feb-21 5:44 PM, Van Haaren, Harry wrote:
> >> -----Original Message-----
> >> From: Bruce Richardson <bruce.richardson at intel.com>
> >> Sent: Tuesday, February 16, 2021 5:31 PM
> >> To: Van Haaren, Harry <harry.van.haaren at intel.com>
> >> Cc: Burakov, Anatoly <anatoly.burakov at intel.com>; dev at dpdk.org
> >> Subject: Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no-
> >> affinitization
> >>
> >> On Tue, Feb 16, 2021 at 05:22:25PM +0000, Van Haaren, Harry wrote:
> >>>> -----Original Message-----
> >>>> From: Burakov, Anatoly <anatoly.burakov at intel.com>
> >>>> Sent: Tuesday, February 16, 2021 10:53 AM
> >>>> To: Richardson, Bruce <bruce.richardson at intel.com>; Van Haaren, Harry
> >>>> <harry.van.haaren at intel.com>
> >>>> Cc: dev at dpdk.org
> >>>> Subject: Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no-
> >>>> affinitization
> >>>>
> >>>> On 16-Feb-21 10:46 AM, Bruce Richardson wrote:
> >>>>> On Tue, Feb 16, 2021 at 10:36:13AM +0000, Burakov, Anatoly wrote:
> >>>>>> On 16-Feb-21 9:43 AM, Bruce Richardson wrote:
> >>>>>>> Allow the user to specify that they don't want any core pinning from
> >> DPDK
> >>>>>>> by passing in the coremask of 0.
> >>>>>>> ---
> >>>>>>
> >>>>>> I haven't checked what happens yet, but down the line we also set
> affinity
> >>>>>> for service cores as well as interrupt thread. what would be the
> semantics
> >>>>>> of those in this particular case? do we want the same ability for service
> >>>>>> cores (i.e. pick a non-affinitized core)? And where does interrupt thread
> >>>>>> affinitize in this case (presumably, nowhere too)?
> >>>>>>
> >>>>> I have not checked the service core setup, because a) I forgot about them
> >>>>> and b) I'm not sure how their affinity rules work with respect to the main
> >>>>> lcore mask. On the other hand I did check out that the lcore mask for all
> >>>>> non-pinned threads, or control threads, is the full set of bits as
> >>>>> expected.
> >>>>>
> >>>>> /Bruce
> >>>>>
> >>>>
> >>>> +Harry,
> >>>>
> >>>> I believe service core mask must not overlap with lcore masks, so
> >>>> presumably using 0 as lcore mask would make it so that any service core
> >>>> mask will be valid (which is presumably what we want?).
> >>>
> >>> Services cores -S list or -s <mask> *must* overlap with the RTE lcores, EAL
> >>> then"steals" the service cores from the application lcores, code that
> >> implements here:
> >>> http://git.dpdk.org/dpdk-
> >> stable/tree/lib/librte_eal/common/eal_common_options.c?h=20.11#n657
> >>>
> >>>> Should service cores also have a "just pick a core" parameter?
> >>>
> >>> I'm not sure, depends on what the bigger goal is here.
> >>> Assuming we're enabling this for ROLE_RTE threads, then
> >>> it would seem to me that ROLE_SERVICE and control threads
> >>> would require similar treatment?
> >>>
> >> Control threads are affinitised to all cores not in the coremask, which
> >> means in this case that they can run anywhere on the system the OS chooses.
> >
> > Ah ok, fair enough yes.
> >
> >> In case of service cores, it would seem that using service cores with an
> >> empty coremask is just not compatible. I would assume that this
> >> incompatibility already exists when one has a coremask with only one core
> >> already in it.
> >
> > Yes, correct, it would leave zero lcores for ROLE_RTE, meaning no lcores for the
> application.
> > A possible solution would be to special case a zero service core mask and apply
> the same
> > treatment as ROLE_RTE coremask?
> >
> > Others likely have better ideas - I don't have time to follow DPDK
> threading/pinning topic
> > closely at the moment.
> >
> 
> I don't think it's a good idea to disallow service cores functionality
> in this case, but i don't have a way to solve this, other than
> implementing similar 0x0 coremask for service cores and assume it always
> means "one core affinitized to wherever the OS feels like it". After
> all, with lcore mask 0x0 we assume user wants one single core only, so
> following that, one single service core is a valid extrapolation IMO.

OK with me - seems reasonable.

> Perhaps specifying the number of l/s cores when using 0x0 would be
> interesting, but IMO unless there's ask for it, i'd rather not
> overcomplicate things and go with similar semantics for service cores,
> and just allow a 0x0 coremask that means only one unaffinitized service
> core will be created.
> 
> Thoughts?

Agree with keeping-it-simple if possible, and agree that unaffinitized with
a single service-core with a 0x0 mask makes sense.

Most important to me is to maintain backward compatibility with existing
usage of -S and -s, but this shouldn't break anything? (Famous last words..)

> --
> Thanks,
> Anatoly