[dpdk-dev] [PATCH v13 09/10] eal: add EAL argument for setting thread priority

[Stephen Hemminger]

On Thu, 19 Aug 2021 14:30:19 -0700
Narcisa Ana Maria Vasile <navasile at linux.microsoft.com> wrote:

> On Thu, Aug 19, 2021 at 10:06:06AM +0100, Bruce Richardson wrote:
> > On Wed, Aug 18, 2021 at 02:28:33PM -0700, Stephen Hemminger wrote:  
> > > On Tue,  3 Aug 2021 12:01:30 -0700 Narcisa Ana Maria Vasile
> > > <navasile at linux.microsoft.com> wrote:
> > >   
> > > > +static int +eal_parse_thread_priority(const char *arg) +{ +
> > > > struct internal_config *internal_conf = +
> > > > eal_get_internal_configuration(); +	enum rte_thread_priority priority;
> > > > + +	if (!strncmp("normal", arg, sizeof("normal"))) +
> > > > priority = RTE_THREAD_PRIORITY_NORMAL; +	else if
> > > > (!strncmp("realtime", arg, sizeof("realtime"))) +		priority =
> > > > RTE_THREAD_PRIORITY_REALTIME_CRITICAL; +	else +		return -1;
> > > > + +	internal_conf->thread_priority = priority; +	return 0; +} +  
> > > 
> > > In my experience using real time priority is dangerous and risks
> > > starvation and deadlock. The problem is that DPDK applications are
> > > typically 100% CPU poll mode with no system calls; but the kernel has a
> > > number of worker threads that can be required on those CPUs.
> > > 
> > > The typical failure is a disk completion interrupt happens on a CPU that
> > > is being used by DPDK lcore thread. With RT priority, the kernel thread
> > > to process that I/O completion never runs because the RT user thread has
> > > higher priority than the kernel I/O completion thread.
> > > 
> > > It maybe possible to workaround this with lots of hand crafting through
> > > sysfs to reassign threads and irq's. Also, later kernels with full RT
> > > might also help.
> > > 
> > > Before putting this in as part of DPDK in EAL, a full set of testing and
> > > documentation of how to configure these kind of applications and systems
> > > is needed.
> > >  
> > I would tend to agree caution here, based on my experience of having locked
> > up a number of systems in the past when testing running DPDK apps with RT
> > priority!  
> 
> Thank you for the comments! I've added this option since it was requested by
> multiple users. I understand RT priority causes issues on Linux platforms.
> On Windows we want to be able to use REALTIME priority in certain scenarios.
> 
> Would it be acceptable to replace this option with a "HIGH_PRIORITY" one
> and keep it realtime on Windows and choose a higher (but non-realtime) option on Linux?
> However, there are 2 issues here:
>  * We will have different behaviors between the 2 platforms.
>  * Not sure if I can set a normal but higher priority on Linux. SCHED_OTHER only allows
>    one value and Linux "nice" values don't help. If anyone knows of a way to accomplish
>    this on Linux, please do advise.
> Alternatively, we can have this option for Windows only.
> 
> In the meantime, I've removed this patch from this patchset in v14 as the cmdline option is not
> being enabled yet, as DmitryK noted.
> 
> 

I think on Linux it should produce a big a*** warning message.