[dpdk-dev] [RFC] ethdev: make rte flow API thread safe

[Suanming Mou]

> -----Original Message-----
> From: Stephen Hemminger <stephen at networkplumber.org>
> Sent: Wednesday, September 9, 2020 12:03 AM
> To: NBU-Contact-Thomas Monjalon <thomas at monjalon.net>
> Cc: Suanming Mou <suanmingm at nvidia.com>; Ori Kam <orika at mellanox.com>;
> Matan Azrad <matan at mellanox.com>; Shahaf Shuler <shahafs at mellanox.com>;
> Viacheslav Ovsiienko <viacheslavo at mellanox.com>; Ferruh Yigit
> <ferruh.yigit at intel.com>; Andrew Rybchenko <arybchenko at solarflare.com>;
> dev at dpdk.org; joyce.kong at arm.com; phil.yang at arm.com;
> steve.capper at arm.com; honnappa.nagarahalli at arm.com
> Subject: Re: [dpdk-dev] [RFC] ethdev: make rte flow API thread safe
> 
> On Tue, 08 Sep 2020 17:03:53 +0200
> Thomas Monjalon <thomas at monjalon.net> wrote:
> 
> > 08/09/2020 16:52, Stephen Hemminger:
> > > On Mon, 7 Sep 2020 02:36:48 +0000
> > > Suanming Mou <suanmingm at nvidia.com> wrote:
> > > > > What is the performance impact of this for currently working
> > > > > applications that use a single thread to program flow rules.  You are
> adding a couple of system
> > > > > calls to what was formerly a totally usermode operation.
> > >
> > > Read the source for glibc and see what pthread_mutex does
> >
> > What would be the best lock for rte_flow?
> > We have spin lock, ticket lock, MCS lock (and rwlock) in DPDK.
> 
> The tradeoff is between speed, correctness, and simplicity.
> The flow case is performance sensitive (for connection per second tests); but
> not super critical (ie every packet).
> Fastest would be RCU but probably not necessary here.
> 
> There would rarely be contention on this (thread safety is new), and there is no
> case where reader makes sense. For hardware, programming flow rules would
> basic interaction with TCAM (ie fast). For software drivers, typically making flow
> rule requires system call to set classifier etc. Holding spin lock across system
> calls leads to preemption and other issues.

Very good explanation, thank you Stephen. Totally agree with that.

> 
> Would it be possible to push the choice of mutual exclusion down to the device
> driver? For fast HW devices they could use spinlock and for slow SW devices it
> would be pthread.

That's a good hint. But that will also introduce the vendor PMD to update. I tried to list some points here.
1. For single thread case, pthread mutex acts similar as spinlock, spinlock or pthread mutex will not make difference here. 
2. For multiple threads lock contention, spinlock will introduce more CPU usage. DPDK applications currently use mutex later get rid of the outer mutex will suffer higher CPU usage with the spinlock(fast hardware PMD).

And one more general question, can we find some DPDK applications now use spinlock with the rte flow APIs?