[dpdk-dev] [RFC 0/3] tqs: add thread quiescent state library

[Mattias Rönnblom]

On 2018-11-30 03:13, Honnappa Nagarahalli wrote:
>>
>> Reinventing RCU is not helping anyone.
> IMO, this depends on what the rte_tqs has to offer and what the requirements are. Before starting this patch, I looked at the liburcu APIs. I have to say, fairly quickly (no offense) I concluded that this does not address DPDK's needs. I took a deeper look at the APIs/code in the past day and I still concluded the same. My partial analysis (analysis of more APIs can be done, I do not have cycles at this point) is as follows:
> 
> The reader threads' information is maintained in a linked list[1]. This linked list is protected by a mutex lock[2]. Any additions/deletions/traversals of this list are blocking and cannot happen in parallel.
> 
> The API, 'synchronize_rcu' [3] (similar functionality to rte_tqs_check call) is a blocking call. There is no option provided to make it non-blocking. The writer spins cycles while waiting for the grace period to get over.
> 

Wouldn't the options be call_rcu, which rarely blocks, or defer_rcu() 
which never? Why would the average application want to wait for the 
grace period to be over anyway?

> 'synchronize_rcu' also has grace period lock [4]. If I have multiple writers running on data plane threads, I cannot call this API to reclaim the memory in the worker threads as it will block other worker threads. This means, there is an extra thread required (on the control plane?) which does garbage collection and a method to push the pointers from worker threads to the garbage collection thread. This also means the time duration from delete to free increases putting pressure on amount of memory held up.
> Since this API cannot be called concurrently by multiple writers, each writer has to wait for other writer's grace period to get over (i.e. multiple writer threads cannot overlap their grace periods).

"Real" DPDK applications typically have to interact with the outside 
world using interfaces beyond DPDK packet I/O, and this is best done via 
an intermediate "control plane" thread running in the DPDK application. 
Typically, this thread would also be the RCU writer and "garbage 
collector", I would say.

> 
> This API also has to traverse the linked list which is not very well suited for calling on data plane.
> 
> I have not gone too much into rcu_thread_offline[5] API. This again needs to be used in worker cores and does not look to be very optimal.
> 
> I have glanced at rcu_quiescent_state [6], it wakes up the thread calling 'synchronize_rcu' which seems good amount of code for the data plane.
> 

Wouldn't the typical DPDK lcore worker call rcu_quiescent_state() after 
processing a burst of packets? If so, I would more lean toward 
"negligible overhead", than "a good amount of code".

I must admit I didn't look at your library in detail, but I must still 
ask: if TQS is basically RCU, why isn't it called RCU? And why isn't the 
API calls named in a similar manner?