[dpdk-dev] rte_mempools / rte_rings thread safe?

[Olivier MATZ]

Hi Jyotiswarup,

On 12/28/2013 03:55 PM, Jyotiswarup Raiturkar wrote:
 > The rte_mempool and rte_ring libs have
 > multi-producer/multi-consumer versions. But it's also mentioned
 > in the header files that the implementation is not
 > pre-emtable : " Note: the mempool implementation is not
 > preemptable. A lcore must not be interrupted by another task
 > that uses the same mempool (because it uses a ring which is not
 > preemptable)"

Correct.

 > - Does having mutually exclusive core masks for a set of
 > threads which use the ring/mempool suffice for thread safety
 > (threads will have different core ids but they will not be
 > pinned to cores) ?

As you noticed, and as explained in [1], one problem with the mempool
is related to the cache structure, which is table indexed by the
lcore_id. This cache structure is accessed without lock as it is
supposed that each pthread has its own lcore_id. So having different
lcore_ids would solve this problem even if the pthreads are not pinned
to different cores.

But this is not the only problem. One design paradigm of the dpdk
is that the execution units are never interrupted. Let's imagine the
following case, similar to what you describe:

- 2 pthreads are running on the same cpu (taskset on with only one
   bit set in the mask). Their respective lcore_id (internal variable)
   are 0 and 1.
- pthread 0 takes a spinlock
- the kernel interrupts pthread 0, and schedules pthread 1
- pthread 1 wants to take the same spinlock, but it cannot be
   released by pthread 0 as it is not running
- pthread 1 will loop using 100% of the cpu for nothing

The same can occur in rte_ring. If a pthread is interrupted
in an enqueue process, between lines 392 and 414 (see [2]), it would
prevent any other pthread to enqueue another object.

So, as you can see, doing this could lead to significant performance
issues.

[1] http://www.dpdk.org/ml/archives/dev/2013-December/001002.html

[2] 
http://dpdk.org/browse/dpdk/tree/lib/librte_ring/rte_ring.h?id=142dfe1eedb215bd2a0762afcc65ef5a7fba10aa

 > - If i want to use this data structures in a pthread ( created
 > outside of DPDK environment), is it ok to use this if i
 > do "RTE_DEFINE_PER_LCORE(unsigned, _lcore_id);" with a core_id
 > exclusive of all other core masks for DPDK processes.

The short answer is no.

However, it is possible to use a rte_ring between an external process
and a dpdk application at the following conditions:

(on the external app, the pthreads must be pinned to a cpu
   OR
  on the external app the ring must be single consumer/producer)
   AND
(external apps must do enqueues only and dpdk app dequeues only
   OR
  external apps must do dequeues only and dpdk app enqueues only)

Regards,
Olivier