[dpdk-dev] Question about cache_size in rte_mempool_create

[Bruce Richardson]

On Fri, Nov 24, 2017 at 11:39:54AM +0200, roy wrote:
> Thanks for your answer, but I cannot understand the dimension of the ring
> and it is affected by the cache size.
> 
> On 24/11/17 11:30, Bruce Richardson wrote:
> > On Thu, Nov 23, 2017 at 11:05:11PM +0200, Roy Shterman wrote:
> > > Hi,
> > > 
> > > In the documentation it says that:
> > > 
> > >   * @param cache_size
> > >   *   If cache_size is non-zero, the rte_mempool library will try to
> > >   *   limit the accesses to the common lockless pool, by maintaining a
> > >   *   per-lcore object cache. This argument must be lower or equal to
> > >   *   CONFIG_RTE_MEMPOOL_CACHE_MAX_SIZE and n / 1.5.* It is advised to
> > > choose*
> > > * *   cache_size to have "n modulo cache_size == 0": if this is*
> > > * *   not the case, some elements will always stay in the pool and will*
> > > * *   never be used.* The access to the per-lcore table is of course
> > >   *   faster than the multi-producer/consumer pool. The cache can be
> > >   *   disabled if the cache_size argument is set to 0; it can be useful to
> > >   *   avoid losing objects in cache.
> > > 
> > > I wonder if someone can please explain the high-lightened sentence, how the
> > > cache size affects the objects inside the ring.
> > It has no effects upon the objects themselves. Having a cache is
> > strongly recommended for performance reasons. Accessing a shared ring
> > for a mempool is very slow compared to pulling packets from a per-core
> > cache. To test this you can run testpmd using different --mbcache
> > parameters.
> Still, I didn't understand the sentence from above:
> 
> *It is advised to choose cache_size to have "n modulo cache_size == 0": if
> this is* not the case, some elements will always stay in the pool and will*
> never be used.*
>

This would be an artifact of the way in which the elements are taken
from the pool ring. If a full cache-size burst of elements is not
available in the ring, no elements from the ring are put in the cache.
It just means that the pool can never fully be emptied. However, in most
cases, even having the pool nearly empty indicates a problem, so
practically I wouldn't be worried about this.

> > 
> > > And also does it mean that
> > > if I'm sharing pool between different cores can it be that a core sees the
> > > pool as empty although it has objects in it?
> > > 
> > Yes, that can occur. You need to dimension the pool to take account of
> > your cache usage.
> 
> can you please elaborate more on this issue? I'm working with multi-consumer
> multi-producer pools, in my understanding object can or in lcore X cache or
> in ring.
> Each core when looking for objects in pool (ring) is looking at prod/cons
> head/tail so how can it be that the cache of different cores affects this?
> 

Mempool elements in the caches are free elements that are available for
use. However, they are inaccessible to any core except the core which
freed them to that cache. For a slightly simplified example, consider a
pool with 256 elements, and a cache size of 64. If 4 cores all request 1
buffer each, those four cores will each fill their caches and then take
1 element from those caches. This means that the ring will be empty even
though only 4 buffers are actually in use - the other 63*4 buffers are
in per-core caches. A 5th core which comes along and requests a buffer
will be told that the pool is empty, despite there being 252 free
elements.

Therefore you need to take account of the possibilities of buffers being
stored in caches when dimensioning your mempool.

/Bruce