[dpdk-dev] [PATCH v4 0/3] external mempool manager

Olivier Matz olivier.matz at 6wind.com
Thu Apr 14 15:57:47 CEST 2016


Here's a reworked version of the patch initially sent by David Hunt.
The main change is that it is rebased on top of the "mempool: rework
memory allocation" series [1], which simplifies a lot the first patch.

[1] http://dpdk.org/ml/archives/dev/2016-April/037464.html

v4 changes:
 * remove the rte_mempool_create_ext() function. To change the handler, the
   user has to do the following:
   - mp = rte_mempool_create_empty()
   - rte_mempool_set_handler(mp, "my_handler")
   - rte_mempool_populate_default(mp)
   This avoids to add another function with more than 10 arguments, duplicating
   the doxygen comments
 * change the api of rte_mempool_alloc_t: only the mempool pointer is required
   as all information is available in it
 * change the api of rte_mempool_free_t: remove return value
 * move inline wrapper functions from the .c to the .h (else they won't be
   inlined). This implies to have one header file (rte_mempool.h), or it
   would have generate cross dependencies issues.
 * remove now unused MEMPOOL_F_INT_HANDLER (note: it was misused anyway due
   to the use of && instead of &)
 * fix build in debug mode (__MEMPOOL_STAT_ADD(mp, put_pool, n) remaining)
 * fix build with shared libraries (global handler has to be declared in
   the .map file)
 * rationalize #include order
 * remove unused function rte_mempool_get_handler_name()
 * rename some structures, fields, functions
 * remove the static in front of rte_tailq_elem rte_mempool_tailq (comment
   from Yuanhan)
 * test the ext mempool handler in the same file than standard mempool tests,
   avoiding to duplicate the code
 * rework the custom handler in mempool_test
 * rework a bit the patch selecting default mbuf pool handler
 * fix some doxygen comments

Things that should still be discussed:

- Panu pointed out that having a compile-time configuration
  option for selecting the default mbuf handler is not a good idea.
  I mostly agree, except in one case (and that's why I kept this patch):
  if a specific architecture has its own way to provide an efficient
  pool handler for mbufs, it could be the proper place to have this
  option. But as far as I know, there is no such architecture today
  in dpdk.

- The other question I would like to raise is about the use cases.
  The cover letter below could be a bit more explicit about what this
  feature will be used for.



This is the initial unmodified cover letter from David Hunt:

Hi list.

Here's the v3 version patch for an external mempool manager

v3 changes:
 * simplified the file layout, renamed to rte_mempool_handler.[hc]
 * moved the default handlers into rte_mempool_default.c
 * moved the example handler out into app/test/test_ext_mempool.c
 * removed is_mc/is_mp change, slight perf degredation on sp cached operation
 * removed stack hanler, may re-introduce at a later date
 * Changes out of code reviews

v2 changes:
 * There was a lot of duplicate code between rte_mempool_xmem_create and
   rte_mempool_create_ext. This has now been refactored and is now
   hopefully cleaner.
 * The RTE_NEXT_ABI define is now used to allow building of the library
   in a format that is compatible with binaries built against previous
   versions of DPDK.
 * Changes out of code reviews. Hopefully I've got most of them included.

The External Mempool Manager is an extension to the mempool API that allows
users to add and use an external mempool manager, which allows external memory
subsystems such as external hardware memory management systems and software
based memory allocators to be used with DPDK.

The existing API to the internal DPDK mempool manager will remain unchanged
and will be backward compatible. However, there will be an ABI breakage, as
the mempool struct is changing. These changes are all contained withing
RTE_NEXT_ABI defs, and the current or next code can be changed with
the CONFIG_RTE_NEXT_ABI config setting

There are two aspects to external mempool manager.
  1. Adding the code for your new mempool handler. This is achieved by adding a
     new mempool handler source file into the librte_mempool library, and
     using the REGISTER_MEMPOOL_HANDLER macro.
  2. Using the new API to call rte_mempool_create_ext to create a new mempool
     using the name parameter to identify which handler to use.

New API calls added
 1. A new mempool 'create' function which accepts mempool handler name.
 2. A new mempool 'rte_get_mempool_handler' function which accepts mempool
    handler name, and returns the index to the relevant set of callbacks for
    that mempool handler

Several external mempool managers may be used in the same application. A new
mempool can then be created by using the new 'create' function, providing the
mempool handler name to point the mempool to the relevant mempool manager
callback structure.

The old 'create' function can still be called by legacy programs, and will
internally work out the mempool handle based on the flags provided (single
producer, single consumer, etc). By default handles are created internally to
implement the built-in DPDK mempool manager and mempool types.

The external mempool manager needs to provide the following functions.
 1. alloc     - allocates the mempool memory, and adds each object onto a ring
 2. put       - puts an object back into the mempool once an application has
                finished with it
 3. get       - gets an object from the mempool for use by the application
 4. get_count - gets the number of available objects in the mempool
 5. free      - frees the mempool memory

Every time a get/put/get_count is called from the application/PMD, the
callback for that mempool is called. These functions are in the fastpath,
and any unoptimised handlers may limit performance.

The new APIs are as follows:

1. rte_mempool_create_ext

struct rte_mempool *
rte_mempool_create_ext(const char * name, unsigned n,
        unsigned cache_size, unsigned private_data_size,
        int socket_id, unsigned flags,
        const char * handler_name);

2. rte_mempool_get_handler_name

char *
rte_mempool_get_handler_name(struct rte_mempool *mp);

Please see rte_mempool.h for further information on the parameters.


The important thing to note is that the mempool handler is passed by name
to rte_mempool_create_ext, and that in turn calls rte_get_mempool_handler to
get the handler index, which is stored in the rte_memool structure. This
allow multiple processes to use the same mempool, as the function pointers
are accessed via handler index.

The mempool handler structure contains callbacks to the implementation of
the handler, and is set up for registration as follows:

static struct rte_mempool_handler handler_sp_mc = {
    .name = "ring_sp_mc",
    .alloc = rte_mempool_common_ring_alloc,
    .put = common_ring_sp_put,
    .get = common_ring_mc_get,
    .get_count = common_ring_get_count,
    .free = common_ring_free,
};

And then the following macro will register the handler in the array of handlers

REGISTER_MEMPOOL_HANDLER(handler_mp_mc);

For and example of a simple malloc based mempool manager, see
lib/librte_mempool/custom_mempool.c

For an example of API usage, please see app/test/test_ext_mempool.c, which
implements a rudimentary mempool manager using simple mallocs for each
mempool object. This file also contains the callbacks and self registration
for the new handler.

David Hunt (2):
  mempool: support external handler
  mbuf: get default mempool handler from configuration

Olivier Matz (1):
  app/test: test external mempool handler

 app/test/test_mempool.c                    | 113 +++++++++++++++
 app/test/test_mempool_perf.c               |   1 -
 config/common_base                         |   1 +
 lib/librte_mbuf/rte_mbuf.c                 |  21 ++-
 lib/librte_mempool/Makefile                |   2 +
 lib/librte_mempool/rte_mempool.c           |  72 ++++------
 lib/librte_mempool/rte_mempool.h           | 212 +++++++++++++++++++++++++----
 lib/librte_mempool/rte_mempool_default.c   | 147 ++++++++++++++++++++
 lib/librte_mempool/rte_mempool_handler.c   | 139 +++++++++++++++++++
 lib/librte_mempool/rte_mempool_version.map |   4 +
 10 files changed, 637 insertions(+), 75 deletions(-)
 create mode 100644 lib/librte_mempool/rte_mempool_default.c
 create mode 100644 lib/librte_mempool/rte_mempool_handler.c

-- 
2.1.4



More information about the dev mailing list