[PATCH] net/iavf:fix slow memory allocation

[Ferruh Yigit]

On 11/17/2022 6:57 AM, Kaisen You wrote:
> In some cases, the DPDK does not allocate hugepage heap memory to
> some sockets due to the user setting parameters
> (e.g. -l 40-79, SOCKET 0 has no memory).
> When the interrupt thread runs on the corresponding core of this
> socket, each allocation/release will execute a whole set of heap
> allocation/release operations,resulting in poor performance.
> Instead we call malloc() to get memory from the system's
> heap space to fix this problem.
> 

Hi Kaisen,

Using libc malloc can improve performance for this case, but I would
like to understand root cause of the problem.


As far as I can see, interrupt callbacks are run by interrupt thread
("eal-intr-thread"),
and interrupt thread created by 'rte_ctrl_thread_create()' API.

'rte_ctrl_thread_create()' comment mentions that "CPU affinity retrieved
at the time 'rte_eal_init()' was called,"

And 'rte_eal_init()' is run on main lcore, which is the first lcore in
the core list (unless otherwise defined with --main-lcore).

So, the interrupts should be running on a core that has hugepages
allocated for it, am I missing something here?




And what about using 'rte_malloc_socket()' API (instead of rte_malloc),
which gets 'socket' as parameter, and provide the socket that devices is
on as parameter to this API? Is it possible to test this?



> Fixes: cb5c1b91f76f ("net/iavf: add thread for event callbacks")
> Cc: stable at dpdk.org
> 
> Signed-off-by: Kaisen You <kaisenx.you at intel.com>
> ---
>  drivers/net/iavf/iavf_vchnl.c | 8 +++-----
>  1 file changed, 3 insertions(+), 5 deletions(-)
> 
> diff --git a/drivers/net/iavf/iavf_vchnl.c b/drivers/net/iavf/iavf_vchnl.c
> index f92daf97f2..a05791fe48 100644
> --- a/drivers/net/iavf/iavf_vchnl.c
> +++ b/drivers/net/iavf/iavf_vchnl.c
> @@ -36,7 +36,6 @@ struct iavf_event_element {
>  	struct rte_eth_dev *dev;
>  	enum rte_eth_event_type event;
>  	void *param;
> -	size_t param_alloc_size;
>  	uint8_t param_alloc_data[0];
>  };
>  
> @@ -80,7 +79,7 @@ iavf_dev_event_handle(void *param __rte_unused)
>  		TAILQ_FOREACH_SAFE(pos, &pending, next, save_next) {
>  			TAILQ_REMOVE(&pending, pos, next);
>  			rte_eth_dev_callback_process(pos->dev, pos->event, pos->param);
> -			rte_free(pos);
> +			free(pos);
>  		}
>  	}
>  
> @@ -94,14 +93,13 @@ iavf_dev_event_post(struct rte_eth_dev *dev,
>  {
>  	struct iavf_event_handler *handler = &event_handler;
>  	char notify_byte;
> -	struct iavf_event_element *elem = rte_malloc(NULL, sizeof(*elem) + param_alloc_size, 0);
> +	struct iavf_event_element *elem = malloc(sizeof(*elem) + param_alloc_size);
>  	if (!elem)
>  		return;
>  
>  	elem->dev = dev;
>  	elem->event = event;
>  	elem->param = param;
> -	elem->param_alloc_size = param_alloc_size;
>  	if (param && param_alloc_size) {
>  		rte_memcpy(elem->param_alloc_data, param, param_alloc_size);
>  		elem->param = elem->param_alloc_data;
> @@ -165,7 +163,7 @@ iavf_dev_event_handler_fini(void)
>  	struct iavf_event_element *pos, *save_next;
>  	TAILQ_FOREACH_SAFE(pos, &handler->pending, next, save_next) {
>  		TAILQ_REMOVE(&handler->pending, pos, next);
> -		rte_free(pos);
> +		free(pos);
>  	}
>  }
>