[dpdk-dev] [PATCH v2] kni: add new mbuf in alloc_q only based on its empty slots

[Ferruh Yigit]

On 5/11/2017 12:51 PM, Gowrishankar wrote:
> From: Gowrishankar Muthukrishnan <gowrishankar.m at linux.vnet.ibm.com>
> 
> In kni_allocate_mbufs(), we attempt to add max_burst (32) count of mbuf
> always into alloc_q, which is excessively leading too many rte_pktmbuf_
> free() when alloc_q is contending at high packet rate (for eg 10Gig data).
> In a situation when alloc_q fifo can only accommodate very few (or zero)
> mbuf, create only what needed and add in fifo.

I remember I have tried similar, also tried allocating amount of
nb_packets read from kernel, both produced worse performance.
Can you please share your before/after performance numbers?

kni_allocate_mbufs() called within rte_kni_rx_burst() if any packet
received from kernel. If there is a heavy traffic, kernel will always
consume the alloc_q before this function called and this function will
fill it back. So there shouldn't be much cases that alloc_q fifo already
full.

Perhaps this can happen if application burst Rx from kernel in a number
less than 32, but fifo filled with fixed 32mbufs, is this your case?

Can you measure number of times rte_pktmbuf_free() called because of
alloc_q is full?

> 
> With this patch, we could stop random network stall in KNI at higher packet
> rate (eg 1G or 10G data between vEth0 and PMD) sufficiently exhausting
> alloc_q on above condition. I tested i40e PMD for this purpose in ppc64le.

If stall happens from NIC to kernel, this is kernel receive path, and
alloc_q is in kernel transmit path.

> 
> Changes:
>  v2 - alloc_q free count calculation corrected.
>       line wrap fixed for commit message.
> 
> Signed-off-by: Gowrishankar Muthukrishnan <gowrishankar.m at linux.vnet.ibm.com>
> ---
>  lib/librte_kni/rte_kni.c | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/lib/librte_kni/rte_kni.c b/lib/librte_kni/rte_kni.c
> index c3f9208..9c5d485 100644
> --- a/lib/librte_kni/rte_kni.c
> +++ b/lib/librte_kni/rte_kni.c
> @@ -624,6 +624,7 @@ struct rte_kni *
>  	int i, ret;
>  	struct rte_mbuf *pkts[MAX_MBUF_BURST_NUM];
>  	void *phys[MAX_MBUF_BURST_NUM];
> +	int allocq_free;
>  
>  	RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, pool) !=
>  			 offsetof(struct rte_kni_mbuf, pool));
> @@ -646,7 +647,9 @@ struct rte_kni *
>  		return;
>  	}
>  
> -	for (i = 0; i < MAX_MBUF_BURST_NUM; i++) {
> +	allocq_free = (kni->alloc_q->read - kni->alloc_q->write - 1) \
> +			& (MAX_MBUF_BURST_NUM - 1);
> +	for (i = 0; i < allocq_free; i++) {
>  		pkts[i] = rte_pktmbuf_alloc(kni->pktmbuf_pool);
>  		if (unlikely(pkts[i] == NULL)) {
>  			/* Out of memory */
>