[dpdk-users] Facing issue as data processing packet got stuck in rte_pause

[Manish Jangid]

Second Retry:

Could anyone please  help in this issue ? Thanks in advance.

Regards
Manish

On Jul 6, 2017 6:34 PM, "Manish Jangid" <manish.jangid08 at gmail.com> wrote:

> Hello all,
>
> I am facing an issue while processing packets in dpdk in which we reach a
> situation where packet processing gets stopped. When I attach gdb to
> process, I see the following back trace. The data processing thread is
> stuck in rte_pause() call inside common_ring_mp_enqueue because of a check
> on freepool state. The following is the analysis which I found while
> debugging it.
>
> ring <MP_FEDP_MBUF_POOL>@0x7f33b0f0f0c0
> flags=0
> size=131072
> ct=3736608500
> ch=3736608500
> pt=3736611582
> ph=3736612099
> used=3082
> avail=127989
> watermark=0
> no statistics available
>
>
> gdb) bt
> #0 0x00007f9fb36ff425 in *common_ring_mp_enqueue (*) from
> /usr/lib64/librte_mempool.so.2.1
> #1 0x000000000040ad68 in rte_mempool_ops_enqueue_bulk (n=<optimized out>,
> obj_table=<optimized out>, mp=<optimized out>)
> at /home/osboxes/Downloads/dpdk/dpdk-latest/x86_64-native-
> linuxapp-gcc/include/rte_mempool.h:495
> #2 __mempool_generic_put (flags=<optimized out>, cache=0x7f33c3844e40,
> n=1, obj_table=0x7f9fab0a9b88, mp=<optimized out>)
> at /home/osboxes/Downloads/dpdk/dpdk-latest/x86_64-native-
> linuxapp-gcc/include/rte_mempool.h:1071
> #3 rte_mempool_generic_put (flags=<optimized out>, cache=0x7f33c3844e40,
> n=1, obj_table=0x7f9fab0a9b88, mp=<optimized out>)
> at /home/osboxes/Downloads//dpdk/dpdk-latest/x86_64-native-
> linuxapp-gcc/include/rte_mempool.h:1110
> #4 rte_mempool_put_bulk (n=1, obj_table=0x7f9fab0a9b88, mp=<optimized out>)
> at /home/osboxes/Downloads//dpdk/dpdk-latest/x86_64-native-
> linuxapp-gcc/include/rte_mempool.h:1173
> #5 rte_mempool_put (obj=0x7f33c1634780, mp=<optimized out>)
> at /home/osboxes/Downloads//dpdk/dpdk-latest/x86_64-native-
> linuxapp-gcc/include/rte_mempool.h:1225
> #6 __rte_mbuf_raw_free (m=0x7f33c1634780)
> at /home/osboxes/Downloads//dpdk/dpdk-latest/x86_64-native-
> linuxapp-gcc/include/rte_mbuf.h:783
> #7 rte_pktmbuf_free_seg (m=<optimized out>)
> at /home/osboxes/Downloads//dpdk/dpdk-latest/x86_64-native-
> linuxapp-gcc/include/rte_mbuf.h:1224
> #8 rte_pktmbuf_free (m=0x7f33c1634780)
> at /home/osboxes/Downloads//dpdk/dpdk-latest/x86_64-native-
> linuxapp-gcc/include/rte_mbuf.h:1245
> #9 lcore_main (args=args at entry=0x0) at dp_app.c:565
>
> When we try to  use debug APIs given by Dpdk I see that MEMPOOL is giving
> error of invalid cache len as shown below.
>
> Auditing mbuf_pool using 'rte_mempool_audit'
> MEMPOOL: badness on cache[27]
> PANIC in mempool_audit_cache():
>
> *MEMPOOL: invalid cache len-*
> When I further dump rte_mempool cache , I saw the following
>
> stqh_last = 0x7f33b0f0f040}}
> (gdb) p ((struct rte_mempool *)0x7f33c380e900)->local_cache
> $22 = (struct rte_mempool_cache *) 0x7f33c380e9c0
> (gdb) p *(((struct rte_mempool *)0x7f33c380e900)->local_cache+26)
> $49 = {size = 250, flushthresh = 375, len = 0, objs = {0x0 <repeats 1536
> times>}}
> *(gdb) p *(((struct rte_mempool *)0x7f33c380e900)->local_cache+27)*
> $50 = {size = 250, flushthresh = 375, len = 377, objs = {0x7f33b7f50780,
> 0x7f33b7f510c0, 0x7f33b7f51a00, 0x7f33b7f52340, 0x7f33b7f52c80,
> 0x7f33b7f535c0, 0x7f33b7f53f00,
>
> I am really looking forward for help as under what circumstances we can
> reach to such condition. Thank you in advance for your help.
>
> -
> Thanks
> Manish Kumar
>
>
>
>