[dpdk-users] multiple use rte_pktmbuf_free occurs competition
forsakening at sina.cn
forsakening at sina.cn
Sat Aug 20 10:19:07 CEST 2016
Hi :
I use the below struct to handle the internet packet based on the dpdk-16.04 interfaces.
And my hardware: Intel(R) Xeon(R) CPU E5-2618L v3 @ 2.30GHz , which has 32 logic cores .
And my platform: CentOs 7.0.
[stuct removed cuz not allowed pic.]
Structure:
Internet Packet --> NIC --> DPDK-RX --> Dispatch --> Handle Packet Threads (1 to N) --> rte_pktmbuf_free
And now I use about 12 or more handle threads, and these threads are set affinity to my intel logic core, also, in order not to schedule by other thread, i isolate these logic cores out.
Cuz the Packet may not be very standard Ethernet Packet, or My 82599ES NIC cannot RSS the packet well, I use only one Mempool, and setup only one rx que.
To handle these packet, I use one dispatch thread to "Decode" the packet and dispatch them to my handle threads.
And then , question comes,when these handle threads free the packet, they are “blocked” or "slow" on rte_pktmbuf_free interface.
Below is my "perf result" of one thread:
Samples: 488K of event 'cycles', Event count (approx.): 22076989633
78.92% libvst_driver.so [.] rte_pktmbuf_free
.....Other Info
you can see the free interface slow my thread.....
and, annonate of the rte_pktmbuf_free like this:
/*
* If there are other enqueues in progress that preceded us,
* we need to wait for them to complete
*/
while (unlikely(r->prod.tail != prod_head)) {
rte_pause();
/* Set RTE_RING_PAUSE_REP_COUNT to avoid spin too long waiting
* for other thread finish. It gives pre-empted thread a chance
* to proceed and finish with ring dequeue operation. */
if (RTE_RING_PAUSE_REP_COUNT &&
++rep == RTE_RING_PAUSE_REP_COUNT) {
rep = 0;
sched_yield();
}
}
According to the hotspot,the free interface is waiting for other thread to compelete the enque process.
I know the thread of my free thread are little more , but can anybody give me some advise about how can i use the free interface more effectively ?
Thanks very much !!
Best Regards
forsakening at sina.cn
MYTtech, NanJing, CHN
More information about the users
mailing list