[dpdk-dev] rte_hash_del_key crash in multi-process environment

[张伟]

Thanks so much! That fixes my problem. 








At 2016-04-19 15:39:16, "De Lara Guarch, Pablo" <pablo.de.lara.guarch at intel.com> wrote:
>Hi,
>
>> -----Original Message-----
>> From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of ??
>> Sent: Tuesday, April 19, 2016 5:58 AM
>> To: De Lara Guarch, Pablo
>> Cc: Thomas Monjalon; Gonzalez Monroy, Sergio; dev at dpdk.org; Dhana
>> Eadala; Richardson, Bruce; Qiu, Michael
>> Subject: [dpdk-dev] rte_hash_del_key crash in multi-process environment
>> 
>> Hi all,
>> 
>> 
>> In the multi-process environment, before I met a bug when calling
>> rte_hash_lookup_with_hash. Using Dhana's patch fixed my problem. Now I
>> need to remove the flow in the multi-process environment, the system gets
>> crashed when calling rte_hash_del_key function. The following is the gdb
>> trace. Does anybody meet this problem or know how to fix it?
>
>First of all, another fix for the multi-process support was implemented and merged for 16.04 release,
>so take a look at it, if you can.
>Regarding the rte_hash_del_key() function, you should use rte_hash_del_key_with_hash,
>if you want to use it in a multi-process environment (as you did for the lookup function).
>
>Thanks,
>Pablo
>
>> 
>> 
>> 
>> 
>> Program received signal SIGILL, Illegal instruction.
>> 
>> 0x000000000048a0dd in rte_port_ring_reader_frag_free
>> (port=0x7ffe113d4100) at /home/zhangwei1984/timopenNetVM/dpdk-
>> 2.2.0/lib/librte_port/rte_port_frag.c:266
>> 
>> 266            return -1;
>> 
>> (gdb) bt
>> 
>> #0  0x000000000048a0dd in rte_port_ring_reader_frag_free
>> (port=0x7ffe113d4100) at /home/zhangwei1984/timopenNetVM/dpdk-
>> 2.2.0/lib/librte_port/rte_port_frag.c:266
>> 
>> #1  0x000000000049c537 in rte_hash_del_key (h=0x7ffe113d4100,
>> key=0x7ffe092e1000)
>> 
>>    at /home/zhangwei1984/timopenNetVM/dpdk-
>> 2.2.0/lib/librte_hash/rte_cuckoo_hash.c:917
>> 
>> #2  0x000000000043716a in onvm_ft_remove_key (table=0x7ffe113c3e80,
>> key=0x7ffe092e1000) at /home/zhangwei1984/onvm-shared-
>> cpu/onvm/shared/onvm_flow_table.c:160
>> 
>> #3  0x000000000043767e in onvm_flow_dir_del_and_free_key
>> (key=0x7ffe092e1000) at /home/zhangwei1984/onvm-shared-
>> cpu/onvm/shared/onvm_flow_dir.c:144
>> 
>> #4  0x0000000000437619 in onvm_flow_dir_del_key (key=0x7ffe092e1000) at
>> /home/zhangwei1984/onvm-shared-
>> cpu/onvm/shared/onvm_flow_dir.c:128
>> 
>> #5  0x0000000000423ded in remove_flow_rule (idx=3) at
>> /home/zhangwei1984/onvm-shared-cpu/examples/flow_dir/flow_dir.c:130
>> 
>> #6  0x0000000000423e44 in clear_stat_remove_flow_rule
>> (nf_info=0x7fff3e652100) at /home/zhangwei1984/onvm-shared-
>> cpu/examples/flow_dir/flow_dir.c:145
>> 
>> #7  0x00000000004247e3 in alloc_nfs_install_flow_rule (services=0xd66e90
>> <services>, pkt=0x7ffe13f56400)
>> 
>>    at /home/zhangwei1984/onvm-shared-
>> cpu/examples/flow_dir/flow_dir.c:186
>> 
>> #8  0x0000000000424bdb in packet_handler (pkt=0x7ffe13f56400,
>> meta=0x7ffe13f56440) at /home/zhangwei1984/onvm-shared-
>> cpu/examples/flow_dir/flow_dir.c:294
>> 
>> #9  0x000000000043001d in onvm_nf_run (info=0x7fff3e651b00,
>> handler=0x424b21 <packet_handler>) at /home/zhangwei1984/onvm-
>> shared-cpu/onvm/onvm_nf/onvm_nflib.c:462
>> 
>> #10 0x0000000000424cc2 in main (argc=3, argv=0x7fffffffe660) at
>> /home/zhangwei1984/onvm-shared-cpu/examples/flow_dir/flow_dir.c:323
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> At 2016-03-23 03:53:43, "De Lara Guarch, Pablo"
>> <pablo.de.lara.guarch at intel.com> wrote:
>> >Hi Thomas,
>> >
>> >> -----Original Message-----
>> >> From: Thomas Monjalon [mailto:thomas.monjalon at 6wind.com]
>> >> Sent: Tuesday, March 22, 2016 11:42 AM
>> >> To: De Lara Guarch, Pablo; Gonzalez Monroy, Sergio
>> >> Cc: dev at dpdk.org; Dhana Eadala; Richardson, Bruce; Qiu, Michael
>> >> Subject: Re: [dpdk-dev] [PATCH] hash: fix memcmp function pointer in
>> multi-
>> >> process environment
>> >>
>> >> Hi,
>> >>
>> >> Pablo, Sergio, please could you help with this issue?
>> >
>> >I agree this is not the best way to fix this. I will try to have a fix without
>> having to use ifdefs.
>> >
>> >Thanks,
>> >Pablo
>> >>
>> >> 2016-03-13 22:16, Dhana Eadala:
>> >> > We found a problem in dpdk-2.2 using under multi-process
>> environment.
>> >> > Here is the brief description how we are using the dpdk:
>> >> >
>> >> > We have two processes proc1, proc2 using dpdk. These proc1 and proc2
>> >> are
>> >> > two different compiled binaries.
>> >> > proc1 is started as primary process and proc2 as secondary process.
>> >> >
>> >> > proc1:
>> >> > Calls srcHash = rte_hash_create("src_hash_name") to create rte_hash
>> >> structure.
>> >> > As part of this, this api initalized the rte_hash structure and set the
>> >> > srcHash->rte_hash_cmp_eq to the address of memcmp() from proc1
>> >> address space.
>> >> >
>> >> > proc2:
>> >> > calls srcHash =  rte_hash_find_existing("src_hash_name").
>> >> > This function call returns the rte_hash created by proc1.
>> >> > This srcHash->rte_hash_cmp_eq still points to the address of
>> >> > memcmp() from proc1 address space.
>> >> > Later proc2  calls
>> >> > rte_hash_lookup_with_hash(srcHash, (const void*) &key, key.sig);
>> >> > rte_hash_lookup_with_hash() invokes __rte_hash_lookup_with_hash(),
>> >> > which in turn calls h->rte_hash_cmp_eq(key, k->key, h->key_len).
>> >> > This leads to a crash as h->rte_hash_cmp_eq is an address
>> >> > from proc1 address space and is invalid address in proc2 address space.
>> >> >
>> >> > We found, from dpdk documentation, that
>> >> >
>> >> > "
>> >> >  The use of function pointers between multiple processes
>> >> >  running based of different compiled
>> >> >  binaries is not supported, since the location of a given function
>> >> >  in one process may be different to
>> >> >  its location in a second. This prevents the librte_hash library
>> >> >  from behaving properly as in a  multi-
>> >> >  threaded instance, since it uses a pointer to the hash function
>> internally.
>> >> >
>> >> >  To work around this issue, it is recommended that
>> >> >  multi-process applications perform the hash
>> >> >  calculations by directly calling the hashing function
>> >> >  from the code and then using the
>> >> >  rte_hash_add_with_hash()/rte_hash_lookup_with_hash() functions
>> >> >  instead of the functions which do
>> >> >  the hashing internally, such as rte_hash_add()/rte_hash_lookup().
>> >> > "
>> >> >
>> >> > We did follow the recommended steps by invoking
>> >> rte_hash_lookup_with_hash().
>> >> > It was no issue up to and including dpdk-2.0.
>> >> > In later releases started crashing because rte_hash_cmp_eq is
>> >> > introduced in dpdk-2.1
>> >> >
>> >> > We fixed it with the following patch and would like to
>> >> > submit the patch to dpdk.org.
>> >> > Patch is created such that, if anyone wanted to use dpdk in
>> >> > multi-process environment with function pointers not shared, they need
>> to
>> >> > define RTE_LIB_MP_NO_FUNC_PTR in their Makefile.
>> >> > Without defining this flag in Makefile, it works as it is now.
>> >>
>> >> Introducing #ifdef RTE_LIB_MP_NO_FUNC_PTR is not recommended.
>> >