[dpdk-stable] 答复: [dpdk-dev] [PATCH] lib/librte_hash: add rte_hash_del_key_fixed without compact

[Lilijun (Jerry)]

> -----邮件原件-----
> 发件人: Honnappa Nagarahalli [mailto:Honnappa.Nagarahalli at arm.com]
> 发送时间: 2020年5月14日 9:22
> 收件人: Lilijun (Jerry) <jerry.lilijun at huawei.com>; 'dev at dpdk.org'
> <dev at dpdk.org>
> 抄送: wangyunjian <wangyunjian at huawei.com>; xudingke
> <xudingke at huawei.com>; 'stable at dpdk.org' <stable at dpdk.org>; nd
> <nd at arm.com>; yipeng1.wang at intel.com; Honnappa Nagarahalli
> <Honnappa.Nagarahalli at arm.com>; nd <nd at arm.com>
> 主题: RE: [dpdk-dev] [PATCH] lib/librte_hash: add rte_hash_del_key_fixed
> without compact
> 
> <snip>
> 
> > > > >
> > > > > Adding Yipeng, maintainer for hash library
> > > > >
> > > > > >
> > > > > > Thanks for your reply.
> > > > > >
> > > > > > Using rte_hash iterate and delete keys is to free the related
> > > > > > data's
> > > > memory.
> > > > > > There are two reasons why rte_hash_reset() is not properly:
> > > > > > 1)  the reset function just clear all keys, the key's related
> > > > > > data are
> > > leaked.
> > > > > That is a good point. I think this should be documented in the API.
> > > > >
> > > > > > 2)  In some cases, I don't need delete all keys. Just some
> > > > > > selected keys and data are deleted and released.
> > > > > I understand the problem you have pointed out and understand how
> > > > > to reproduce it. But, the use case is not clear to me. Can you
> > > > > please explain the use case?
> > > > [Lilijun (Jerry)]
> > > >
> > > > As you know, the dpdk rte_hash use a fixed size table to store all
> > > keys/datas.
> > > > The memory used by hash table is related with this fixed size.
> > > > In my case, normally the count of keys is about 100,000 but
> > > > sometimes the count may burst up to 30,000,000.
> > > > In order to save memory usage, I create a small hash table with
> > > > 100,000 size and replace to a bigger one with 30,000,000 size when
> > > > there are more keys to be stored. Also when the key's count
> > > > reduced to less than 100,000, I replace the hash table with a
> > > > small one to save the
> > > memory.
> > > Thank you for explaining this. What happens to the reader when you
> > > are deleting from old table and inserting in the new one? Which
> > > table does the reader lookup from?
> > [Lilijun (Jerry)]
> > Lookup functions works well at any time. The problem is in
> > rte_hash_iterate() functions. Some example codes like this:
> > *next = 0;
> > //If rh has 10000 entries at first.
> > while ((idx = rte_hash_iterate(rh, key, data, next)) >= 0) {
> >               rte_hash_del_key(rh, key);  //BUT HERE maybe only delete
> > 9990 keys !!!
> >               free(*data);
> > }
> > //There are still 10 key/datas not freed and will be leaked.
> > rte_hash_free(rh);
> I understand this problem.
> I am trying to understand if there are other problems in the process you are
> following.
> For ex: when you are transferring an entry from the old table, if the reader is
> looking up from the old table, the entry will not be found, even though the
> entry is available in the new table. Can this happen?
> 
[Lilijun (Jerry)] 
Your example may be happen if read and resize are in two thread context and it's like a RCU cases. 
But I think it can be fixed for reader can retry the lookup or use a read-lock while resize using a write-lock.

> <snip the diff>