[PATCH] net/iavf:fix slow memory allocation

[You, KaisenX]

> -----Original Message-----
> From: David Marchand <david.marchand at redhat.com>
> Sent: 2022年12月21日 18:50
> To: You, KaisenX <kaisenx.you at intel.com>
> Cc: Ferruh Yigit <ferruh.yigit at amd.com>; dev at dpdk.org; Burakov, Anatoly
> <anatoly.burakov at intel.com>; stable at dpdk.org; Yang, Qiming
> <qiming.yang at intel.com>; Zhou, YidingX <yidingx.zhou at intel.com>; Wu,
> Jingjing <jingjing.wu at intel.com>; Xing, Beilei <beilei.xing at intel.com>; Zhang,
> Qi Z <qi.z.zhang at intel.com>; Luca Boccassi <bluca at debian.org>; Mcnamara,
> John <john.mcnamara at intel.com>; Kevin Traynor <ktraynor at redhat.com>
> Subject: Re: [PATCH] net/iavf:fix slow memory allocation
> 
> On Wed, Dec 21, 2022 at 10:12 AM You, KaisenX <kaisenx.you at intel.com>
> wrote:
> > > -----Original Message-----
> > > From: David Marchand <david.marchand at redhat.com>
> > > Sent: 2022年12月20日 18:33
> > > To: You, KaisenX <kaisenx.you at intel.com>
> > > Cc: Ferruh Yigit <ferruh.yigit at amd.com>; dev at dpdk.org; Burakov,
> > > Anatoly <anatoly.burakov at intel.com>; stable at dpdk.org; Yang, Qiming
> > > <qiming.yang at intel.com>; Zhou, YidingX <yidingx.zhou at intel.com>; Wu,
> > > Jingjing <jingjing.wu at intel.com>; Xing, Beilei
> > > <beilei.xing at intel.com>; Zhang, Qi Z <qi.z.zhang at intel.com>; Luca
> > > Boccassi <bluca at debian.org>; Mcnamara, John
> > > <john.mcnamara at intel.com>; Kevin Traynor <ktraynor at redhat.com>
> > > Subject: Re: [PATCH] net/iavf:fix slow memory allocation
> > >
> > > On Tue, Dec 20, 2022 at 11:12 AM You, KaisenX
> > > <kaisenx.you at intel.com>
> > > wrote:
> > > > > I tried to play a bit with a E810 nic on a dual numa and I can't
> > > > > see anything wrong for now.
> > > > > Can you provide a simple and small reproducer of your issue?
> > > > >
> > > > > Thanks.
> > > > >
> > > > This is my environment:
> > > > Enter "lscpu" on the command line:
> > > > NUMA:
> > > >         NUMA node(s): 2
> > > >         NUMA node0 CPU(S) : 0-27,56-83
> > > >         NUMA node1 CPU(S) : 28-55,84-111
> > > >
> > > > List the steps to reproduce the issue:
> > > >
> > > > 1. create vf and blind to dpdk
> > > > echo 1 > /sys/bus/pci/devices/0000\:ca\:00.0/sriov_ numvfs
> > > > ./usertools/dpdk-devbind. py -b vfio-pci 0000:ca:01.0 2. launch
> > > > testpmd ./x86_ 64-native-linuxapp-clang/app/dpdk-testpmd -l 28-48
> > > > -n 4 -a 0000:ca:01.0 --file-prefix=dpdk_ 525342_ 20221104042659 --
> > > > -i
> > > > --rxq=256 --txq=256
> > > > --total-num-mbufs=500000
> > > >
> > > > Parameter Description:
> > > >  "-l 28-48":The range of parameter values after "-l" must be on
> > > > "NUMA
> > > node1 CPU(S)"
> > > >  "0000:ca:01.0":inset on node1
> > > - Back to your topic.
> > > Can you try this simple hack:
> > >
> > > diff --git a/lib/eal/common/eal_common_thread.c
> > > b/lib/eal/common/eal_common_thread.c
> > > index c5d8b4327d..92160c7fa6 100644
> > > --- a/lib/eal/common/eal_common_thread.c
> > > +++ b/lib/eal/common/eal_common_thread.c
> > > @@ -253,6 +253,7 @@ static void *ctrl_thread_init(void *arg)
> > >         void *routine_arg = params->arg;
> > >
> > >         __rte_thread_init(rte_lcore_id(), cpuset);
> > > +       RTE_PER_LCORE(_socket_id) = SOCKET_ID_ANY;
> > >         params->ret = pthread_setaffinity_np(pthread_self(),
> sizeof(*cpuset),
> > >                 cpuset);
> > >         if (params->ret != 0) {
> > >
> > Thanks for your advice.
> >
> > But this issue still exists after I tried.
> 
> Ok, I think I understand what is wrong... but I am still guessing as I am not
> sure what your "issue" is.
> Can you have a try with:
> https://patchwork.dpdk.org/project/dpdk/patch/20221221104858.296530-1-
> david.marchand at redhat.com/
> 
> Thanks.
> 
I think this issue is similar to the description in the patch you gave me.

when the DPDK application is started only on one numa node, Interrupt thread 
find memory on another numa node. This leads to a whole set of memory 
allocation/release operations every time when "rte_malloc" is called.
This is the root cause of this issue.

This issue can be solved after I tried.
Thanks for your advice.
> 
> --
> David Marchand