[PATCH v7] enhance NUMA affinity heuristic

[You, KaisenX]

> -----Original Message-----
> From: Burakov, Anatoly <anatoly.burakov at intel.com>
> Sent: 2023年5月23日 18:45
> To: You, KaisenX <kaisenx.you at intel.com>; dev at dpdk.org
> Cc: Zhou, YidingX <yidingx.zhou at intel.com>; thomas at monjalon.net;
> david.marchand at redhat.com; Matz, Olivier <olivier.matz at 6wind.com>;
> ferruh.yigit at amd.com; zhoumin at loongson.cn; stable at dpdk.org
> Subject: Re: [PATCH v7] enhance NUMA affinity heuristic
> 
> On 5/23/2023 3:50 AM, Kaisen You wrote:
> > When a DPDK application is started on only one numa node, memory is
> > allocated for only one socket. When interrupt threads use memory,
> > memory may not be found on the socket where the interrupt thread is
> > currently located, and memory has to be reallocated on the hugepage,
> > this operation will lead to performance degradation.
> >
> > Fixes: 705356f0811f ("eal: simplify control thread creation")
> > Fixes: 770d41bf3309 ("malloc: fix allocation with unknown socket ID")
> > Cc: stable at dpdk.org
> >
> > Signed-off-by: Kaisen You <kaisenx.you at intel.com>
> 
> Hi You,
> 
> I've suggested comment rewordings based on my understanding of the issue.
> 
> > ---
> > Changes since v6:
> > - New explanation for easy understanding,
> >
> > Changes since v5:
> > - Add comments to the code,
> >
> > Changes since v4:
> > - mod the patch title,
> >
> > Changes since v3:
> > - add the assignment of socket_id in thread initialization,
> >
> > Changes since v2:
> > - add uncommitted local change and fix compilation,
> >
> > Changes since v1:
> > - accomodate for configurations with main lcore running on multiples
> >    physical cores belonging to different numa,
> > ---
> >   lib/eal/common/eal_common_thread.c | 6 ++++++
> >   lib/eal/common/malloc_heap.c       | 9 +++++++++
> >   2 files changed, 15 insertions(+)
> >
> > diff --git a/lib/eal/common/eal_common_thread.c
> > b/lib/eal/common/eal_common_thread.c
> > index 079a385630..6479b66da1 100644
> > --- a/lib/eal/common/eal_common_thread.c
> > +++ b/lib/eal/common/eal_common_thread.c
> > @@ -252,6 +252,12 @@ static int ctrl_thread_init(void *arg)
> >   	struct rte_thread_ctrl_params *params = arg;
> >
> >   	__rte_thread_init(rte_lcore_id(), cpuset);
> > +	/* set the value of the per-core variable _socket_id to
> SOCKET_ID_ANY.
> > +	 * Satisfy the judgment condition when threads find memory.
> > +	 * If SOCKET_ID_ANY is not specified, the thread may go to a node
> with
> > +	 * unallocated memory in a subsequent memory search.
> 
> I suggest a different comment wording:
> 
> Set control thread socket ID to SOCKET_ID_ANY as control threads may be
> scheduled on any NUMA node.
> 
> > +	 */
> > +	RTE_PER_LCORE(_socket_id) = SOCKET_ID_ANY;
> >   	params->ret = rte_thread_set_affinity_by_id(rte_thread_self(),
> cpuset);
> >   	if (params->ret != 0) {
> >   		__atomic_store_n(&params->ctrl_thread_status,
> > diff --git a/lib/eal/common/malloc_heap.c
> > b/lib/eal/common/malloc_heap.c index d25bdc98f9..6d37f8afee 100644
> > --- a/lib/eal/common/malloc_heap.c
> > +++ b/lib/eal/common/malloc_heap.c
> > @@ -716,6 +716,15 @@ malloc_get_numa_socket(void)
> >   		if (conf->socket_mem[socket_id] != 0)
> >   			return socket_id;
> >   	}
> > +	/* Trying to allocate memory on the main lcore numa node.
> > +	 * especially when the DPDK application is started only on one numa
> node.
> > +	 */
> 
> I suggest the following comment wording:
> 
> We couldn't find quickly find a NUMA node where memory was available, so
> fall back to using main lcore socket ID.
> 
> > +	socket_id = rte_lcore_to_socket_id(rte_get_main_lcore());
> > +	/* When the socket_id obtained in the main lcore numa is
> SOCKET_ID_ANY,
> > +	 * The probability of finding memory on rte_socket_id_by_idx(0) is
> higher.
> > +	 */
> 
> I suggest the following comment wording:
> 
> Main lcore socket ID may be SOCKET_ID_ANY in cases when main lcore
> thread is affinitized to multiple NUMA nodes.
> 
> > +	if (socket_id != (unsigned int)SOCKET_ID_ANY)
> > +		return socket_id;
> >
> 
> I suggest adding comment here:
> 
> Failed to find meaningful socket ID, so just use the first one available.
> 
> >   	return rte_socket_id_by_idx(0);
> >   }
> 
> I believe these comments offer better explanation as to why we are doing
> the things we do here.
> 
> Whether or not you decide to take these corrections on board,
> 
> Acked-by: Anatoly Burakov <anatoly.burakov at intel.com>

Thank you for your acked and suggestions, I will adopt your suggestions in the V8 version.
> 
> --
> Thanks,
> Anatoly