[PATCH] net/af_xdp: parse numa node id from sysfs

[Du, Frank]

> -----Original Message-----
> From: Ferruh Yigit <ferruh.yigit at amd.com>
> Sent: Tuesday, January 17, 2023 5:14 PM
> To: Du, Frank <frank.du at intel.com>; Loftus, Ciara <ciara.loftus at intel.com>
> Cc: dev at dpdk.org
> Subject: Re: [PATCH] net/af_xdp: parse numa node id from sysfs
> 
> On 1/17/2023 1:35 AM, Du, Frank wrote:
> 
> Moved down, please don't top post.
> 
> >>
> >> -----Original Message-----
> >> From: Ferruh Yigit, <ferruh.yigit at amd.com>
> >> Sent: Monday, January 16, 2023 9:15 PM
> >> To: Du, Frank <frank.du at intel.com>; Loftus, Ciara
> >> <ciara.loftus at intel.com>
> >> Cc: dev at dpdk.org
> >> Subject: Re: [PATCH] net/af_xdp: parse numa node id from sysfs
> >>
> >> On 12/12/2022 12:48 AM, Frank Du wrote:
> >>> Get from /sys/class/net/{if}/device/numa_node.
> >>>
> >>> Signed-off-by: Frank Du <frank.du at intel.com>
> >>> ---
> >>>  drivers/net/af_xdp/rte_eth_af_xdp.c | 17 ++++++++++++++---
> >>>  1 file changed, 14 insertions(+), 3 deletions(-)
> >>>
> >>> diff --git a/drivers/net/af_xdp/rte_eth_af_xdp.c
> >>> b/drivers/net/af_xdp/rte_eth_af_xdp.c
> >>> index b6ec9bf490..38b9d36ab5 100644
> >>> --- a/drivers/net/af_xdp/rte_eth_af_xdp.c
> >>> +++ b/drivers/net/af_xdp/rte_eth_af_xdp.c
> >>> @@ -39,6 +39,7 @@
> >>>  #include <rte_power_intrinsics.h>
> >>>
> >>>  #include "compat.h"
> >>> +#include "eal_filesystem.h"
> >>>
> >>>  #ifndef SO_PREFER_BUSY_POLL
> >>>  #define SO_PREFER_BUSY_POLL 69
> >>> @@ -2038,9 +2039,6 @@ rte_pmd_af_xdp_probe(struct rte_vdev_device
> *dev)
> >>>  		return -EINVAL;
> >>>  	}
> >>>
> >>> -	if (dev->device.numa_node == SOCKET_ID_ANY)
> >>> -		dev->device.numa_node = rte_socket_id();
> >>> -
> >>>  	if (parse_parameters(kvlist, if_name, &xsk_start_queue_idx,
> >>>  			     &xsk_queue_cnt, &shared_umem, prog_path,
> >>>  			     &busy_budget, &force_copy) < 0) { @@ -2053,6
> +2051,19 @@
> >>> rte_pmd_af_xdp_probe(struct rte_vdev_device *dev)
> >>>  		return -EINVAL;
> >>>  	}
> >>>
> >>> +	/* get numa node id from net sysfs */
> >>> +	if (dev->device.numa_node == SOCKET_ID_ANY) {
> >>> +		unsigned long numa = 0;
> >>> +		char numa_path[PATH_MAX];
> >>> +
> >>> +		snprintf(numa_path, sizeof(numa_path),
> "/sys/class/net/%s/device/numa_node",
> >>> +			 if_name);
> >>> +		if (eal_parse_sysfs_value(numa_path, &numa) != 0)
> >>> +			dev->device.numa_node = rte_socket_id();
> >>> +		else
> >>> +			dev->device.numa_node = numa;
> >>> +	}
> >>> +
> >>>  	busy_budget = busy_budget == -1 ? ETH_AF_XDP_DFLT_BUSY_BUDGET :
> >>>  					busy_budget;
> >>>
> >>
> >> Hi Frank,
> >>
> >> It looks reasonable to set virtual DPDK af_xdp device socket to actual
> underlying device socket. And as I checked quickly, it works as expected.
> >>
> >> But what is the impact and motivation of the patch? In other words why you
> are doing this patch and what output you are expecting as a result?
> >> Did you able to do any performance testing, and are you observing any
> difference before and after this test?
> >>
> >
> > Hi ferruh,
> >
> > Our application use rte_eth_dev_socket_id to query the socket that a NIC port
> connected, then allocate lcore/memory according to this affinity.
> >
> > The remote memory access is really slow compared to local.
> >
> 
> As you observing any performance gain after change? If so, how much?

Hi Ferruh,

The NIC in our setup is on the socket 1, if our workload(both memory and cpu) are running on socket 0, it can get max 12g/s throughput on a single core. With this patch, the workload is running on the correct socket 1 cpus, it can get up to 16g/s on a single core.