[dpdk-dev] [RFC] ethdev: support Tx queue free descriptor query

[Dumitrescu, Cristian]

> -----Original Message-----
> From: jerinj at marvell.com <jerinj at marvell.com>
> Sent: Tuesday, December 19, 2023 5:30 PM
> To: dev at dpdk.org; Thomas Monjalon <thomas at monjalon.net>; Ferruh Yigit
> <ferruh.yigit at amd.com>; Andrew Rybchenko <andrew.rybchenko at oktetlabs.ru>
> Cc: ferruh.yigit at xilinx.com; ajit.khaparde at broadcom.com;
> aboyer at pensando.io; Xing, Beilei <beilei.xing at intel.com>; Richardson, Bruce
> <bruce.richardson at intel.com>; chas3 at att.com; chenbo.xia at intel.com; Loftus,
> Ciara <ciara.loftus at intel.com>; dsinghrawat at marvell.com; Czeck, Ed
> <ed.czeck at atomicrules.com>; evgenys at amazon.com; grive at u256.net;
> g.singh at nxp.com; zhouguoyang at huawei.com; Wang, Haiyue
> <haiyue.wang at intel.com>; hkalra at marvell.com; heinrich.kuhn at corigine.com;
> hemant.agrawal at nxp.com; hyonkim at cisco.com; igorch at amazon.com;
> irusskikh at marvell.com; jgrajcia at cisco.com; Singh, Jasvinder
> <jasvinder.singh at intel.com>; jianwang at trustnetic.com;
> jiawenwu at trustnetic.com; Wu, Jingjing <jingjing.wu at intel.com>;
> johndale at cisco.com; john.miller at atomicrules.com; linville at tuxdriver.com;
> Wiles, Keith <keith.wiles at intel.com>; kirankumark at marvell.com;
> oulijun at huawei.com; lironh at marvell.com; longli at microsoft.com;
> mw at semihalf.com; spinler at cesnet.cz; matan at nvidia.com; Peters, Matt
> <matt.peters at windriver.com>; maxime.coquelin at redhat.com;
> mk at semihalf.com; humin29 at huawei.com; pnalla at marvell.com;
> ndabilpuram at marvell.com; Yang, Qiming <qiming.yang at intel.com>; Zhang, Qi Z
> <qi.z.zhang at intel.com>; radhac at marvell.com; rahul.lakkireddy at chelsio.com;
> rmody at marvell.com; Xu, Rosen <rosen.xu at intel.com>;
> sachin.saxena at oss.nxp.com; skoteshwar at marvell.com; shshaikh at marvell.com;
> shaibran at amazon.com; Siegel, Shepard <shepard.siegel at atomicrules.com>;
> asomalap at amd.com; somnath.kotur at broadcom.com;
> sthemmin at microsoft.com; Webster, Steven <steven.webster at windriver.com>;
> skori at marvell.com; mtetsuyah at gmail.com; vburru at marvell.com;
> viacheslavo at nvidia.com; Wang, Xiao W <xiao.w.wang at intel.com>;
> cloud.wangxiaoyun at huawei.com; yisen.zhuang at huawei.com; Wang, Yong
> <yongwang at vmware.com>; xuanziyang2 at huawei.com; Dumitrescu, Cristian
> <cristian.dumitrescu at intel.com>; Jerin Jacob <jerinj at marvell.com>
> Subject: [dpdk-dev] [RFC] ethdev: support Tx queue free descriptor query
> 
> From: Jerin Jacob <jerinj at marvell.com>
> 
> Introduce a new API to retrieve the number of available free descriptors
> in a Tx queue. Applications can leverage this API in the fast path to
> inspect the Tx queue occupancy and take appropriate actions based on the
> available free descriptors.
> 
> A notable use case could be implementing Random Early Discard (RED)
> in software based on Tx queue occupancy.
> 
> Signed-off-by: Jerin Jacob <jerinj at marvell.com>
> ---
>  doc/guides/nics/features.rst         | 10 ++++
>  doc/guides/nics/features/default.ini |  1 +
>  lib/ethdev/ethdev_trace_points.c     |  3 ++
>  lib/ethdev/rte_ethdev.h              | 78 ++++++++++++++++++++++++++++
>  lib/ethdev/rte_ethdev_core.h         |  7 ++-
>  lib/ethdev/rte_ethdev_trace_fp.h     |  8 +++
>  6 files changed, 106 insertions(+), 1 deletion(-)
> 
> diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
> index f7d9980849..9d6655473a 100644
> --- a/doc/guides/nics/features.rst
> +++ b/doc/guides/nics/features.rst
> @@ -962,6 +962,16 @@ management (see :doc:`../prog_guide/power_man` for
> more details).
> 
>  * **[implements] eth_dev_ops**: ``get_monitor_addr``
> 
> +.. _nic_features_tx_queue_free_desc_query:
> +
> +Tx queue free descriptor query
> +------------------------------
> +
> +Supports to get the number of free descriptors in a Tx queue.
> +
> +* **[implements] eth_dev_ops**: ``tx_queue_free_desc_get``.
> +* **[related] API**: ``rte_eth_tx_queue_free_desc_get()``.
> +
>  .. _nic_features_other:
> 
>  Other dev ops not represented by a Feature
> diff --git a/doc/guides/nics/features/default.ini
> b/doc/guides/nics/features/default.ini
> index 806cb033ff..b30002b1c1 100644
> --- a/doc/guides/nics/features/default.ini
> +++ b/doc/guides/nics/features/default.ini
> @@ -59,6 +59,7 @@ Packet type parsing  =
>  Timesync             =
>  Rx descriptor status =
>  Tx descriptor status =
> +Tx free descriptor query =
>  Basic stats          =
>  Extended stats       =
>  Stats per queue      =
> diff --git a/lib/ethdev/ethdev_trace_points.c b/lib/ethdev/ethdev_trace_points.c
> index 91f71d868b..346f37f2e4 100644
> --- a/lib/ethdev/ethdev_trace_points.c
> +++ b/lib/ethdev/ethdev_trace_points.c
> @@ -481,6 +481,9 @@
> RTE_TRACE_POINT_REGISTER(rte_eth_trace_count_aggr_ports,
>  RTE_TRACE_POINT_REGISTER(rte_eth_trace_map_aggr_tx_affinity,
>  	lib.ethdev.map_aggr_tx_affinity)
> 
> +RTE_TRACE_POINT_REGISTER(rte_eth_trace_tx_queue_free_desc_get,
> +	lib.ethdev.tx_queue_free_desc_get)
> +
>  RTE_TRACE_POINT_REGISTER(rte_flow_trace_copy,
>  	lib.ethdev.flow.copy)
> 
> diff --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h
> index 77331ce652..033fcb8c9b 100644
> --- a/lib/ethdev/rte_ethdev.h
> +++ b/lib/ethdev/rte_ethdev.h
> @@ -6802,6 +6802,84 @@ rte_eth_recycle_mbufs(uint16_t rx_port_id, uint16_t
> rx_queue_id,
>  __rte_experimental
>  int rte_eth_buffer_split_get_supported_hdr_ptypes(uint16_t port_id, uint32_t
> *ptypes, int num);
> 
> +/**
> + * @warning
> + * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
> + *
> + * Get the number of free descriptors in a Tx queue.
> + *
> + * This function retrieves the number of available free descriptors in a
> + * transmit queue. Applications can use this API in the fast path to inspect
> + * Tx queue occupancy and take appropriate actions based on the available
> + * free descriptors. An example action could be implementing the
> + * Random Early Discard (RED).
> + *
> + * If there are no packets in the Tx queue, the function returns the value
> + * of `nb_tx_desc` provided during the initialization of the Tx queue using
> + * rte_eth_tx_queue_setup(), signifying that all descriptors are free.
> + *
> + * @param port_id
> + *   The port identifier of the device.
> + * @param tx_queue_id
> + *   The index of the transmit queue.
> + *   The value must be in the range [0, nb_tx_queue - 1] previously supplied
> + *   to rte_eth_dev_configure().
> + * @return
> + *   - (<= UINT16_MAX) Number of free descriptors in a Tx queue
> + *   - (> UINT16_MAX) if error. Enabled only when RTE_ETHDEV_DEBUG_TX is
> enabled
> + *
> + * @note This function is designed for fast-path use.
> + *
> + */
> +__rte_experimental
> +static inline uint32_t
> +rte_eth_tx_queue_free_desc_get(uint16_t port_id, uint16_t tx_queue_id)
> +{
> +	struct rte_eth_fp_ops *fops;
> +	uint32_t rc;
> +	void *qd;
> +
> +#ifdef RTE_ETHDEV_DEBUG_TX
> +	rc = UINT32_MAX;
> +	if (port_id >= RTE_MAX_ETHPORTS || tx_queue_id >=
> RTE_MAX_QUEUES_PER_PORT) {
> +		RTE_ETHDEV_LOG(ERR, "Invalid port_id=%u or
> tx_queue_id=%u\n",
> +				port_id, tx_queue_id);
> +
> +		rte_eth_trace_tx_queue_free_desc_get(port_id, tx_queue_id,
> rc);
> +		return rc;
> +	}
> +#endif
> +
> +	/* Fetch pointer to Tx queue data */
> +	fops = &rte_eth_fp_ops[port_id];
> +	qd = fops->txq.data[tx_queue_id];
> +
> +#ifdef RTE_ETHDEV_DEBUG_TX
> +	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, 0);
> +
> +	if (qd == NULL) {
> +		RTE_ETHDEV_LOG(ERR, "Invalid Tx queue_id=%u for
> port_id=%u\n",
> +				tx_queue_id, port_id);
> +
> +		rte_eth_trace_tx_queue_free_desc_get(port_id, tx_queue_id,
> rc);
> +		return rc;
> +	}
> +
> +	if (fops->tx_queue_free_desc_get == NULL) {
> +		RTE_ETHDEV_LOG(ERR, "tx_queue_free_desc_get callback not
> implementedd Tx queue_id=%u for port_id=%u\n",
> +				tx_queue_id, port_id);
> +
> +		rte_eth_trace_tx_queue_free_desc_get(port_id, tx_queue_id,
> rc);
> +		return rc;
> +	}
> +#endif
> +	rc = fops->tx_queue_free_desc_get(qd);
> +
> +	rte_eth_trace_tx_queue_free_desc_get(port_id, tx_queue_id, rc);
> +
> +	return rc;
> +}
> +
>  #ifdef __cplusplus
>  }
>  #endif
> diff --git a/lib/ethdev/rte_ethdev_core.h b/lib/ethdev/rte_ethdev_core.h
> index 4bfaf79c6c..5b7ee66ee7 100644
> --- a/lib/ethdev/rte_ethdev_core.h
> +++ b/lib/ethdev/rte_ethdev_core.h
> @@ -60,6 +60,9 @@ typedef uint16_t (*eth_recycle_tx_mbufs_reuse_t)(void
> *txq,
>  /** @internal Refill Rx descriptors with the recycling mbufs */
>  typedef void (*eth_recycle_rx_descriptors_refill_t)(void *rxq, uint16_t nb);
> 
> +/** @internal Get the number of free descriptors count of a Tx queue */
> +typedef uint16_t (*eth_tx_queue_free_desc_get_t)(void *txq);
> +
>  /**
>   * @internal
>   * Structure used to hold opaque pointers to internal ethdev Rx/Tx
> @@ -116,7 +119,9 @@ struct rte_eth_fp_ops {
>  	eth_tx_descriptor_status_t tx_descriptor_status;
>  	/** Copy used mbufs from Tx mbuf ring into Rx. */
>  	eth_recycle_tx_mbufs_reuse_t recycle_tx_mbufs_reuse;
> -	uintptr_t reserved2[2];
> +	/** Get the number of free descriptors count of a Tx queue. */
> +	eth_tx_queue_free_desc_get_t tx_queue_free_desc_get;
> +	uintptr_t reserved2[1];
>  	/**@}*/
> 
>  } __rte_cache_aligned;
> diff --git a/lib/ethdev/rte_ethdev_trace_fp.h b/lib/ethdev/rte_ethdev_trace_fp.h
> index 186271c9ff..2c57b39bd2 100644
> --- a/lib/ethdev/rte_ethdev_trace_fp.h
> +++ b/lib/ethdev/rte_ethdev_trace_fp.h
> @@ -73,6 +73,14 @@ RTE_TRACE_POINT_FP(
>  	rte_trace_point_emit_u64(count);
>  )
> 
> +RTE_TRACE_POINT_FP(
> +	rte_eth_trace_tx_queue_free_desc_get,
> +	RTE_TRACE_POINT_ARGS(uint16_t port_id, uint16_t tx_queue_id,
> uint32_t nb_free_desc),
> +	rte_trace_point_emit_u16(port_id);
> +	rte_trace_point_emit_u16(tx_queue_id);
> +	rte_trace_point_emit_u32(nb_free_desc);
> +)
> +
>  #ifdef __cplusplus
>  }
>  #endif
> --
> 2.43.0


Hi Jerin,

I think having an API to get the number of free descriptors per queue is a good idea. Why have it only for TX queues and not for RX queues as well?

Regards,
Cristian