[PATCH] net/iavf: fix iavf query stats in intr thread

[Deng, KaiwenX]

> -----Original Message-----
> From: Zhang, Qi Z <qi.z.zhang at intel.com>
> Sent: Monday, February 27, 2023 8:56 AM
> To: Deng, KaiwenX <kaiwenx.deng at intel.com>; dev at dpdk.org
> Cc: stable at dpdk.org; Yang, Qiming <qiming.yang at intel.com>; Zhou, YidingX
> <yidingx.zhou at intel.com>; Wu, Jingjing <jingjing.wu at intel.com>; Xing,
> Beilei <beilei.xing at intel.com>
> Subject: RE: [PATCH] net/iavf: fix iavf query stats in intr thread
> 
> 
> 
> > -----Original Message-----
> > From: Deng, KaiwenX <kaiwenx.deng at intel.com>
> > Sent: Wednesday, February 22, 2023 12:40 PM
> > To: dev at dpdk.org
> > Cc: stable at dpdk.org; Yang, Qiming <qiming.yang at intel.com>; Zhou,
> > YidingX <yidingx.zhou at intel.com>; Deng, KaiwenX
> > <kaiwenx.deng at intel.com>; Wu, Jingjing <jingjing.wu at intel.com>; Xing,
> > Beilei <beilei.xing at intel.com>; Zhang, Qi Z <qi.z.zhang at intel.com>
> > Subject: [PATCH] net/iavf: fix iavf query stats in intr thread
> >
> > When iavf send query-stats command in eal-intr-thread through virtual
> > channel, there will be no response received from
> > iavf_dev_virtchnl_handler for this command during block and wait.
> > Because iavf_dev_virtchnl_handler is also registered in eal-intr-thread.
> >
> > When vf device is bonded as BONDING_MODE_TLB mode, the slave
> update
> > callback will registered in alarm and called by eal-intr-thread, it
> > would also raise the above issue.
> >
> > This commit add local stats return to iavf_dev_stats_get immediately
> > when it is called by eal-intr-thread. And update local stats in iavf-virtchnl-
> thread.
> >
> > Fixes: cb5c1b91f76f ("net/iavf: add thread for event callbacks")
> > Fixes: 22b123a36d07 ("net/avf: initialize PMD")
> > Cc: stable at dpdk.org
> >
> > Signed-off-by: Kaiwen Deng <kaiwenx.deng at intel.com>
> > ---
> >  drivers/net/iavf/iavf.h        |   9 ++-
> >  drivers/net/iavf/iavf_ethdev.c |  24 ++++++--
> > drivers/net/iavf/iavf_vchnl.c  |
> > 107 ++++++++++++++++++++++++---------
> >  3 files changed, 104 insertions(+), 36 deletions(-)
> >
> > diff --git a/drivers/net/iavf/iavf.h b/drivers/net/iavf/iavf.h index
> > 1edebab8dc..3a249b90a3 100644
> > --- a/drivers/net/iavf/iavf.h
> > +++ b/drivers/net/iavf/iavf.h
> > @@ -128,6 +128,7 @@ struct iavf_vsi {
> >  	uint16_t base_vector;
> >  	uint16_t msix_intr;      /* The MSIX interrupt binds to VSI */
> >  	struct iavf_eth_xstats eth_stats_offset;
> > +	struct virtchnl_eth_stats eth_stats;
> >  };
> >
> >  struct rte_flow;
> > @@ -325,6 +326,8 @@ struct iavf_adapter {
> >  	struct iavf_devargs devargs;
> >  };
> >
> > +typedef void (*virtchnl_callback)(struct rte_eth_dev *dev, void
> > +*args);
> > +
> >  /* IAVF_DEV_PRIVATE_TO */
> >  #define IAVF_DEV_PRIVATE_TO_ADAPTER(adapter) \
> >  	((struct iavf_adapter *)adapter)
> > @@ -424,8 +427,10 @@ _atomic_set_async_response_cmd(struct
> iavf_info
> > *vf, enum virtchnl_ops ops)  }  int iavf_check_api_version(struct
> > iavf_adapter *adapter);  int iavf_get_vf_resource(struct iavf_adapter
> > *adapter); -void iavf_dev_event_handler_fini(void);
> > -int iavf_dev_event_handler_init(void);
> > +void iavf_dev_virtchnl_handler_fini(void);
> > +void iavf_dev_virtchnl_callback_post(struct rte_eth_dev *dev,
> > +			 virtchnl_callback cb, void *args); int
> > +iavf_dev_virtchnl_handler_init(void);
> >  void iavf_handle_virtchnl_msg(struct rte_eth_dev *dev);  int
> > iavf_enable_vlan_strip(struct iavf_adapter *adapter);  int
> > iavf_disable_vlan_strip(struct iavf_adapter *adapter); diff --git
> > a/drivers/net/iavf/iavf_ethdev.c b/drivers/net/iavf/iavf_ethdev.c
> > index 3196210f2c..fcbab5b26a 100644
> > --- a/drivers/net/iavf/iavf_ethdev.c
> > +++ b/drivers/net/iavf/iavf_ethdev.c
> > @@ -1729,6 +1729,17 @@ iavf_update_stats(struct iavf_vsi *vsi, struct
> > virtchnl_eth_stats *nes)
> >  	iavf_stat_update_32(&oes->tx_discards, &nes->tx_discards);  }
> >
> > +static void iavf_dev_stats_get_callback(struct rte_eth_dev *dev, void
> > +*args) {
> > +	struct virtchnl_eth_stats *eth_stats = (struct virtchnl_eth_stats
> *)args;
> > +	struct virtchnl_eth_stats *pstats = NULL;
> > +	struct iavf_adapter *adapter =
> > +		IAVF_DEV_PRIVATE_TO_ADAPTER(dev->data->dev_private);
> > +	int ret = iavf_query_stats(adapter, &pstats);
> > +	if (ret == 0)
> > +		rte_memcpy(eth_stats, pstats, sizeof(struct
> > virtchnl_eth_stats)); }
> > +
> >  static int
> >  iavf_dev_stats_get(struct rte_eth_dev *dev, struct rte_eth_stats
> > *stats) { @@ -1738,8 +1749,13 @@ iavf_dev_stats_get(struct rte_eth_dev
> > *dev, struct rte_eth_stats *stats)
> >  	struct iavf_vsi *vsi = &vf->vsi;
> >  	struct virtchnl_eth_stats *pstats = NULL;
> >  	int ret;
> > -
> > -	ret = iavf_query_stats(adapter, &pstats);
> > +	if (rte_thread_is_intr()) {
> > +		pstats = &vsi->eth_stats;
> > +		iavf_dev_virtchnl_callback_post(dev,
> > iavf_dev_stats_get_callback, (void *)pstats);
> > +		ret = 0;
> 
> 
> I assume this is going to return a copy of outdated stats, then the stats will be
> updated it in future and this happens in another thread, is it correct?
> If true, then will be 2 issues.
> 1. How the caller know when the updated data will be ready?
> 2. even caller will consume the outdated stats directly, is that will be a race
> condition if the virtchnl thread update the stats at the same time?
>> Hi，Qi
>>1.The caller cannot know when the updated data will be ready in interrupt thread. if caller block for stats update in interrupt thread,
>>    the stats will not be updated, because interrupt  thread is blocked. The caller can only get local stats which is updated last time.
>>    I can't find any other better solution. If you have any suggestions, please let me know.
>>2.I submit patch v2 to avoid race condition.
>>Thanks
> 
> > +	} else {
> > +		ret = iavf_query_stats(adapter, &pstats);
> > +	}
> >  	if (ret == 0) {
> >  		uint8_t crc_stats_len = (dev->data-
> > >dev_conf.rxmode.offloads &
> >  					 RTE_ETH_RX_OFFLOAD_KEEP_CRC) ?
> > 0 :