[PATCH v2] examples/vhost: fix retry logic on eth rx path

[Xia, Chenbo]

> -----Original Message-----
> From: Wang, YuanX <yuanx.wang at intel.com>
> Sent: Monday, June 20, 2022 5:19 PM
> To: Xia, Chenbo <chenbo.xia at intel.com>; Hu, Jiayu <jiayu.hu at intel.com>;
> David Marchand <david.marchand at redhat.com>; maxime.coquelin at redhat.com
> Cc: dev at dpdk.org; He, Xingguang <xingguang.he at intel.com>; stable at dpdk.org;
> Ling, WeiX <weix.ling at intel.com>; jin.liu at corigine.com;
> louis.peens at corigine.com; peng.zhang at corigine.com; Heinrich Kuhn
> <heinrich.kuhn at corigine.com>
> Subject: RE: [PATCH v2] examples/vhost: fix retry logic on eth rx path
> 
> 
> 
> > -----Original Message-----
> > From: Xia, Chenbo <chenbo.xia at intel.com>
> > Sent: Monday, June 20, 2022 5:10 PM
> > To: Hu, Jiayu <jiayu.hu at intel.com>; David Marchand
> > <david.marchand at redhat.com>; maxime.coquelin at redhat.com
> > Cc: Wang, YuanX <yuanx.wang at intel.com>; dev at dpdk.org; He, Xingguang
> > <xingguang.he at intel.com>; stable at dpdk.org; Ling, WeiX
> > <weix.ling at intel.com>; jin.liu at corigine.com; louis.peens at corigine.com;
> > peng.zhang at corigine.com; Heinrich Kuhn <heinrich.kuhn at corigine.com>
> > Subject: RE: [PATCH v2] examples/vhost: fix retry logic on eth rx path
> >
> > > -----Original Message-----
> > > From: Hu, Jiayu <jiayu.hu at intel.com>
> > > Sent: Monday, June 20, 2022 4:59 PM
> > > To: Xia, Chenbo <chenbo.xia at intel.com>; David Marchand
> > > <david.marchand at redhat.com>; maxime.coquelin at redhat.com
> > > Cc: Wang, YuanX <yuanx.wang at intel.com>; dev at dpdk.org; He, Xingguang
> > > <xingguang.he at intel.com>; stable at dpdk.org; Ling, WeiX
> > > <weix.ling at intel.com>; jin.liu at corigine.com; louis.peens at corigine.com;
> > > peng.zhang at corigine.com; Heinrich Kuhn <heinrich.kuhn at corigine.com>
> > > Subject: RE: [PATCH v2] examples/vhost: fix retry logic on eth rx path
> > >
> > >
> > >
> > > > -----Original Message-----
> > > > From: Xia, Chenbo <chenbo.xia at intel.com>
> > > > Sent: Monday, June 20, 2022 3:49 PM
> > > > To: David Marchand <david.marchand at redhat.com>;
> > > > maxime.coquelin at redhat.com
> > > > Cc: Wang, YuanX <yuanx.wang at intel.com>; dev at dpdk.org; Hu, Jiayu
> > > > <jiayu.hu at intel.com>; He, Xingguang <xingguang.he at intel.com>;
> > > > stable at dpdk.org; Ling, WeiX <weix.ling at intel.com>;
> > > > jin.liu at corigine.com; louis.peens at corigine.com;
> > > > peng.zhang at corigine.com; Heinrich Kuhn <heinrich.kuhn at corigine.com>
> > > > Subject: RE: [PATCH v2] examples/vhost: fix retry logic on eth rx
> > > > path
> > > >
> > > > > -----Original Message-----
> > > > > From: David Marchand <david.marchand at redhat.com>
> > > > > Sent: Monday, June 20, 2022 3:36 PM
> > > > > To: Xia, Chenbo <chenbo.xia at intel.com>;
> > maxime.coquelin at redhat.com
> > > > > Cc: Wang, YuanX <yuanx.wang at intel.com>; dev at dpdk.org; Hu, Jiayu
> > > > > <jiayu.hu at intel.com>; He, Xingguang <xingguang.he at intel.com>;
> > > > > stable at dpdk.org; Ling, WeiX <weix.ling at intel.com>;
> > > > > jin.liu at corigine.com; louis.peens at corigine.com;
> > > > > peng.zhang at corigine.com; Heinrich Kuhn
> > > > > <heinrich.kuhn at corigine.com>
> > > > > Subject: Re: [PATCH v2] examples/vhost: fix retry logic on eth rx
> > > > > path
> > > > >
> > > > > On Mon, Jun 20, 2022 at 5:20 AM Xia, Chenbo <chenbo.xia at intel.com>
> > > > wrote:
> > > > > > > drain_eth_rx() uses rte_vhost_avail_entries() to calculate the
> > > > > > > available entries to determine if a retry is required.
> > > > > > > However, this function only works with split rings, and
> > > > > > > calculating packed rings will return the wrong value and cause
> > > > > > > unnecessary retries resulting in a significant performance
> penalty.
> > > > > > >
> > > > > > > This patch fix that by using the difference between tx/rx
> > > > > > > burst as the retry condition.
> > > > > >
> > > > > > Does it mean we don't need the API rte_vhost_avail_entries()
> > anymore?
> > > > > >
> > > > > > Jiayu/Yuan/Maxime, what do you think?
> > > > >
> > > > > FWIW, I still see a user:
> > > > > virtio-forwarder/virtio_vhostuser.c:     * This check ensures that
> we
> > > > > do not call rte_vhost_avail_entries
> > > > > virtio-forwarder/virtio_worker.c:        try_rcv =
> > > > > rte_vhost_avail_entries((int)relay->vio.vio_dev,
> > > > >
> > > > > Cc'd a few Corigine guys.
> > > >
> > > > Thanks David for this info! Then I guess only split ring is used in
> > > > this
> > > use case?
> > > > If we want to keep it, then this API should also be fixed as it's
> > > > not
> > > supporting
> > > > packed ring.
> > >
> > > Same issue for rte_vhost_rx_queue_count(), and it is used in OVS.
> > >
> > > But if look into the implementation of rte_vhost_avail_entries(), it
> > > calculates the number of available descriptors by " vq->avail->idx -
> > > vq-
> > > >last_used_idx".
> > > This logic looks strange. Anyone knows the reason of this
> implementation?
> >
> > I was not in the history, but as I checked the git log. Seems it's
> because in this
> > commit, this API was not improved (This API is introduced before the
> > commit).
> >
> > commit f6be82d7259ee35683721092d61283d99a47aff1
> > Author: Yuanhan Liu <yuanhan.liu at linux.intel.com>
> > Date:   Sun Oct 9 15:27:56 2016 +0800
> >
> >     vhost: introduce last available index for dequeue
> >
> >     So far, we retrieve both the used ring and avail ring idx by the var
> >     last_used_idx; it won't be a problem because the used ring is
> updated
> >     immediately after those avail entries are consumed.
> >
> >     But that's not true when dequeue zero copy is enabled, that used
> ring is
> >     updated only when the mbuf is consumed. Thus, we need use another
> var
> > to
> >     note the last avail ring idx we have consumed.
> >
> >     Therefore, last_avail_idx is introduced.
> >
> >     Signed-off-by: Yuanhan Liu <yuanhan.liu at linux.intel.com>
> >     Reviewed-by: Maxime Coquelin <maxime.coquelin at redhat.com>
> >     Tested-by: Qian Xu <qian.q.xu at intel.com>
> >
> 
> It was introduced by this commit.

Yes, of course both last_XXX_idx are introduced for split ring. I was saying
the story seems to be:

At first, vhost usage is very trivial, get available and set used, so one idx
is enough. Then dequeue_zero_copy comes (later removed) and you may not update
used after you get avail in one func call. Then last_avail_idx is introduced but
rte_vhost_avail_entries() is not updated. And when we introduced packed ring,
this API is also not updated.

Correct me if I misunderstand the story.

Thanks,
Chenbo

> 
> commit 7202b0a8240158b317665c20525f81d55f16f602
> Author: Huawei Xie <huawei.xie at intel.com>
> Date:   Thu Oct 9 02:54:51 2014 +0800
> 
>     vhost: get available vring entries
> 
>     Signed-off-by: Huawei Xie <huawei.xie at intel.com>
>     Acked-by: Changchun Ouyang <changchun.ouyang at intel.com>
>     [Thomas: split patch]
> 
> Check for the define of VQ, it is obvious for split ring.
> 
> struct vhost_virtqueue {
> 	struct vring_desc	*desc;			/**< Virtqueue descriptor
> ring. */
> 	struct vring_avail	*avail;			/**< Virtqueue
> available ring. */
> 	struct vring_used	*used;			/**< Virtqueue used ring.
> */
> 	uint32_t		size;			/**< Size of descriptor ring. */
> 	int			backend;		/**< Backend value to determine
> if device should started/stopped. */
> 	uint16_t		vhost_hlen;		/**< Vhost header length (varies
> depending on RX merge buffers. */
> 	volatile uint16_t	last_used_idx;		/**< Last index used on
> the available ring */
> 	volatile uint16_t	last_used_idx_res;	/**< Used for multiple
> devices reserving buffers. */
> #define VIRTIO_INVALID_EVENTFD		(-1)
> #define VIRTIO_UNINITIALIZED_EVENTFD	(-2)
> 	int			callfd;			/**< Used to notify the
> guest (trigger interrupt). */
> 	int			kickfd;			/**< Currently unused as
> polling mode is enabled. */
> 	int			enabled;
> 	uint64_t		log_guest_addr;		/**< Physical address of
> used ring, for logging */
> 	uint64_t		reserved[15];		/**< Reserve some spaces
> for future extension. */
> 	struct buf_vector	buf_vec[BUF_VECTOR_MAX];	/**< for scatter RX.
> */
> } __rte_cache_aligned;
> 
> Thanks,
> Yuan
> 
> > Thanks,
> > Chenbo
> >
> > >
> > > Thanks,
> > > Jiayu
> > >
> > > >
> > > > Thanks,
> > > > Chenbo
> > > >
> > > > >
> > > > >
> > > > > --
> > > > > David Marchand
> > > >
> > >