[v2] vhost: avoid iotlb mempool allocation while IOMMU disabled
Checks
Commit Message
If vhost device's IOMMU feature is disabled, iotlb mempool allocation
is unnecessary.
Reported-by: Peng He <hepeng.0320@bytedance.com>
Signed-off-by: Wan Junjie <wanjunjie@bytedance.com>
Reviewed-by: Zhihong Wang <wangzhihong.wzh@bytedance.com>
---
lib/librte_vhost/vhost.c | 4 ++--
lib/librte_vhost/vhost_user.c | 6 +++++-
2 files changed, 7 insertions(+), 3 deletions(-)
Comments
> -----Original Message-----
> From: Wan Junjie <wanjunjie@bytedance.com>
> Sent: Tuesday, February 2, 2021 4:15 PM
> To: Xia, Chenbo <chenbo.xia@intel.com>; Maxime Coquelin
> <maxime.coquelin@redhat.com>
> Cc: dev@dpdk.org; Wan Junjie <wanjunjie@bytedance.com>; Peng He
> <hepeng.0320@bytedance.com>; Zhihong Wang <wangzhihong.wzh@bytedance.com>
> Subject: [PATCH v2] vhost: avoid iotlb mempool allocation while IOMMU disabled
>
> If vhost device's IOMMU feature is disabled, iotlb mempool allocation
> is unnecessary.
>
> Reported-by: Peng He <hepeng.0320@bytedance.com>
> Signed-off-by: Wan Junjie <wanjunjie@bytedance.com>
> Reviewed-by: Zhihong Wang <wangzhihong.wzh@bytedance.com>
> ---
> 2.11.0
Patch applied to next-virtio/main with conflict resolved, Thanks
On Wed, Apr 7, 2021 at 9:10 AM Xia, Chenbo <chenbo.xia@intel.com> wrote:
>
> > -----Original Message-----
> > From: Wan Junjie <wanjunjie@bytedance.com>
> > Sent: Tuesday, February 2, 2021 4:15 PM
> > To: Xia, Chenbo <chenbo.xia@intel.com>; Maxime Coquelin
> > <maxime.coquelin@redhat.com>
> > Cc: dev@dpdk.org; Wan Junjie <wanjunjie@bytedance.com>; Peng He
> > <hepeng.0320@bytedance.com>; Zhihong Wang <wangzhihong.wzh@bytedance.com>
> > Subject: [PATCH v2] vhost: avoid iotlb mempool allocation while IOMMU disabled
> >
> > If vhost device's IOMMU feature is disabled, iotlb mempool allocation
> > is unnecessary.
> >
> > Reported-by: Peng He <hepeng.0320@bytedance.com>
> > Signed-off-by: Wan Junjie <wanjunjie@bytedance.com>
> > Reviewed-by: Zhihong Wang <wangzhihong.wzh@bytedance.com>
> > ---
> > 2.11.0
>
> Patch applied to next-virtio/main with conflict resolved, Thanks
This patch triggered a regression reported by Red Hat QE.
https://bugs.dpdk.org/show_bug.cgi?id=703
Reverting the patch is enough to fix the regression.
I'll let you guys decide on the next step for 21.05.
> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> Sent: Thursday, May 13, 2021 4:39 PM
> To: Xia, Chenbo <chenbo.xia@intel.com>; Maxime Coquelin
> <maxime.coquelin@redhat.com>
> Cc: Wan Junjie <wanjunjie@bytedance.com>; dev@dpdk.org; Peng He
> <hepeng.0320@bytedance.com>; Zhihong Wang <wangzhihong.wzh@bytedance.com>
> Subject: Re: [dpdk-dev] [PATCH v2] vhost: avoid iotlb mempool allocation while
> IOMMU disabled
>
> On Wed, Apr 7, 2021 at 9:10 AM Xia, Chenbo <chenbo.xia@intel.com> wrote:
> >
> > > -----Original Message-----
> > > From: Wan Junjie <wanjunjie@bytedance.com>
> > > Sent: Tuesday, February 2, 2021 4:15 PM
> > > To: Xia, Chenbo <chenbo.xia@intel.com>; Maxime Coquelin
> > > <maxime.coquelin@redhat.com>
> > > Cc: dev@dpdk.org; Wan Junjie <wanjunjie@bytedance.com>; Peng He
> > > <hepeng.0320@bytedance.com>; Zhihong Wang <wangzhihong.wzh@bytedance.com>
> > > Subject: [PATCH v2] vhost: avoid iotlb mempool allocation while IOMMU
> disabled
> > >
> > > If vhost device's IOMMU feature is disabled, iotlb mempool allocation
> > > is unnecessary.
> > >
> > > Reported-by: Peng He <hepeng.0320@bytedance.com>
> > > Signed-off-by: Wan Junjie <wanjunjie@bytedance.com>
> > > Reviewed-by: Zhihong Wang <wangzhihong.wzh@bytedance.com>
> > > ---
> > > 2.11.0
> >
> > Patch applied to next-virtio/main with conflict resolved, Thanks
>
> This patch triggered a regression reported by Red Hat QE.
> https://bugs.dpdk.org/show_bug.cgi?id=703
>
> Reverting the patch is enough to fix the regression.
>
> I'll let you guys decide on the next step for 21.05.
>
>
> --
> David Marchand
I cannot reproduce this issue on my machine.
@Wan Junjie Could you reproduce this? I prefer to revert the patch if things
are not very clear as it is only a minor optimization.
Thanks,
Chenbo
On Thu, May 13, 2021 at 6:24 PM Xia, Chenbo <chenbo.xia@intel.com> wrote:
>
> > -----Original Message-----
> > From: David Marchand <david.marchand@redhat.com>
> > Sent: Thursday, May 13, 2021 4:39 PM
> > To: Xia, Chenbo <chenbo.xia@intel.com>; Maxime Coquelin
> > <maxime.coquelin@redhat.com>
> > Cc: Wan Junjie <wanjunjie@bytedance.com>; dev@dpdk.org; Peng He
> > <hepeng.0320@bytedance.com>; Zhihong Wang <wangzhihong.wzh@bytedance.com>
> > Subject: Re: [dpdk-dev] [PATCH v2] vhost: avoid iotlb mempool allocation while
> > IOMMU disabled
> >
> > On Wed, Apr 7, 2021 at 9:10 AM Xia, Chenbo <chenbo.xia@intel.com> wrote:
> > >
> > > > -----Original Message-----
> > > > From: Wan Junjie <wanjunjie@bytedance.com>
> > > > Sent: Tuesday, February 2, 2021 4:15 PM
> > > > To: Xia, Chenbo <chenbo.xia@intel.com>; Maxime Coquelin
> > > > <maxime.coquelin@redhat.com>
> > > > Cc: dev@dpdk.org; Wan Junjie <wanjunjie@bytedance.com>; Peng He
> > > > <hepeng.0320@bytedance.com>; Zhihong Wang <wangzhihong.wzh@bytedance.com>
> > > > Subject: [PATCH v2] vhost: avoid iotlb mempool allocation while IOMMU
> > disabled
> > > >
> > > > If vhost device's IOMMU feature is disabled, iotlb mempool allocation
> > > > is unnecessary.
> > > >
> > > > Reported-by: Peng He <hepeng.0320@bytedance.com>
> > > > Signed-off-by: Wan Junjie <wanjunjie@bytedance.com>
> > > > Reviewed-by: Zhihong Wang <wangzhihong.wzh@bytedance.com>
> > > > ---
> > > > 2.11.0
> > >
> > > Patch applied to next-virtio/main with conflict resolved, Thanks
> >
> > This patch triggered a regression reported by Red Hat QE.
> > https://bugs.dpdk.org/show_bug.cgi?id=703
> >
> > Reverting the patch is enough to fix the regression.
> >
> > I'll let you guys decide on the next step for 21.05.
> >
> >
> > --
> > David Marchand
>
> I cannot reproduce this issue on my machine.
>
> @Wan Junjie Could you reproduce this? I prefer to revert the patch if things
> are not very clear as it is only a minor optimization.
>
> Thanks,
> Chenbo
>
>
Hi Chenbo,
I am not available for this now. Please revert it.
Will dig it later.
Thanks,
Junjie
> -----Original Message-----
> From: Wan Junjie <wanjunjie@bytedance.com>
> Sent: Thursday, May 13, 2021 6:52 PM
> To: Xia, Chenbo <chenbo.xia@intel.com>
> Cc: David Marchand <david.marchand@redhat.com>; Maxime Coquelin
> <maxime.coquelin@redhat.com>; dev@dpdk.org; Peng He
> <hepeng.0320@bytedance.com>; Zhihong Wang <wangzhihong.wzh@bytedance.com>;
> Thomas Monjalon <thomas@monjalon.net>
> Subject: Re: [External] RE: [dpdk-dev] [PATCH v2] vhost: avoid iotlb mempool
> allocation while IOMMU disabled
>
> On Thu, May 13, 2021 at 6:24 PM Xia, Chenbo <chenbo.xia@intel.com> wrote:
> >
> > > -----Original Message-----
> > > From: David Marchand <david.marchand@redhat.com>
> > > Sent: Thursday, May 13, 2021 4:39 PM
> > > To: Xia, Chenbo <chenbo.xia@intel.com>; Maxime Coquelin
> > > <maxime.coquelin@redhat.com>
> > > Cc: Wan Junjie <wanjunjie@bytedance.com>; dev@dpdk.org; Peng He
> > > <hepeng.0320@bytedance.com>; Zhihong Wang <wangzhihong.wzh@bytedance.com>
> > > Subject: Re: [dpdk-dev] [PATCH v2] vhost: avoid iotlb mempool allocation
> while
> > > IOMMU disabled
> > >
> > > On Wed, Apr 7, 2021 at 9:10 AM Xia, Chenbo <chenbo.xia@intel.com> wrote:
> > > >
> > > > > -----Original Message-----
> > > > > From: Wan Junjie <wanjunjie@bytedance.com>
> > > > > Sent: Tuesday, February 2, 2021 4:15 PM
> > > > > To: Xia, Chenbo <chenbo.xia@intel.com>; Maxime Coquelin
> > > > > <maxime.coquelin@redhat.com>
> > > > > Cc: dev@dpdk.org; Wan Junjie <wanjunjie@bytedance.com>; Peng He
> > > > > <hepeng.0320@bytedance.com>; Zhihong Wang
> <wangzhihong.wzh@bytedance.com>
> > > > > Subject: [PATCH v2] vhost: avoid iotlb mempool allocation while IOMMU
> > > disabled
> > > > >
> > > > > If vhost device's IOMMU feature is disabled, iotlb mempool allocation
> > > > > is unnecessary.
> > > > >
> > > > > Reported-by: Peng He <hepeng.0320@bytedance.com>
> > > > > Signed-off-by: Wan Junjie <wanjunjie@bytedance.com>
> > > > > Reviewed-by: Zhihong Wang <wangzhihong.wzh@bytedance.com>
> > > > > ---
> > > > > 2.11.0
> > > >
> > > > Patch applied to next-virtio/main with conflict resolved, Thanks
> > >
> > > This patch triggered a regression reported by Red Hat QE.
> > > https://bugs.dpdk.org/show_bug.cgi?id=703
> > >
>
>
> > > Reverting the patch is enough to fix the regression.
> > >
> > > I'll let you guys decide on the next step for 21.05.
> > >
> > >
> > > --
> > > David Marchand
> >
> > I cannot reproduce this issue on my machine.
> >
> > @Wan Junjie Could you reproduce this? I prefer to revert the patch if things
> > are not very clear as it is only a minor optimization.
> >
> > Thanks,
> > Chenbo
> >
> >
>
> Hi Chenbo,
>
> I am not available for this now. Please revert it.
> Will dig it later.
>
> Thanks,
> Junjie
Luckily I can reproduce now. I will take a look.
Chenbo
Hi,
> -----Original Message-----
> From: Wan Junjie <wanjunjie@bytedance.com>
> Sent: Thursday, May 13, 2021 6:52 PM
> To: Xia, Chenbo <chenbo.xia@intel.com>
> Cc: David Marchand <david.marchand@redhat.com>; Maxime Coquelin
> <maxime.coquelin@redhat.com>; dev@dpdk.org; Peng He
> <hepeng.0320@bytedance.com>; Zhihong Wang <wangzhihong.wzh@bytedance.com>;
> Thomas Monjalon <thomas@monjalon.net>
> Subject: Re: [External] RE: [dpdk-dev] [PATCH v2] vhost: avoid iotlb mempool
> allocation while IOMMU disabled
>
> On Thu, May 13, 2021 at 6:24 PM Xia, Chenbo <chenbo.xia@intel.com> wrote:
> >
> > > -----Original Message-----
> > > From: David Marchand <david.marchand@redhat.com>
> > > Sent: Thursday, May 13, 2021 4:39 PM
> > > To: Xia, Chenbo <chenbo.xia@intel.com>; Maxime Coquelin
> > > <maxime.coquelin@redhat.com>
> > > Cc: Wan Junjie <wanjunjie@bytedance.com>; dev@dpdk.org; Peng He
> > > <hepeng.0320@bytedance.com>; Zhihong Wang <wangzhihong.wzh@bytedance.com>
> > > Subject: Re: [dpdk-dev] [PATCH v2] vhost: avoid iotlb mempool allocation
> while
> > > IOMMU disabled
> > >
> > > On Wed, Apr 7, 2021 at 9:10 AM Xia, Chenbo <chenbo.xia@intel.com> wrote:
> > > >
> > > > > -----Original Message-----
> > > > > From: Wan Junjie <wanjunjie@bytedance.com>
> > > > > Sent: Tuesday, February 2, 2021 4:15 PM
> > > > > To: Xia, Chenbo <chenbo.xia@intel.com>; Maxime Coquelin
> > > > > <maxime.coquelin@redhat.com>
> > > > > Cc: dev@dpdk.org; Wan Junjie <wanjunjie@bytedance.com>; Peng He
> > > > > <hepeng.0320@bytedance.com>; Zhihong Wang
> <wangzhihong.wzh@bytedance.com>
> > > > > Subject: [PATCH v2] vhost: avoid iotlb mempool allocation while IOMMU
> > > disabled
> > > > >
> > > > > If vhost device's IOMMU feature is disabled, iotlb mempool allocation
> > > > > is unnecessary.
> > > > >
> > > > > Reported-by: Peng He <hepeng.0320@bytedance.com>
> > > > > Signed-off-by: Wan Junjie <wanjunjie@bytedance.com>
> > > > > Reviewed-by: Zhihong Wang <wangzhihong.wzh@bytedance.com>
> > > > > ---
> > > > > 2.11.0
> > > >
> > > > Patch applied to next-virtio/main with conflict resolved, Thanks
> > >
> > > This patch triggered a regression reported by Red Hat QE.
> > > https://bugs.dpdk.org/show_bug.cgi?id=703
> > >
>
>
> > > Reverting the patch is enough to fix the regression.
> > >
> > > I'll let you guys decide on the next step for 21.05.
> > >
> > >
> > > --
> > > David Marchand
> >
> > I cannot reproduce this issue on my machine.
> >
> > @Wan Junjie Could you reproduce this? I prefer to revert the patch if things
> > are not very clear as it is only a minor optimization.
> >
> > Thanks,
> > Chenbo
> >
> >
>
> Hi Chenbo,
>
> I am not available for this now. Please revert it.
> Will dig it later.
>
> Thanks,
> Junjie
I know the root cause, it is because when multi-queue is on, qemu only enable the
first queue pair and does not 'VHOST_USER_SET_VRING_NUM' for other queue pairs, which leads
to disabled queue pairs' iotlb_pool not init-ed. Then comes the iotlb message that operate
on all queue pairs iotlb pool.
A fix is coming.
Thanks,
Chenbo
@@ -352,7 +352,8 @@ free_vq(struct virtio_net *dev, struct vhost_virtqueue *vq)
vhost_free_async_mem(vq);
}
rte_free(vq->batch_copy_elems);
- rte_mempool_free(vq->iotlb_pool);
+ if (vq->iotlb_pool)
+ rte_mempool_free(vq->iotlb_pool);
rte_free(vq);
}
@@ -556,7 +557,6 @@ init_vring_queue(struct virtio_net *dev, uint32_t vring_idx)
vq->callfd = VIRTIO_UNINITIALIZED_EVENTFD;
vq->notif_enable = VIRTIO_UNINITIALIZED_NOTIF;
- vhost_user_iotlb_init(dev, vring_idx);
/* Backends are set to -1 indicating an inactive device. */
vq->backend = -1;
}
@@ -470,6 +470,10 @@ vhost_user_set_vring_num(struct virtio_net **pdev,
return RTE_VHOST_MSG_RESULT_ERR;
}
+ if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM)) {
+ if (vhost_user_iotlb_init(dev, msg->payload.state.index))
+ return RTE_VHOST_MSG_RESULT_ERR;
+ }
return RTE_VHOST_MSG_RESULT_OK;
}
@@ -575,7 +579,7 @@ numa_realloc(struct virtio_net *dev, int index)
dev->virtqueue[index] = vq;
vhost_devices[dev->vid] = dev;
- if (old_vq != vq)
+ if (old_vq != vq && (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM)))
vhost_user_iotlb_init(dev, index);
return dev;