[dpdk-dev] Why do we need iommu=pt?

[Alex Markuze]

Danny, you are correct in that DPDK shouldn't see any performance
degradation when IOMMU is present.
I was rather presenting an existing issue with kernel drivers
(bnx2x,ixgbe,mlx4, etc...) that create new RX/TX descriptors for each
packet and perform lots of map/unmap operations.

On Wed, Oct 22, 2014 at 6:21 PM, Zhou, Danny <danny.zhou at intel.com> wrote:

> Echo Cunming and we did not see obvious performance impact when iommu = pt
> is used despite of
> igb_uio or VFIO is used.
>
> Alex,
> The map and umap operation for each e/ingress packet is done by hw rather
> than sw, so
> performance impact to DPDK should be minimum in my mind. If it actually
> impacst perf, say on 100G NIC,
> I am sure it will be resolved in next generation Intel silicon. We will be
> performing some performance
> tests with iommu = on to see any performance degradation. I cannot share
> the detailed performance
> result here on the community, but I could tell if it really bring negative
> performance impact to DPDK.
> Please stay tuned.
>
> Alex,
>
> > -----Original Message-----
> > From: Liang, Cunming
> > Sent: Wednesday, October 22, 2014 4:53 PM
> > To: alex; Zhou, Danny
> > Cc: dev at dpdk.org
> > Subject: RE: [dpdk-dev] Why do we need iommu=pt?
> >
> > I thinks it's a good point using dma_addr rather than phys_addr.
> > Without iommu, the value of them are the same.
> > With iommu, the dma_addr value equal to the iova.
> > It's not all for DPDK working with iommu but not pass through.
> >
> > We know each iova belongs to one iommu domain.
> > And each device can attach to one domain.
> > It means the iova will have coupling relationship with domain/device.
> >
> > Looking back to DPDK descriptor ring, it's all right, already coupling
> with device.
> > But if for mbuf mempool, in most cases, it's shared by multiple ports.
> > So if keeping the way, all those ports/device need to put into the same
> iommu domain.
> > And the mempool has attach to specific domain, but not just the device.
> > On this time, iommu domain no longer be transparent in DPDK.
> > Vfio provide the verbs to control domain, we still need library to
> manager such domain with mempool.
> >
> > All that overhead just make DPDK works with iommu in host, but remember
> pt always works.
> > The isolation of devices mainly for security concern.
> > If it's not necessary, pt definitely is a good choice without
> performance impact.
> >
> > For those self-implemented PMD using the DMA kernel interface to set up
> its mappings appropriately.
> > It don't require "iommu=pt". The default option "iommu=on" also works.
> >
> > > -----Original Message-----
> > > From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of alex
> > > Sent: Wednesday, October 22, 2014 3:36 PM
> > > To: Zhou, Danny
> > > Cc: dev at dpdk.org
> > > Subject: Re: [dpdk-dev] Why do we need iommu=pt?
> > >
> > > Shiva.
> > > The cost of disabling iommu=pt when intel_iommu=on is dire. DPDK won't
> work
> > > as the RX/TX descriptors will be useless.
> > > Any dam access by the device will be dropped as no dam-mapping will
> exists.
> > >
> > > Danny.
> > > The IOMMU hurts performance in kernel drivers which perform a map and
> umap
> > > operation for each e/ingress packet.
> > > The costs of unmapping when under strict protection limit a +10Gb to
> 3Gb
> > > with cpu maxed out at 100%. DPDK apps shouldn't feel any difference
> IFF the
> > > rx descriptors contain iova and not real physical addresses which are
> used
> > > currently.
> > >
> > >
> > > On Tue, Oct 21, 2014 at 10:10 PM, Zhou, Danny <danny.zhou at intel.com>
> wrote:
> > >
> > > > IMHO, if memory protection with IOMMU is needed or not really
> depends on
> > > > how you use
> > > > and deploy your DPDK based applications. For Telco network middle
> boxes,
> > > > which adopts
> > > > a "close model" solution to achieve extremely high performance, the
> entire
> > > > system including
> > > > HW, software in kernel and userspace are controlled by Telco vendors
> and
> > > > assumed trustable, so
> > > > memory protection is not so important. While for Datacenters, which
> > > > generally adopts a "open model"
> > > > solution allows running user space applications(e.g. tenant
> applications
> > > > and VMs) which could
> > > > direct access NIC and DMA engine inside the NIC using modified DPDK
> PMD
> > > > are not trustable
> > > > as they can potentially DAM to/from arbitrary memory regions using
> > > > physical addresses, so IOMMU
> > > > is needed to provide strict memory protection, at the cost of
> negative
> > > > performance impact.
> > > >
> > > > So if you want to seek high performance, disable IOMMU in BIOS or
> OS. And
> > > > if security is a major
> > > > concern, tune it on and tradeoff between performance and security.
> But I
> > > > do NOT think is comes with
> > > > an extremely high performance costs according to our performance
> > > > measurement, but it probably true
> > > > for 100G NIC.
> > > >
> > > > > -----Original Message-----
> > > > > From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of Shivapriya
> Hiremath
> > > > > Sent: Wednesday, October 22, 2014 12:54 AM
> > > > > To: Alex Markuze
> > > > > Cc: dev at dpdk.org
> > > > > Subject: Re: [dpdk-dev] Why do we need iommu=pt?
> > > > >
> > > > > Hi,
> > > > >
> > > > > Thank you for all the replies.
> > > > > I am trying to understand the impact of this on DPDK. What will be
> the
> > > > > repercussions of disabling "iommu=pt" on the DPDK performance?
> > > > >
> > > > >
> > > > > On Tue, Oct 21, 2014 at 12:32 AM, Alex Markuze <alex at weka.io>
> wrote:
> > > > >
> > > > > > DPDK uses a 1:1 mapping and doesn't support IOMMU.  IOMMU allows
> for
> > > > > > simpler VM physical address translation.
> > > > > > The second role of IOMMU is to allow protection from unwanted
> memory
> > > > > > access by an unsafe devise that has DMA privileges.
> Unfortunately this
> > > > > > protection comes with an extremely high performance costs for
> high
> > > > speed
> > > > > > nics.
> > > > > >
> > > > > > To your question iommu=pt disables IOMMU support for the
> hypervisor.
> > > > > >
> > > > > > On Tue, Oct 21, 2014 at 1:39 AM, Xie, Huawei <
> huawei.xie at intel.com>
> > > > wrote:
> > > > > >
> > > > > >>
> > > > > >>
> > > > > >> > -----Original Message-----
> > > > > >> > From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of
> Shivapriya
> > > > > >> Hiremath
> > > > > >> > Sent: Monday, October 20, 2014 2:59 PM
> > > > > >> > To: dev at dpdk.org
> > > > > >> > Subject: [dpdk-dev] Why do we need iommu=pt?
> > > > > >> >
> > > > > >> > Hi,
> > > > > >> >
> > > > > >> > My question is that if the Poll mode  driver used the DMA
> kernel
> > > > > >> interface
> > > > > >> > to set up its mappings appropriately, would it still require
> that
> > > > > >> iommu=pt
> > > > > >> > be set?
> > > > > >> > What is the purpose of setting iommu=pt ?
> > > > > >> PMD allocates memory though hugetlb file system, and fills the
> > > > physical
> > > > > >> address
> > > > > >> into the descriptor.
> > > > > >> pt is used to pass through iotlb translation. Refer to the
> below link.
> > > > > >> http://lkml.iu.edu/hypermail/linux/kernel/0906.2/02129.html
> > > > > >> >
> > > > > >> > Thank you.
> > > > > >>
> > > > > >
> > > > > >
> > > >
>