[dpdk-dev,v7,7/9] linuxapp/eal_vfio: honor iova mode before mapping
Checks
Commit Message
Check iova mode and accordingly map iova to pa or va.
Signed-off-by: Santosh Shukla <santosh.shukla@caviumnetworks.com>
Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
lib/librte_eal/linuxapp/eal/eal_vfio.c | 10 ++++++++--
1 file changed, 8 insertions(+), 2 deletions(-)
Comments
> From: Santosh Shukla [mailto:santosh.shukla@caviumnetworks.com]
> Sent: Thursday, August 31, 2017 4:26 AM
> To: dev@dpdk.org
> Cc: thomas@monjalon.net; jerin.jacob@caviumnetworks.com;
> hemant.agrawal@nxp.com; olivier.matz@6wind.com;
> maxime.coquelin@redhat.com; Gonzalez Monroy, Sergio
> <sergio.gonzalez.monroy@intel.com>; Richardson, Bruce
> <bruce.richardson@intel.com>; shreyansh.jain@nxp.com;
> gaetan.rivet@6wind.com; Burakov, Anatoly <anatoly.burakov@intel.com>;
> stephen@networkplumber.org; aconole@redhat.com; Santosh Shukla
> <santosh.shukla@caviumnetworks.com>
> Subject: [PATCH v7 7/9] linuxapp/eal_vfio: honor iova mode before mapping
>
> Check iova mode and accordingly map iova to pa or va.
>
> Signed-off-by: Santosh Shukla <santosh.shukla@caviumnetworks.com>
> Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
> ---
Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
Hi @all
I just stumbled upon this patch while testing on POWER. RTE_IOVA_VA will
not work for the sPAPR code since the dma window size is currently
determined by the physical address only. I'm preparing a patch to address
this.
Thanks,
Jonas
"dev" <dev-bounces@dpdk.org> wrote on 08/31/2017 05:26:16 AM:
> From: Santosh Shukla <santosh.shukla@caviumnetworks.com>
> To: dev@dpdk.org
> Cc: thomas@monjalon.net, jerin.jacob@caviumnetworks.com,
> hemant.agrawal@nxp.com, olivier.matz@6wind.com,
> maxime.coquelin@redhat.com, sergio.gonzalez.monroy@intel.com,
> bruce.richardson@intel.com, shreyansh.jain@nxp.com,
> gaetan.rivet@6wind.com, anatoly.burakov@intel.com,
> stephen@networkplumber.org, aconole@redhat.com, Santosh Shukla
> <santosh.shukla@caviumnetworks.com>
> Date: 08/31/2017 05:28 AM
> Subject: [dpdk-dev] [PATCH v7 7/9] linuxapp/eal_vfio: honor iova
> mode before mapping
> Sent by: "dev" <dev-bounces@dpdk.org>
>
> Check iova mode and accordingly map iova to pa or va.
>
> Signed-off-by: Santosh Shukla <santosh.shukla@caviumnetworks.com>
> Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
> ---
> lib/librte_eal/linuxapp/eal/eal_vfio.c | 10 ++++++++--
> 1 file changed, 8 insertions(+), 2 deletions(-)
>
> diff --git a/lib/librte_eal/linuxapp/eal/eal_vfio.c b/lib/
> librte_eal/linuxapp/eal/eal_vfio.c
> index c8a97b7e7..b32cd09a2 100644
> --- a/lib/librte_eal/linuxapp/eal/eal_vfio.c
> +++ b/lib/librte_eal/linuxapp/eal/eal_vfio.c
> @@ -706,7 +706,10 @@ vfio_type1_dma_map(int vfio_container_fd)
> dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
> dma_map.vaddr = ms[i].addr_64;
> dma_map.size = ms[i].len;
> - dma_map.iova = ms[i].phys_addr;
> + if (rte_eal_iova_mode() == RTE_IOVA_VA)
> + dma_map.iova = dma_map.vaddr;
> + else
> + dma_map.iova = ms[i].phys_addr;
> dma_map.flags = VFIO_DMA_MAP_FLAG_READ | VFIO_DMA_MAP_FLAG_WRITE;
>
> ret = ioctl(vfio_container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);
> @@ -792,7 +795,10 @@ vfio_spapr_dma_map(int vfio_container_fd)
> dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
> dma_map.vaddr = ms[i].addr_64;
> dma_map.size = ms[i].len;
> - dma_map.iova = ms[i].phys_addr;
> + if (rte_eal_iova_mode() == RTE_IOVA_VA)
> + dma_map.iova = dma_map.vaddr;
> + else
> + dma_map.iova = ms[i].phys_addr;
> dma_map.flags = VFIO_DMA_MAP_FLAG_READ |
> VFIO_DMA_MAP_FLAG_WRITE;
>
> --
> 2.13.0
>
Hi
26/10/2017 14:57, Jonas Pfefferle1:
>
> Hi @all
>
> I just stumbled upon this patch while testing on POWER. RTE_IOVA_VA will
> not work for the sPAPR code since the dma window size is currently
> determined by the physical address only.
Is it affecting POWER8?
> I'm preparing a patch to address this.
Any news?
Can you use virtual addresses?
Thomas Monjalon <thomas@monjalon.net> wrote on 11/02/2017 11:17:10 AM:
> From: Thomas Monjalon <thomas@monjalon.net>
> To: Jonas Pfefferle1 <JPF@zurich.ibm.com>
> Cc: dev@dpdk.org, Santosh Shukla
> <santosh.shukla@caviumnetworks.com>, jerin.jacob@caviumnetworks.com,
> hemant.agrawal@nxp.com, olivier.matz@6wind.com,
> maxime.coquelin@redhat.com, sergio.gonzalez.monroy@intel.com,
> bruce.richardson@intel.com, shreyansh.jain@nxp.com,
> gaetan.rivet@6wind.com, anatoly.burakov@intel.com,
> stephen@networkplumber.org, aconole@redhat.com
> Date: 11/02/2017 11:17 AM
> Subject: Re: [dpdk-dev] [PATCH v7 7/9] linuxapp/eal_vfio: honor iova
> mode before mapping
>
> Hi
>
> 26/10/2017 14:57, Jonas Pfefferle1:
> >
> > Hi @all
> >
> > I just stumbled upon this patch while testing on POWER. RTE_IOVA_VA
will
> > not work for the sPAPR code since the dma window size is currently
> > determined by the physical address only.
>
> Is it affecting POWER8?
It is.
>
> > I'm preparing a patch to address this.
>
> Any news?
> Can you use virtual addresses?
After a long discussion with Alexey (CC) we came to the conclusion that
with the current sPAPR iommu driver we cannot use virtual addresses since
the iova is restricted to lay in the DMA window which itself is restricted
to physical RAM addresses resp. with the current code 0 to hotplug memory
max. However, Alexey is working on a patch to lift this restriction on the
DMA window size which should allow us to do VA:VA mappings in the future.
For now we should fall back to PA in the dynamic iova mode check. I will
send an according patch later today.
>
>
"dev" <dev-bounces@dpdk.org> wrote on 11/02/2017 11:26:57 AM:
> From: "Jonas Pfefferle1" <JPF@zurich.ibm.com>
> To: Thomas Monjalon <thomas@monjalon.net>
> Cc: aconole@redhat.com, anatoly.burakov@intel.com,
> bruce.richardson@intel.com, dev@dpdk.org, gaetan.rivet@6wind.com,
> hemant.agrawal@nxp.com, jerin.jacob@caviumnetworks.com,
> maxime.coquelin@redhat.com, olivier.matz@6wind.com, Santosh Shukla
> <santosh.shukla@caviumnetworks.com>,
> sergio.gonzalez.monroy@intel.com, shreyansh.jain@nxp.com,
> stephen@networkplumber.org, "Alexey Kardashevskiy" <aik@ozlabs.ru>
> Date: 11/02/2017 11:27 AM
> Subject: Re: [dpdk-dev] [PATCH v7 7/9] linuxapp/eal_vfio: honor iova
> mode before mapping
> Sent by: "dev" <dev-bounces@dpdk.org>
>
>
> Thomas Monjalon <thomas@monjalon.net> wrote on 11/02/2017 11:17:10 AM:
>
> > From: Thomas Monjalon <thomas@monjalon.net>
> > To: Jonas Pfefferle1 <JPF@zurich.ibm.com>
> > Cc: dev@dpdk.org, Santosh Shukla
> > <santosh.shukla@caviumnetworks.com>, jerin.jacob@caviumnetworks.com,
> > hemant.agrawal@nxp.com, olivier.matz@6wind.com,
> > maxime.coquelin@redhat.com, sergio.gonzalez.monroy@intel.com,
> > bruce.richardson@intel.com, shreyansh.jain@nxp.com,
> > gaetan.rivet@6wind.com, anatoly.burakov@intel.com,
> > stephen@networkplumber.org, aconole@redhat.com
> > Date: 11/02/2017 11:17 AM
> > Subject: Re: [dpdk-dev] [PATCH v7 7/9] linuxapp/eal_vfio: honor iova
> > mode before mapping
> >
> > Hi
> >
> > 26/10/2017 14:57, Jonas Pfefferle1:
> > >
> > > Hi @all
> > >
> > > I just stumbled upon this patch while testing on POWER. RTE_IOVA_VA
> will
> > > not work for the sPAPR code since the dma window size is currently
> > > determined by the physical address only.
> >
> > Is it affecting POWER8?
>
> It is.
>
> >
> > > I'm preparing a patch to address this.
> >
> > Any news?
> > Can you use virtual addresses?
>
> After a long discussion with Alexey (CC) we came to the conclusion that
> with the current sPAPR iommu driver we cannot use virtual addresses since
> the iova is restricted to lay in the DMA window which itself is
restricted
> to physical RAM addresses resp. with the current code 0 to hotplug memory
> max. However, Alexey is working on a patch to lift this restriction on
the
> DMA window size which should allow us to do VA:VA mappings in the future.
> For now we should fall back to PA in the dynamic iova mode check. I will
> send an according patch later today.
I looked into this yesterday but I'm not sure what the right solution is
here.
At the time rte_pci_get_iommu_class is called we already know which IOMMU
types are supported because vfio_get_container_fd resp.
vfio_has_supported_extensions has been called however we do not know which
one is going to be used (Decided later in vfio_setup_device resp.
vfio_set_iommu_type). We can choose a iova mode which is supported by all
types but if the modes are exclusive to the types we have to guess which
one is going to be used. Or let the user decide?
Thanks,
Jonas
>
> >
> >
>
03/11/2017 10:56, Jonas Pfefferle1:
> Thomas Monjalon <thomas@monjalon.net> wrote on 11/02/2017 11:17:10 AM:
> > > 26/10/2017 14:57, Jonas Pfefferle1:
> > > >
> > > > Hi @all
> > > >
> > > > I just stumbled upon this patch while testing on POWER. RTE_IOVA_VA
> > will
> > > > not work for the sPAPR code since the dma window size is currently
> > > > determined by the physical address only.
> > >
> > > Is it affecting POWER8?
> >
> > It is.
> >
> > >
> > > > I'm preparing a patch to address this.
> > >
> > > Any news?
> > > Can you use virtual addresses?
> >
> > After a long discussion with Alexey (CC) we came to the conclusion that
> > with the current sPAPR iommu driver we cannot use virtual addresses since
> > the iova is restricted to lay in the DMA window which itself is
> restricted
> > to physical RAM addresses resp. with the current code 0 to hotplug memory
> > max. However, Alexey is working on a patch to lift this restriction on
> the
> > DMA window size which should allow us to do VA:VA mappings in the future.
> > For now we should fall back to PA in the dynamic iova mode check. I will
> > send an according patch later today.
>
> I looked into this yesterday but I'm not sure what the right solution is
> here.
> At the time rte_pci_get_iommu_class is called we already know which IOMMU
> types are supported because vfio_get_container_fd resp.
> vfio_has_supported_extensions has been called however we do not know which
> one is going to be used (Decided later in vfio_setup_device resp.
> vfio_set_iommu_type). We can choose a iova mode which is supported by all
> types but if the modes are exclusive to the types we have to guess which
> one is going to be used. Or let the user decide?
You can keep the old behaviour, restricting to physical memory,
until you support virtual addressing.
It can be just a #ifdef RTE_ARCH_PPC_64.
Thomas Monjalon <thomas@monjalon.net> wrote on 11/03/2017 11:28:10 AM:
> From: Thomas Monjalon <thomas@monjalon.net>
> To: Jonas Pfefferle1 <JPF@zurich.ibm.com>
> Cc: aconole@redhat.com, anatoly.burakov@intel.com,
> bruce.richardson@intel.com, dev@dpdk.org, gaetan.rivet@6wind.com,
> hemant.agrawal@nxp.com, jerin.jacob@caviumnetworks.com,
> maxime.coquelin@redhat.com, olivier.matz@6wind.com, Santosh Shukla
> <santosh.shukla@caviumnetworks.com>,
> sergio.gonzalez.monroy@intel.com, shreyansh.jain@nxp.com,
> stephen@networkplumber.org, Alexey Kardashevskiy <aik@ozlabs.ru>
> Date: 11/03/2017 11:28 AM
> Subject: Re: [dpdk-dev] [PATCH v7 7/9] linuxapp/eal_vfio: honor iova
> mode before mapping
>
> 03/11/2017 10:56, Jonas Pfefferle1:
> > Thomas Monjalon <thomas@monjalon.net> wrote on 11/02/2017 11:17:10 AM:
> > > > 26/10/2017 14:57, Jonas Pfefferle1:
> > > > >
> > > > > Hi @all
> > > > >
> > > > > I just stumbled upon this patch while testing on POWER.
RTE_IOVA_VA
> > > will
> > > > > not work for the sPAPR code since the dma window size is
currently
> > > > > determined by the physical address only.
> > > >
> > > > Is it affecting POWER8?
> > >
> > > It is.
> > >
> > > >
> > > > > I'm preparing a patch to address this.
> > > >
> > > > Any news?
> > > > Can you use virtual addresses?
> > >
> > > After a long discussion with Alexey (CC) we came to the conclusion
that
> > > with the current sPAPR iommu driver we cannot use virtual addresses
since
> > > the iova is restricted to lay in the DMA window which itself is
> > restricted
> > > to physical RAM addresses resp. with the current code 0 to hotplug
memory
> > > max. However, Alexey is working on a patch to lift this restriction
on
> > the
> > > DMA window size which should allow us to do VA:VA mappings in the
future.
> > > For now we should fall back to PA in the dynamic iova mode check. I
will
> > > send an according patch later today.
> >
> > I looked into this yesterday but I'm not sure what the right solution
is
> > here.
> > At the time rte_pci_get_iommu_class is called we already know which
IOMMU
> > types are supported because vfio_get_container_fd resp.
> > vfio_has_supported_extensions has been called however we do not know
which
> > one is going to be used (Decided later in vfio_setup_device resp.
> > vfio_set_iommu_type). We can choose a iova mode which is supported by
all
> > types but if the modes are exclusive to the types we have to guess
which
> > one is going to be used. Or let the user decide?
>
> You can keep the old behaviour, restricting to physical memory,
> until you support virtual addressing.
> It can be just a #ifdef RTE_ARCH_PPC_64.
>
Ok but we might want to refine this in the future. IMO It looks much
cleaner
to decide this on the iommu type plus this would also cover the noiommu
case without having this extra check reading the sysfs variable.
03/11/2017 11:44, Jonas Pfefferle1:
> Thomas Monjalon <thomas@monjalon.net> wrote on 11/03/2017 11:28:10 AM:
> > 03/11/2017 10:56, Jonas Pfefferle1:
> > > Thomas Monjalon <thomas@monjalon.net> wrote on 11/02/2017 11:17:10 AM:
> > > > > 26/10/2017 14:57, Jonas Pfefferle1:
> > > > > >
> > > > > > Hi @all
> > > > > >
> > > > > > I just stumbled upon this patch while testing on POWER.
> RTE_IOVA_VA
> > > > will
> > > > > > not work for the sPAPR code since the dma window size is
> currently
> > > > > > determined by the physical address only.
> > > > >
> > > > > Is it affecting POWER8?
> > > >
> > > > It is.
> > > >
> > > > >
> > > > > > I'm preparing a patch to address this.
> > > > >
> > > > > Any news?
> > > > > Can you use virtual addresses?
> > > >
> > > > After a long discussion with Alexey (CC) we came to the conclusion
> that
> > > > with the current sPAPR iommu driver we cannot use virtual addresses
> since
> > > > the iova is restricted to lay in the DMA window which itself is
> > > restricted
> > > > to physical RAM addresses resp. with the current code 0 to hotplug
> memory
> > > > max. However, Alexey is working on a patch to lift this restriction
> on
> > > the
> > > > DMA window size which should allow us to do VA:VA mappings in the
> future.
> > > > For now we should fall back to PA in the dynamic iova mode check. I
> will
> > > > send an according patch later today.
> > >
> > > I looked into this yesterday but I'm not sure what the right solution
> is
> > > here.
> > > At the time rte_pci_get_iommu_class is called we already know which
> IOMMU
> > > types are supported because vfio_get_container_fd resp.
> > > vfio_has_supported_extensions has been called however we do not know
> which
> > > one is going to be used (Decided later in vfio_setup_device resp.
> > > vfio_set_iommu_type). We can choose a iova mode which is supported by
> all
> > > types but if the modes are exclusive to the types we have to guess
> which
> > > one is going to be used. Or let the user decide?
> >
> > You can keep the old behaviour, restricting to physical memory,
> > until you support virtual addressing.
> > It can be just a #ifdef RTE_ARCH_PPC_64.
> >
>
> Ok but we might want to refine this in the future. IMO It looks much
> cleaner
> to decide this on the iommu type plus this would also cover the noiommu
> case without having this extra check reading the sysfs variable.
You are using the word "this" too many times to help me understand :)
Anyway, please send a quick fix today for 17.11.
The RC3 will be probably closed before Monday.
Thomas Monjalon <thomas@monjalon.net> wrote on 11/03/2017 11:54:45 AM:
> From: Thomas Monjalon <thomas@monjalon.net>
> To: Jonas Pfefferle1 <JPF@zurich.ibm.com>
> Cc: aconole@redhat.com, Alexey Kardashevskiy <aik@ozlabs.ru>,
> anatoly.burakov@intel.com, bruce.richardson@intel.com, dev@dpdk.org,
> gaetan.rivet@6wind.com, hemant.agrawal@nxp.com,
> jerin.jacob@caviumnetworks.com, maxime.coquelin@redhat.com,
> olivier.matz@6wind.com, Santosh Shukla
> <santosh.shukla@caviumnetworks.com>,
> sergio.gonzalez.monroy@intel.com, shreyansh.jain@nxp.com,
> stephen@networkplumber.org
> Date: 11/03/2017 11:55 AM
> Subject: Re: [dpdk-dev] [PATCH v7 7/9] linuxapp/eal_vfio: honor iova
> mode before mapping
>
> 03/11/2017 11:44, Jonas Pfefferle1:
> > Thomas Monjalon <thomas@monjalon.net> wrote on 11/03/2017 11:28:10 AM:
> > > 03/11/2017 10:56, Jonas Pfefferle1:
> > > > Thomas Monjalon <thomas@monjalon.net> wrote on 11/02/2017 11:17:10
AM:
> > > > > > 26/10/2017 14:57, Jonas Pfefferle1:
> > > > > > >
> > > > > > > Hi @all
> > > > > > >
> > > > > > > I just stumbled upon this patch while testing on POWER.
> > RTE_IOVA_VA
> > > > > will
> > > > > > > not work for the sPAPR code since the dma window size is
> > currently
> > > > > > > determined by the physical address only.
> > > > > >
> > > > > > Is it affecting POWER8?
> > > > >
> > > > > It is.
> > > > >
> > > > > >
> > > > > > > I'm preparing a patch to address this.
> > > > > >
> > > > > > Any news?
> > > > > > Can you use virtual addresses?
> > > > >
> > > > > After a long discussion with Alexey (CC) we came to the
conclusion
> > that
> > > > > with the current sPAPR iommu driver we cannot use virtual
addresses
> > since
> > > > > the iova is restricted to lay in the DMA window which itself is
> > > > restricted
> > > > > to physical RAM addresses resp. with the current code 0 to
hotplug
> > memory
> > > > > max. However, Alexey is working on a patch to lift this
restriction
> > on
> > > > the
> > > > > DMA window size which should allow us to do VA:VA mappings in the
> > future.
> > > > > For now we should fall back to PA in the dynamic iova mode check.
I
> > will
> > > > > send an according patch later today.
> > > >
> > > > I looked into this yesterday but I'm not sure what the right
solution
> > is
> > > > here.
> > > > At the time rte_pci_get_iommu_class is called we already know which
> > IOMMU
> > > > types are supported because vfio_get_container_fd resp.
> > > > vfio_has_supported_extensions has been called however we do not
know
> > which
> > > > one is going to be used (Decided later in vfio_setup_device resp.
> > > > vfio_set_iommu_type). We can choose a iova mode which is supported
by
> > all
> > > > types but if the modes are exclusive to the types we have to guess
> > which
> > > > one is going to be used. Or let the user decide?
> > >
> > > You can keep the old behaviour, restricting to physical memory,
> > > until you support virtual addressing.
> > > It can be just a #ifdef RTE_ARCH_PPC_64.
> > >
> >
> > Ok but we might want to refine this in the future. IMO It looks much
> > cleaner
> > to decide this on the iommu type plus this would also cover the noiommu
> > case without having this extra check reading the sysfs variable.
>
> You are using the word "this" too many times to help me understand :)
What I meant is a fix that decides which iova mode to use based on the
iommu types supported (determined by vfio_get_container_fd) instead of
another extra case for PPC much like the noiommu check. Both should
be covered by the supported types based check.
IMO much cleaner and simpler to support new iommu types.
>
> Anyway, please send a quick fix today for 17.11.
> The RC3 will be probably closed before Monday.
>
Will do.
@@ -706,7 +706,10 @@ vfio_type1_dma_map(int vfio_container_fd)
dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
dma_map.vaddr = ms[i].addr_64;
dma_map.size = ms[i].len;
- dma_map.iova = ms[i].phys_addr;
+ if (rte_eal_iova_mode() == RTE_IOVA_VA)
+ dma_map.iova = dma_map.vaddr;
+ else
+ dma_map.iova = ms[i].phys_addr;
dma_map.flags = VFIO_DMA_MAP_FLAG_READ | VFIO_DMA_MAP_FLAG_WRITE;
ret = ioctl(vfio_container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);
@@ -792,7 +795,10 @@ vfio_spapr_dma_map(int vfio_container_fd)
dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
dma_map.vaddr = ms[i].addr_64;
dma_map.size = ms[i].len;
- dma_map.iova = ms[i].phys_addr;
+ if (rte_eal_iova_mode() == RTE_IOVA_VA)
+ dma_map.iova = dma_map.vaddr;
+ else
+ dma_map.iova = ms[i].phys_addr;
dma_map.flags = VFIO_DMA_MAP_FLAG_READ |
VFIO_DMA_MAP_FLAG_WRITE;