eal: force IOVA mode to physical

  This patch adds a configuration option to force the IOVA mode to
physical address (PA). There exists virtual devices that are not
directly attached to the PCI bus, and therefore the auto detection
of the IOVA mode based on probing the PCI bus and IOMMU configuration
may not report the required addressing mode. Having the configuration
option permits the mode to be explicitly configured in this scenario.

Signed-off-by: eric zhang <eric.zhang@windriver.com>
---
 lib/librte_eal/linuxapp/eal/eal.c | 7 +++++++
 1 file changed, 7 insertions(+)
  

On 8/29/2018 9:28 PM, eric zhang wrote:
> This patch adds a configuration option to force the IOVA mode to
> physical address (PA). There exists virtual devices that are not
> directly attached to the PCI bus, and therefore the auto detection
> of the IOVA mode based on probing the PCI bus and IOMMU configuration
> may not report the required addressing mode. Having the configuration
> option permits the mode to be explicitly configured in this scenario.
>
> Signed-off-by: eric zhang <eric.zhang@windriver.com>
> ---
>   lib/librte_eal/linuxapp/eal/eal.c | 7 +++++++
you will also need the same in bsdapp/eal

>   1 file changed, 7 insertions(+)
>
> diff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/eal/eal.c
> index e0b5ae1..bee4aed 100644
> --- a/lib/librte_eal/linuxapp/eal/eal.c
> +++ b/lib/librte_eal/linuxapp/eal/eal.c
> @@ -805,6 +805,7 @@ static void rte_eal_init_alert(const char *msg)
>   		return -1;
>   	}
>   
> +#ifndef RTE_EAL_IOVA_MODE_PA
Add the default option as 'n'  in config/common_base
>   	/* autodetect the iova mapping mode (default is iova_pa) */
>   	rte_eal_get_configuration()->iova_mode = rte_bus_get_iommu_class();
>   
> @@ -816,6 +817,12 @@ static void rte_eal_init_alert(const char *msg)
>   			"Some devices want IOVA as VA but PA will be used because.. "
>   			"KNI module inserted\n");
>   	}
> +#else
> +	/* Force iova mapping mode to be physical address */
> +	rte_eal_get_configuration()->iova_mode = RTE_IOVA_PA;
> +	RTE_LOG(WARNING, EAL,
> +		"Force the iova mapping mode to be physical address\n");
> +#endif
>   
>   	if (internal_config.no_hugetlbfs == 0 &&
>   			internal_config.process_type != RTE_PROC_SECONDARY &&
  

On 29-Aug-18 4:58 PM, eric zhang wrote:
> This patch adds a configuration option to force the IOVA mode to
> physical address (PA). There exists virtual devices that are not
> directly attached to the PCI bus, and therefore the auto detection
> of the IOVA mode based on probing the PCI bus and IOMMU configuration
> may not report the required addressing mode. Having the configuration
> option permits the mode to be explicitly configured in this scenario.
> 
> Signed-off-by: eric zhang <eric.zhang@windriver.com>
> ---

Defining this at compile-time seems like an overkill. Wouldn't it be 
better to just add an EAL command-line option to force IOVA mode to a 
particular value?
  

Hi,

On Thu, Aug 30, 2018 at 10:09:04AM +0100, Burakov, Anatoly wrote:
> On 29-Aug-18 4:58 PM, eric zhang wrote:
> > This patch adds a configuration option to force the IOVA mode to
> > physical address (PA). There exists virtual devices that are not
> > directly attached to the PCI bus, and therefore the auto detection
> > of the IOVA mode based on probing the PCI bus and IOMMU configuration
> > may not report the required addressing mode. Having the configuration
> > option permits the mode to be explicitly configured in this scenario.
> > 
> > Signed-off-by: eric zhang <eric.zhang@windriver.com>
> > ---
> 
> Defining this at compile-time seems like an overkill. Wouldn't it be better
> to just add an EAL command-line option to force IOVA mode to a particular
> value?
> 
> -- 
> Thanks,
> Anatoly

What is the bus of these devices and why not implement get_iommu_class
in it?
  

Hi,

On 8/30/2018 3:13 PM, Gaëtan Rivet wrote:
> Hi,
>
> On Thu, Aug 30, 2018 at 10:09:04AM +0100, Burakov, Anatoly wrote:
>> On 29-Aug-18 4:58 PM, eric zhang wrote:
>>> This patch adds a configuration option to force the IOVA mode to
>>> physical address (PA). There exists virtual devices that are not
>>> directly attached to the PCI bus, and therefore the auto detection
>>> of the IOVA mode based on probing the PCI bus and IOMMU configuration
>>> may not report the required addressing mode. Having the configuration
>>> option permits the mode to be explicitly configured in this scenario.
>>>
>>> Signed-off-by: eric zhang <eric.zhang@windriver.com>
>>> ---
>> Defining this at compile-time seems like an overkill. Wouldn't it be better
>> to just add an EAL command-line option to force IOVA mode to a particular
>> value?
That is a good suggestion.
>> -- 
>> Thanks,
>> Anatoly
> What is the bus of these devices and why not implement get_iommu_class
> in it?
There are cases, where you are using dpdk libraries with external 
libraries and you need to change the default behavior DPDK lib to use 
physical address instead of virtual address.
Providing an option to user will help.
  

On Thursday 30 August 2018 05:43 PM, Hemant wrote:
> External Email
>
> Hi,
>
> On 8/30/2018 3:13 PM, Gaëtan Rivet wrote:
>> Hi,
>>
>> On Thu, Aug 30, 2018 at 10:09:04AM +0100, Burakov, Anatoly wrote:
>>> On 29-Aug-18 4:58 PM, eric zhang wrote:
>>>> This patch adds a configuration option to force the IOVA mode to
>>>> physical address (PA). There exists virtual devices that are not
>>>> directly attached to the PCI bus, and therefore the auto detection
>>>> of the IOVA mode based on probing the PCI bus and IOMMU configuration
>>>> may not report the required addressing mode. Having the configuration
>>>> option permits the mode to be explicitly configured in this scenario.
>>>>
>>>> Signed-off-by: eric zhang <eric.zhang@windriver.com>
>>>> ---
>>> Defining this at compile-time seems like an overkill. Wouldn't it be better
>>> to just add an EAL command-line option to force IOVA mode to a particular
>>> value?
> That is a good suggestion.
>>> -- 
>>> Thanks,
>>> Anatoly
>> What is the bus of these devices and why not implement get_iommu_class
>> in it?
> There are cases, where you are using dpdk libraries with external
> libraries and you need to change the default behavior DPDK lib to use
> physical address instead of virtual address.
> Providing an option to user will help.
>
>
More appropriate solution could be:
* Either fix it at bus layer .. i.e.. get_iommu_class()..
* Or introduce something like [1] --iova-mode=<pa/va> param.

Former is better solution than latter if autodetection is a key criteria.
Thanks.

[1] http://patchwork.dpdk.org/patch/25192/
  

> -----Original Message-----
> From: santosh [mailto:santosh.shukla@caviumnetworks.com]
> Sent: Thursday, August 30, 2018 8:59 AM
> To: hemant.agrawal@nxp.com; Gaëtan Rivet; Burakov, Anatoly
> Cc: Zhang, Qing Long (Eric); bruce.richardson@intel.com; dev@dpdk.org;
> Legacy, Allain; Peters, Matt
> Subject: Re: [dpdk-dev] [PATCH] eal: force IOVA mode to physical
> 
> 
> On Thursday 30 August 2018 05:43 PM, Hemant wrote:
> > External Email
<...>
> 
> [1] http://patchwork.dpdk.org/patch/25192/

Is there a reason this was abandoned?
  

On Thursday 30 August 2018 07:26 PM, Legacy, Allain wrote:
> External Email
>
>> -----Original Message-----
>> From: santosh [mailto:santosh.shukla@caviumnetworks.com]
>> Sent: Thursday, August 30, 2018 8:59 AM
>> To: hemant.agrawal@nxp.com; Gaëtan Rivet; Burakov, Anatoly
>> Cc: Zhang, Qing Long (Eric); bruce.richardson@intel.com; dev@dpdk.org;
>> Legacy, Allain; Peters, Matt
>> Subject: Re: [dpdk-dev] [PATCH] eal: force IOVA mode to physical
>>
>>
>> On Thursday 30 August 2018 05:43 PM, Hemant wrote:
>>> External Email
> <...>
>> [1] http://patchwork.dpdk.org/patch/25192/
> Is there a reason this was abandoned?

Wasn't abandoned exactly, in-fact we had better solution which is to introduce autodetection notion at bus layer.
  

On 08/30/2018 08:59 AM, santosh wrote:
> On Thursday 30 August 2018 05:43 PM, Hemant wrote:
>> External Email
>>
>> Hi,
>>
>> On 8/30/2018 3:13 PM, Gaëtan Rivet wrote:
>>> Hi,
>>>
>>> On Thu, Aug 30, 2018 at 10:09:04AM +0100, Burakov, Anatoly wrote:
>>>> On 29-Aug-18 4:58 PM, eric zhang wrote:
>>>>> This patch adds a configuration option to force the IOVA mode to
>>>>> physical address (PA). There exists virtual devices that are not
>>>>> directly attached to the PCI bus, and therefore the auto detection
>>>>> of the IOVA mode based on probing the PCI bus and IOMMU configuration
>>>>> may not report the required addressing mode. Having the configuration
>>>>> option permits the mode to be explicitly configured in this scenario.
>>>>>
>>>>> Signed-off-by: eric zhang <eric.zhang@windriver.com>
>>>>> ---
>>>> Defining this at compile-time seems like an overkill. Wouldn't it be better
>>>> to just add an EAL command-line option to force IOVA mode to a particular
>>>> value?
>> That is a good suggestion.
>>>> -- 
>>>> Thanks,
>>>> Anatoly
>>> What is the bus of these devices and why not implement get_iommu_class
>>> in it?
>> There are cases, where you are using dpdk libraries with external
>> libraries and you need to change the default behavior DPDK lib to use
>> physical address instead of virtual address.
>> Providing an option to user will help.
>>
>>
> More appropriate solution could be:
> * Either fix it at bus layer .. i.e.. get_iommu_class()..
> * Or introduce something like [1] --iova-mode=<pa/va> param.
>
> Former is better solution than latter if autodetection is a key criteria.
> Thanks.
>
> [1] http://patchwork.dpdk.org/patch/25192/
>
It's not generic which couldn't be fixed at bus layer.
So what's the preference of EAL option or compile time solution?
Adding --iova-mode as patch [1] will overrivde auto-detection 
rte_bus_get_iommu_class()
make it no use; compile time solution will align with upstream and keep 
new atuodetection
solution in #ifndef.

Thanks
Eric
  

-----Original Message-----
> Date: Tue, 4 Sep 2018 23:40:36 -0400
> From: Eric Zhang <eric.zhang@windriver.com>
> To: santosh <santosh.shukla@caviumnetworks.com>, hemant.agrawal@nxp.com,
>  Gaëtan Rivet <gaetan.rivet@6wind.com>, "Burakov, Anatoly"
>  <anatoly.burakov@intel.com>
> CC: bruce.richardson@intel.com, dev@dpdk.org, Allain.Legacy@windriver.com,
>  Matt.Peters@windriver.com
> Subject: Re: [dpdk-dev] [PATCH] eal: force IOVA mode to physical
> User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101
>  Thunderbird/52.9.1
> 
> On 08/30/2018 08:59 AM, santosh wrote:
> > On Thursday 30 August 2018 05:43 PM, Hemant wrote:
> > > External Email
> > > 
> > > Hi,
> > > 
> > > On 8/30/2018 3:13 PM, Gaëtan Rivet wrote:
> > > > Hi,
> > > > 
> > > > On Thu, Aug 30, 2018 at 10:09:04AM +0100, Burakov, Anatoly wrote:
> > > > > On 29-Aug-18 4:58 PM, eric zhang wrote:
> > > > > > This patch adds a configuration option to force the IOVA mode to
> > > > > > physical address (PA). There exists virtual devices that are not
> > > > > > directly attached to the PCI bus, and therefore the auto detection
> > > > > > of the IOVA mode based on probing the PCI bus and IOMMU configuration
> > > > > > may not report the required addressing mode. Having the configuration
> > > > > > option permits the mode to be explicitly configured in this scenario.
> > > > > > 
> > > > > > Signed-off-by: eric zhang <eric.zhang@windriver.com>
> > > > > > ---
> > > > > Defining this at compile-time seems like an overkill. Wouldn't it be better
> > > > > to just add an EAL command-line option to force IOVA mode to a particular
> > > > > value?
> > > That is a good suggestion.
> > > > > --
> > > > > Thanks,
> > > > > Anatoly
> > > > What is the bus of these devices and why not implement get_iommu_class
> > > > in it?
> > > There are cases, where you are using dpdk libraries with external
> > > libraries and you need to change the default behavior DPDK lib to use
> > > physical address instead of virtual address.
> > > Providing an option to user will help.
> > > 
> > > 
> > More appropriate solution could be:
> > * Either fix it at bus layer .. i.e.. get_iommu_class()..
> > * Or introduce something like [1] --iova-mode=<pa/va> param.
> > 
> > Former is better solution than latter if autodetection is a key criteria.
> > Thanks.
> > 
> > [1] http://patchwork.dpdk.org/patch/25192/
> > 
> It's not generic which couldn't be fixed at bus layer.
> So what's the preference of EAL option or compile time solution?
> Adding --iova-mode as patch [1] will overrivde auto-detection
> rte_bus_get_iommu_class()
> make it no use; compile time solution will align with upstream and keep
> new atuodetection
> solution in #ifndef.

If it is for vdev devices, why not introduce something like
RTE_PCI_DRV_IOVA_AS_VA and let vdev device describe its personality.
And based on the devices(flags) on vdev bus, rte_bus_get_iommu_class()
of vdev can decide the mode just like PCI bus.


> 
> Thanks
> Eric
>
  

On 06-Sep-18 8:34 AM, Jerin Jacob wrote:
> -----Original Message-----
>> Date: Tue, 4 Sep 2018 23:40:36 -0400
>> From: Eric Zhang <eric.zhang@windriver.com>
>> To: santosh <santosh.shukla@caviumnetworks.com>, hemant.agrawal@nxp.com,
>>   Gaëtan Rivet <gaetan.rivet@6wind.com>, "Burakov, Anatoly"
>>   <anatoly.burakov@intel.com>
>> CC: bruce.richardson@intel.com, dev@dpdk.org, Allain.Legacy@windriver.com,
>>   Matt.Peters@windriver.com
>> Subject: Re: [dpdk-dev] [PATCH] eal: force IOVA mode to physical
>> User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101
>>   Thunderbird/52.9.1
>>
>> On 08/30/2018 08:59 AM, santosh wrote:
>>> On Thursday 30 August 2018 05:43 PM, Hemant wrote:
>>>> External Email
>>>>
>>>> Hi,
>>>>
>>>> On 8/30/2018 3:13 PM, Gaëtan Rivet wrote:
>>>>> Hi,
>>>>>
>>>>> On Thu, Aug 30, 2018 at 10:09:04AM +0100, Burakov, Anatoly wrote:
>>>>>> On 29-Aug-18 4:58 PM, eric zhang wrote:
>>>>>>> This patch adds a configuration option to force the IOVA mode to
>>>>>>> physical address (PA). There exists virtual devices that are not
>>>>>>> directly attached to the PCI bus, and therefore the auto detection
>>>>>>> of the IOVA mode based on probing the PCI bus and IOMMU configuration
>>>>>>> may not report the required addressing mode. Having the configuration
>>>>>>> option permits the mode to be explicitly configured in this scenario.
>>>>>>>
>>>>>>> Signed-off-by: eric zhang <eric.zhang@windriver.com>
>>>>>>> ---
>>>>>> Defining this at compile-time seems like an overkill. Wouldn't it be better
>>>>>> to just add an EAL command-line option to force IOVA mode to a particular
>>>>>> value?
>>>> That is a good suggestion.
>>>>>> --
>>>>>> Thanks,
>>>>>> Anatoly
>>>>> What is the bus of these devices and why not implement get_iommu_class
>>>>> in it?
>>>> There are cases, where you are using dpdk libraries with external
>>>> libraries and you need to change the default behavior DPDK lib to use
>>>> physical address instead of virtual address.
>>>> Providing an option to user will help.
>>>>
>>>>
>>> More appropriate solution could be:
>>> * Either fix it at bus layer .. i.e.. get_iommu_class()..
>>> * Or introduce something like [1] --iova-mode=<pa/va> param.
>>>
>>> Former is better solution than latter if autodetection is a key criteria.
>>> Thanks.
>>>
>>> [1] http://patchwork.dpdk.org/patch/25192/
>>>
>> It's not generic which couldn't be fixed at bus layer.
>> So what's the preference of EAL option or compile time solution?
>> Adding --iova-mode as patch [1] will overrivde auto-detection
>> rte_bus_get_iommu_class()
>> make it no use; compile time solution will align with upstream and keep
>> new atuodetection
>> solution in #ifndef.
> 
> If it is for vdev devices, why not introduce something like
> RTE_PCI_DRV_IOVA_AS_VA and let vdev device describe its personality.
> And based on the devices(flags) on vdev bus, rte_bus_get_iommu_class()
> of vdev can decide the mode just like PCI bus.
> 

That seems like a better option to me, +1. As far as i know, at the 
moment if there are no devices attached at all, or if there are only 
vdev devices attached, DPDK will default to IOVA as PA mode for no good 
reason; such a change would certainly fix this.

> 
>>
>> Thanks
>> Eric
>>
>
  

On 09/07/2018 05:26 AM, Burakov, Anatoly wrote:
> On 06-Sep-18 8:34 AM, Jerin Jacob wrote:
>> -----Original Message-----
>>> Date: Tue, 4 Sep 2018 23:40:36 -0400
>>> From: Eric Zhang <eric.zhang@windriver.com>
>>> To: santosh <santosh.shukla@caviumnetworks.com>, 
>>> hemant.agrawal@nxp.com,
>>>   Gaëtan Rivet <gaetan.rivet@6wind.com>, "Burakov, Anatoly"
>>>   <anatoly.burakov@intel.com>
>>> CC: bruce.richardson@intel.com, dev@dpdk.org, 
>>> Allain.Legacy@windriver.com,
>>>   Matt.Peters@windriver.com
>>> Subject: Re: [dpdk-dev] [PATCH] eal: force IOVA mode to physical
>>> User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101
>>>   Thunderbird/52.9.1
>>>
>>> On 08/30/2018 08:59 AM, santosh wrote:
>>>> On Thursday 30 August 2018 05:43 PM, Hemant wrote:
>>>>> External Email
>>>>>
>>>>> Hi,
>>>>>
>>>>> On 8/30/2018 3:13 PM, Gaëtan Rivet wrote:
>>>>>> Hi,
>>>>>>
>>>>>> On Thu, Aug 30, 2018 at 10:09:04AM +0100, Burakov, Anatoly wrote:
>>>>>>> On 29-Aug-18 4:58 PM, eric zhang wrote:
>>>>>>>> This patch adds a configuration option to force the IOVA mode to
>>>>>>>> physical address (PA). There exists virtual devices that are not
>>>>>>>> directly attached to the PCI bus, and therefore the auto detection
>>>>>>>> of the IOVA mode based on probing the PCI bus and IOMMU 
>>>>>>>> configuration
>>>>>>>> may not report the required addressing mode. Having the 
>>>>>>>> configuration
>>>>>>>> option permits the mode to be explicitly configured in this 
>>>>>>>> scenario.
>>>>>>>>
>>>>>>>> Signed-off-by: eric zhang <eric.zhang@windriver.com>
>>>>>>>> ---
>>>>>>> Defining this at compile-time seems like an overkill. Wouldn't 
>>>>>>> it be better
>>>>>>> to just add an EAL command-line option to force IOVA mode to a 
>>>>>>> particular
>>>>>>> value?
>>>>> That is a good suggestion.
>>>>>>> -- 
>>>>>>> Thanks,
>>>>>>> Anatoly
>>>>>> What is the bus of these devices and why not implement 
>>>>>> get_iommu_class
>>>>>> in it?
>>>>> There are cases, where you are using dpdk libraries with external
>>>>> libraries and you need to change the default behavior DPDK lib to use
>>>>> physical address instead of virtual address.
>>>>> Providing an option to user will help.
>>>>>
>>>>>
>>>> More appropriate solution could be:
>>>> * Either fix it at bus layer .. i.e.. get_iommu_class()..
>>>> * Or introduce something like [1] --iova-mode=<pa/va> param.
>>>>
>>>> Former is better solution than latter if autodetection is a key 
>>>> criteria.
>>>> Thanks.
>>>>
>>>> [1] http://patchwork.dpdk.org/patch/25192/
>>>>
>>> It's not generic which couldn't be fixed at bus layer.
>>> So what's the preference of EAL option or compile time solution?
>>> Adding --iova-mode as patch [1] will overrivde auto-detection
>>> rte_bus_get_iommu_class()
>>> make it no use; compile time solution will align with upstream and keep
>>> new atuodetection
>>> solution in #ifndef.
>>
>> If it is for vdev devices, why not introduce something like
>> RTE_PCI_DRV_IOVA_AS_VA and let vdev device describe its personality.
>> And based on the devices(flags) on vdev bus, rte_bus_get_iommu_class()
>> of vdev can decide the mode just like PCI bus.
>>
>
> That seems like a better option to me, +1. As far as i know, at the 
> moment if there are no devices attached at all, or if there are only 
> vdev devices attached, DPDK will default to IOVA as PA mode for no 
> good reason; such a change would certainly fix this.
Thanks for the suggestions however our virtual device doesn't run dpdk 
vdev code so we can't use the flag.
Notice that in eal.c there is one workaround that force iova to be PA 
per virtual device is not directly attached to pci. That case is 
checking kni module. Ours is a similar case that virtual device not 
attach pci directly.
So we have to turn to force iova to PA either 1. compilation option 2. 
eal option.  Which one should be the preference by taking into 
consideration that align with upstream?

Thanks

>
>>
>>>
>>> Thanks
>>> Eric
>>>
>>
>
>
  

On 09/07/2018 04:13 PM, Eric Zhang wrote:
>
>
> On 09/07/2018 05:26 AM, Burakov, Anatoly wrote:
>> On 06-Sep-18 8:34 AM, Jerin Jacob wrote:
>>> -----Original Message-----
>>>> Date: Tue, 4 Sep 2018 23:40:36 -0400
>>>> From: Eric Zhang <eric.zhang@windriver.com>
>>>> To: santosh <santosh.shukla@caviumnetworks.com>, 
>>>> hemant.agrawal@nxp.com,
>>>>   Gaëtan Rivet <gaetan.rivet@6wind.com>, "Burakov, Anatoly"
>>>>   <anatoly.burakov@intel.com>
>>>> CC: bruce.richardson@intel.com, dev@dpdk.org, 
>>>> Allain.Legacy@windriver.com,
>>>>   Matt.Peters@windriver.com
>>>> Subject: Re: [dpdk-dev] [PATCH] eal: force IOVA mode to physical
>>>> User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101
>>>>   Thunderbird/52.9.1
>>>>
>>>> On 08/30/2018 08:59 AM, santosh wrote:
>>>>> On Thursday 30 August 2018 05:43 PM, Hemant wrote:
>>>>>> External Email
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> On 8/30/2018 3:13 PM, Gaëtan Rivet wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> On Thu, Aug 30, 2018 at 10:09:04AM +0100, Burakov, Anatoly wrote:
>>>>>>>> On 29-Aug-18 4:58 PM, eric zhang wrote:
>>>>>>>>> This patch adds a configuration option to force the IOVA mode to
>>>>>>>>> physical address (PA). There exists virtual devices that are not
>>>>>>>>> directly attached to the PCI bus, and therefore the auto 
>>>>>>>>> detection
>>>>>>>>> of the IOVA mode based on probing the PCI bus and IOMMU 
>>>>>>>>> configuration
>>>>>>>>> may not report the required addressing mode. Having the 
>>>>>>>>> configuration
>>>>>>>>> option permits the mode to be explicitly configured in this 
>>>>>>>>> scenario.
>>>>>>>>>
>>>>>>>>> Signed-off-by: eric zhang <eric.zhang@windriver.com>
>>>>>>>>> ---
>>>>>>>> Defining this at compile-time seems like an overkill. Wouldn't 
>>>>>>>> it be better
>>>>>>>> to just add an EAL command-line option to force IOVA mode to a 
>>>>>>>> particular
>>>>>>>> value?
>>>>>> That is a good suggestion.
>>>>>>>> -- 
>>>>>>>> Thanks,
>>>>>>>> Anatoly
>>>>>>> What is the bus of these devices and why not implement 
>>>>>>> get_iommu_class
>>>>>>> in it?
>>>>>> There are cases, where you are using dpdk libraries with external
>>>>>> libraries and you need to change the default behavior DPDK lib to 
>>>>>> use
>>>>>> physical address instead of virtual address.
>>>>>> Providing an option to user will help.
>>>>>>
>>>>>>
>>>>> More appropriate solution could be:
>>>>> * Either fix it at bus layer .. i.e.. get_iommu_class()..
>>>>> * Or introduce something like [1] --iova-mode=<pa/va> param.
>>>>>
>>>>> Former is better solution than latter if autodetection is a key 
>>>>> criteria.
>>>>> Thanks.
>>>>>
>>>>> [1] http://patchwork.dpdk.org/patch/25192/
>>>>>
>>>> It's not generic which couldn't be fixed at bus layer.
>>>> So what's the preference of EAL option or compile time solution?
>>>> Adding --iova-mode as patch [1] will overrivde auto-detection
>>>> rte_bus_get_iommu_class()
>>>> make it no use; compile time solution will align with upstream and 
>>>> keep
>>>> new atuodetection
>>>> solution in #ifndef.
>>>
>>> If it is for vdev devices, why not introduce something like
>>> RTE_PCI_DRV_IOVA_AS_VA and let vdev device describe its personality.
>>> And based on the devices(flags) on vdev bus, rte_bus_get_iommu_class()
>>> of vdev can decide the mode just like PCI bus.
>>>
>>
>> That seems like a better option to me, +1. As far as i know, at the 
>> moment if there are no devices attached at all, or if there are only 
>> vdev devices attached, DPDK will default to IOVA as PA mode for no 
>> good reason; such a change would certainly fix this.
> Thanks for the suggestions however our virtual device doesn't run dpdk 
> vdev code so we can't use the flag.
> Notice that in eal.c there is one workaround that force iova to be PA 
> per virtual device is not directly attached to pci. That case is 
> checking kni module. Ours is a similar case that virtual device not 
> attach pci directly.
> So we have to turn to force iova to PA either 1. compilation option 2. 
> eal option.  Which one should be the preference by taking into 
> consideration that align with upstream?
>
> Thanks
>
Any comments?
>>
>>>
>>>>
>>>> Thanks
>>>> Eric
>>>>
>>>
>>
>>
>
  

Hi,

A little bit of self-advertising:
I recently pushed patches that will make DPDK default to RTE_IOVA_VA when physical addresses were not explicitly requested and are not available, e.g. when running as a non-privileged user. It shouldn't cause any conflicts with the changes you're proposing here, but any review is welcome.

pci/linux: use RTE_IOVA_VA whenever possible http://patches.dpdk.org/patch/44392/ 
[v2] eal/bus: use RTE_IOVA_PA only if phys addresses are available http://patches.dpdk.org/patch/44420/

As for the --iova-mode=<pa/va>, I agree it could help DPDK use cases where most drivers or devices are hotplugged/hotattached at runtime - e.g. SPDK, it could certainly make use of such param.

D.

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Eric Zhang
> Sent: Tuesday, September 11, 2018 7:22 PM
> To: Burakov, Anatoly <anatoly.burakov@intel.com>; Jerin Jacob
> <jerin.jacob@caviumnetworks.com>
> Cc: santosh <santosh.shukla@caviumnetworks.com>;
> hemant.agrawal@nxp.com; Gaëtan Rivet <gaetan.rivet@6wind.com>;
> Richardson, Bruce <bruce.richardson@intel.com>; dev@dpdk.org;
> Allain.Legacy@windriver.com; Matt.Peters@windriver.com
> Subject: Re: [dpdk-dev] [PATCH] eal: force IOVA mode to physical
> 
> 
> 
> On 09/07/2018 04:13 PM, Eric Zhang wrote:
> >
> >
> > On 09/07/2018 05:26 AM, Burakov, Anatoly wrote:
> >> On 06-Sep-18 8:34 AM, Jerin Jacob wrote:
> >>> -----Original Message-----
> >>>> Date: Tue, 4 Sep 2018 23:40:36 -0400
> >>>> From: Eric Zhang <eric.zhang@windriver.com>
> >>>> To: santosh <santosh.shukla@caviumnetworks.com>,
> >>>> hemant.agrawal@nxp.com,
> >>>>   Gaëtan Rivet <gaetan.rivet@6wind.com>, "Burakov, Anatoly"
> >>>>   <anatoly.burakov@intel.com>
> >>>> CC: bruce.richardson@intel.com, dev@dpdk.org,
> >>>> Allain.Legacy@windriver.com,
> >>>>   Matt.Peters@windriver.com
> >>>> Subject: Re: [dpdk-dev] [PATCH] eal: force IOVA mode to physical
> >>>> User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101
> >>>>   Thunderbird/52.9.1
> >>>>
> >>>> On 08/30/2018 08:59 AM, santosh wrote:
> >>>>> On Thursday 30 August 2018 05:43 PM, Hemant wrote:
> >>>>>> External Email
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> On 8/30/2018 3:13 PM, Gaëtan Rivet wrote:
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> On Thu, Aug 30, 2018 at 10:09:04AM +0100, Burakov, Anatoly wrote:
> >>>>>>>> On 29-Aug-18 4:58 PM, eric zhang wrote:
> >>>>>>>>> This patch adds a configuration option to force the IOVA mode
> >>>>>>>>> to physical address (PA). There exists virtual devices that
> >>>>>>>>> are not directly attached to the PCI bus, and therefore the
> >>>>>>>>> auto detection of the IOVA mode based on probing the PCI bus
> >>>>>>>>> and IOMMU configuration may not report the required
> addressing
> >>>>>>>>> mode. Having the configuration option permits the mode to be
> >>>>>>>>> explicitly configured in this scenario.
> >>>>>>>>>
> >>>>>>>>> Signed-off-by: eric zhang <eric.zhang@windriver.com>
> >>>>>>>>> ---
> >>>>>>>> Defining this at compile-time seems like an overkill. Wouldn't
> >>>>>>>> it be better to just add an EAL command-line option to force
> >>>>>>>> IOVA mode to a particular value?
> >>>>>> That is a good suggestion.
> >>>>>>>> --
> >>>>>>>> Thanks,
> >>>>>>>> Anatoly
> >>>>>>> What is the bus of these devices and why not implement
> >>>>>>> get_iommu_class in it?
> >>>>>> There are cases, where you are using dpdk libraries with external
> >>>>>> libraries and you need to change the default behavior DPDK lib to
> >>>>>> use physical address instead of virtual address.
> >>>>>> Providing an option to user will help.
> >>>>>>
> >>>>>>
> >>>>> More appropriate solution could be:
> >>>>> * Either fix it at bus layer .. i.e.. get_iommu_class()..
> >>>>> * Or introduce something like [1] --iova-mode=<pa/va> param.
> >>>>>
> >>>>> Former is better solution than latter if autodetection is a key
> >>>>> criteria.
> >>>>> Thanks.
> >>>>>
> >>>>> [1] http://patchwork.dpdk.org/patch/25192/
> >>>>>
> >>>> It's not generic which couldn't be fixed at bus layer.
> >>>> So what's the preference of EAL option or compile time solution?
> >>>> Adding --iova-mode as patch [1] will overrivde auto-detection
> >>>> rte_bus_get_iommu_class()
> >>>> make it no use; compile time solution will align with upstream and
> >>>> keep new atuodetection solution in #ifndef.
> >>>
> >>> If it is for vdev devices, why not introduce something like
> >>> RTE_PCI_DRV_IOVA_AS_VA and let vdev device describe its personality.
> >>> And based on the devices(flags) on vdev bus,
> >>> rte_bus_get_iommu_class() of vdev can decide the mode just like PCI
> bus.
> >>>
> >>
> >> That seems like a better option to me, +1. As far as i know, at the
> >> moment if there are no devices attached at all, or if there are only
> >> vdev devices attached, DPDK will default to IOVA as PA mode for no
> >> good reason; such a change would certainly fix this.
> > Thanks for the suggestions however our virtual device doesn't run dpdk
> > vdev code so we can't use the flag.
> > Notice that in eal.c there is one workaround that force iova to be PA
> > per virtual device is not directly attached to pci. That case is
> > checking kni module. Ours is a similar case that virtual device not
> > attach pci directly.
> > So we have to turn to force iova to PA either 1. compilation option 2.
> > eal option.  Which one should be the preference by taking into
> > consideration that align with upstream?
> >
> > Thanks
> >
> Any comments?
> >>
> >>>
> >>>>
> >>>> Thanks
> >>>> Eric
> >>>>
> >>>
> >>
> >>
> >
  

Hi All,
Considered that most people commented that eal option "--iova-mode" is a 
better solution that gives user a chance not to use bus iova auto 
detection scheme and want to override for some reason, I will generate 
2nd version patch which will use eal option instead of compilation 
configuration.

Eric

On 09/17/2018 04:32 AM, Stojaczyk, Dariusz wrote:
> Hi,
>
> A little bit of self-advertising:
> I recently pushed patches that will make DPDK default to RTE_IOVA_VA when physical addresses were not explicitly requested and are not available, e.g. when running as a non-privileged user. It shouldn't cause any conflicts with the changes you're proposing here, but any review is welcome.
>
> pci/linux: use RTE_IOVA_VA whenever possible http://patches.dpdk.org/patch/44392/
> [v2] eal/bus: use RTE_IOVA_PA only if phys addresses are available http://patches.dpdk.org/patch/44420/
>
> As for the --iova-mode=<pa/va>, I agree it could help DPDK use cases where most drivers or devices are hotplugged/hotattached at runtime - e.g. SPDK, it could certainly make use of such param.
>
> D.
>
>> -----Original Message-----
>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Eric Zhang
>> Sent: Tuesday, September 11, 2018 7:22 PM
>> To: Burakov, Anatoly <anatoly.burakov@intel.com>; Jerin Jacob
>> <jerin.jacob@caviumnetworks.com>
>> Cc: santosh <santosh.shukla@caviumnetworks.com>;
>> hemant.agrawal@nxp.com; Gaëtan Rivet <gaetan.rivet@6wind.com>;
>> Richardson, Bruce <bruce.richardson@intel.com>; dev@dpdk.org;
>> Allain.Legacy@windriver.com; Matt.Peters@windriver.com
>> Subject: Re: [dpdk-dev] [PATCH] eal: force IOVA mode to physical
>>
>>
>>
>> On 09/07/2018 04:13 PM, Eric Zhang wrote:
>>>
>>> On 09/07/2018 05:26 AM, Burakov, Anatoly wrote:
>>>> On 06-Sep-18 8:34 AM, Jerin Jacob wrote:
>>>>> -----Original Message-----
>>>>>> Date: Tue, 4 Sep 2018 23:40:36 -0400
>>>>>> From: Eric Zhang <eric.zhang@windriver.com>
>>>>>> To: santosh <santosh.shukla@caviumnetworks.com>,
>>>>>> hemant.agrawal@nxp.com,
>>>>>>    Gaëtan Rivet <gaetan.rivet@6wind.com>, "Burakov, Anatoly"
>>>>>>    <anatoly.burakov@intel.com>
>>>>>> CC: bruce.richardson@intel.com, dev@dpdk.org,
>>>>>> Allain.Legacy@windriver.com,
>>>>>>    Matt.Peters@windriver.com
>>>>>> Subject: Re: [dpdk-dev] [PATCH] eal: force IOVA mode to physical
>>>>>> User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101
>>>>>>    Thunderbird/52.9.1
>>>>>>
>>>>>> On 08/30/2018 08:59 AM, santosh wrote:
>>>>>>> On Thursday 30 August 2018 05:43 PM, Hemant wrote:
>>>>>>>> External Email
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> On 8/30/2018 3:13 PM, Gaëtan Rivet wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> On Thu, Aug 30, 2018 at 10:09:04AM +0100, Burakov, Anatoly wrote:
>>>>>>>>>> On 29-Aug-18 4:58 PM, eric zhang wrote:
>>>>>>>>>>> This patch adds a configuration option to force the IOVA mode
>>>>>>>>>>> to physical address (PA). There exists virtual devices that
>>>>>>>>>>> are not directly attached to the PCI bus, and therefore the
>>>>>>>>>>> auto detection of the IOVA mode based on probing the PCI bus
>>>>>>>>>>> and IOMMU configuration may not report the required
>> addressing
>>>>>>>>>>> mode. Having the configuration option permits the mode to be
>>>>>>>>>>> explicitly configured in this scenario.
>>>>>>>>>>>
>>>>>>>>>>> Signed-off-by: eric zhang <eric.zhang@windriver.com>
>>>>>>>>>>> ---
>>>>>>>>>> Defining this at compile-time seems like an overkill. Wouldn't
>>>>>>>>>> it be better to just add an EAL command-line option to force
>>>>>>>>>> IOVA mode to a particular value?
>>>>>>>> That is a good suggestion.
>>>>>>>>>> --
>>>>>>>>>> Thanks,
>>>>>>>>>> Anatoly
>>>>>>>>> What is the bus of these devices and why not implement
>>>>>>>>> get_iommu_class in it?
>>>>>>>> There are cases, where you are using dpdk libraries with external
>>>>>>>> libraries and you need to change the default behavior DPDK lib to
>>>>>>>> use physical address instead of virtual address.
>>>>>>>> Providing an option to user will help.
>>>>>>>>
>>>>>>>>
>>>>>>> More appropriate solution could be:
>>>>>>> * Either fix it at bus layer .. i.e.. get_iommu_class()..
>>>>>>> * Or introduce something like [1] --iova-mode=<pa/va> param.
>>>>>>>
>>>>>>> Former is better solution than latter if autodetection is a key
>>>>>>> criteria.
>>>>>>> Thanks.
>>>>>>>
>>>>>>> [1] http://patchwork.dpdk.org/patch/25192/
>>>>>>>
>>>>>> It's not generic which couldn't be fixed at bus layer.
>>>>>> So what's the preference of EAL option or compile time solution?
>>>>>> Adding --iova-mode as patch [1] will overrivde auto-detection
>>>>>> rte_bus_get_iommu_class()
>>>>>> make it no use; compile time solution will align with upstream and
>>>>>> keep new atuodetection solution in #ifndef.
>>>>> If it is for vdev devices, why not introduce something like
>>>>> RTE_PCI_DRV_IOVA_AS_VA and let vdev device describe its personality.
>>>>> And based on the devices(flags) on vdev bus,
>>>>> rte_bus_get_iommu_class() of vdev can decide the mode just like PCI
>> bus.
>>>> That seems like a better option to me, +1. As far as i know, at the
>>>> moment if there are no devices attached at all, or if there are only
>>>> vdev devices attached, DPDK will default to IOVA as PA mode for no
>>>> good reason; such a change would certainly fix this.
>>> Thanks for the suggestions however our virtual device doesn't run dpdk
>>> vdev code so we can't use the flag.
>>> Notice that in eal.c there is one workaround that force iova to be PA
>>> per virtual device is not directly attached to pci. That case is
>>> checking kni module. Ours is a similar case that virtual device not
>>> attach pci directly.
>>> So we have to turn to force iova to PA either 1. compilation option 2.
>>> eal option.  Which one should be the preference by taking into
>>> consideration that align with upstream?
>>>
>>> Thanks
>>>
>> Any comments?
>>>>>> Thanks
>>>>>> Eric
>>>>>>
>>>>