[dpdk-dev] [PATCH v3 05/11] bus: get iommu class

[Hemant Agrawal]

On 7/14/2017 2:00 PM, santosh wrote:
> On Friday 14 July 2017 01:37 PM, Hemant Agrawal wrote:
>
>> On 7/11/2017 11:46 AM, Santosh Shukla wrote:
>>> API(rte_bus_get_iommu_class) helps to automatically detect and select
>>> appropriate iova mapping scheme for iommu capable device on that bus.
>>>
>>> Algorithm for iova scheme selection for bus:
>>> 0. Iterate through bus_list.
>>> 1. Collect each bus iova mode value and update into 'mode' var.
>>> 2. Here value '1' is _pa and value '2' is _va mode.
>>> So mode selection scheme is like:
>>> if mode == 2 then iova mode is _va.
>>> if mode == 1 then iova mode is _pa
>>> if mode  == 3 then iova mode ia _pa.
>>>
>>> So mode !=2  will be default iova mode.
>>>
>>> Signed-off-by: Santosh Shukla <santosh.shukla at caviumnetworks.com>
>>> Signed-off-by: Jerin Jacob <jerin.jacob at caviumnetworks.com>
>>> ---
>>>  lib/librte_eal/bsdapp/eal/rte_eal_version.map   |  1 +
>>>  lib/librte_eal/common/eal_common_bus.c          | 23 +++++++++++++++++++++++
>>>  lib/librte_eal/common/eal_common_pci.c          |  1 +
>>>  lib/librte_eal/common/include/rte_bus.h         | 22 ++++++++++++++++++++++
>>>  lib/librte_eal/linuxapp/eal/rte_eal_version.map |  1 +
>>>  5 files changed, 48 insertions(+)
>>>
>>> diff --git a/lib/librte_eal/bsdapp/eal/rte_eal_version.map b/lib/librte_eal/bsdapp/eal/rte_eal_version.map
>>> index 33c2c32c0..a2dd65a33 100644
>>> --- a/lib/librte_eal/bsdapp/eal/rte_eal_version.map
>>> +++ b/lib/librte_eal/bsdapp/eal/rte_eal_version.map
>>> @@ -202,6 +202,7 @@ DPDK_17.08 {
>>>      rte_bus_find_by_name;
>>>      rte_pci_match;
>>>      rte_pci_get_iommu_class;
>>> +    rte_bus_get_iommu_class;
>>>
>>>  } DPDK_17.05;
>>>
>>> diff --git a/lib/librte_eal/common/eal_common_bus.c b/lib/librte_eal/common/eal_common_bus.c
>>> index 08bec2d93..5d5753ac9 100644
>>> --- a/lib/librte_eal/common/eal_common_bus.c
>>> +++ b/lib/librte_eal/common/eal_common_bus.c
>>> @@ -222,3 +222,26 @@ rte_bus_find_by_device_name(const char *str)
>>>          c[0] = '\0';
>>>      return rte_bus_find(NULL, bus_can_parse, name);
>>>  }
>>> +
>>> +
>>> +/*
>>> + * Get iommu class of devices on the bus.
>>> + */
>>> +enum rte_iova_mode
>>> +rte_bus_get_iommu_class(void)
>>> +{
>>> +    int mode = 0;
>>> +    struct rte_bus *bus;
>>> +
>>> +    TAILQ_FOREACH(bus, &rte_bus_list, next) {
>>> +
>>> +        if (bus->get_iommu_class)
>>> +            mode |= bus->get_iommu_class();
>>> +    }
>>> +
>>
>> If you change the default return as '0' for buses. This code will work.
>> e.g. PCI will return '0' - when no device is probed. FSL MC will return VA. the default mode will be 'VA'
>>
> I'm confused why it won't work for fslmc case?
>
> Let me walk through the code:
>
> If no-pci device Or (future) no-platform device probed then bus opt
> to use default mapping scheme .. which is iova_pa(default scheme).
>
> Lets take PCI_bus example:
> bus->get_iommu_class()
> 	---> bus->_pci_get_iommu_class()
> 		* Now consider that no interface bound to any of PCI device, then
> 		  it will return RTE_IOVA_PA mode to rte_bus layer (aka bus->get_iommu_class).
> 		  So the iova mapping result from iommu_class scan is RTE_IOVA_PA (default).
> 		  It works for PCI_bus case, tested for both iova_va and iova_pa case, no-pci device case.
>
> Now in fslmc bus case:
> bus->get_iommu_class()
> 	---> bus->_fslmc_get_iommu_class()
> 		
> 		* IIUC your comment - You want fslmc bus to return RTE_IOVA_VA if no device
> 		  detected, Right?
why?

If bus is just present but no device is in use for dpdk, then bus should 
return 0 and it *should not* participate in the IOMMU class decision.

Right now there are only two buses. There can be more buses. (e.g. PCI, 
platform, fslmc in case of dpaa2 as well).

If the bus is not being used at all, why it influence the decision of 
other buses.

if no bus has any device, the System default is anyway PA.


> 		  if so then your fslmc bus handle should do something like below
> 			-- If no device on fslmc bus : return RTE_IOVA_VA.
> 			-- If device detected on fslmc bus and bound to iommu driver : return RTE_IOVA_VA
> 			-- If device detected fslmc but not bound to iommu drv : return RTE_IOVA_PA..
>
> make sense? If not then can you describe fslmc mapping scheme?
>
>> if fslmc is not present. The default mode will be PA.
>>
>>> +    if (mode != RTE_IOVA_VA) {
>>> +        /* Use default IOVA mode */
>>> +        mode = RTE_IOVA_PA;
>>> +    }

The system default is anyway PA.

>>> +    return mode;
>>> +}
>>> diff --git a/lib/librte_eal/common/eal_common_pci.c b/lib/librte_eal/common/eal_common_pci.c
>>> index 8b6ecebd6..bdf2e7c3a 100644
>>> --- a/lib/librte_eal/common/eal_common_pci.c
>>> +++ b/lib/librte_eal/common/eal_common_pci.c
>>> @@ -552,6 +552,7 @@ struct rte_pci_bus rte_pci_bus = {
>>>          .plug = pci_plug,
>>>          .unplug = pci_unplug,
>>>          .parse = pci_parse,
>>> +        .get_iommu_class = rte_pci_get_iommu_class,
>>>      },
>>>      .device_list = TAILQ_HEAD_INITIALIZER(rte_pci_bus.device_list),
>>>      .driver_list = TAILQ_HEAD_INITIALIZER(rte_pci_bus.driver_list),
>>> diff --git a/lib/librte_eal/common/include/rte_bus.h b/lib/librte_eal/common/include/rte_bus.h
>>> index 7a0cfb165..8b2805b7f 100644
>>> --- a/lib/librte_eal/common/include/rte_bus.h
>>> +++ b/lib/librte_eal/common/include/rte_bus.h
>>> @@ -181,6 +181,17 @@ struct rte_bus_conf {
>>>      enum rte_bus_scan_mode scan_mode; /**< Scan policy. */
>>>  };
>>>
>>> +
>>> +/**
>>> + * Get iommu class of devices on the bus.
>>> + * Check that those devices are attached to iommu driver.
>>> + *
>>> + * @return
>>> + *      enum rte_iova_mode value.
>>> + */
>>> +typedef enum rte_iova_mode (*rte_bus_get_iommu_class_t)(void);
>>> +
>>> +
>>>  /**
>>>   * A structure describing a generic bus.
>>>   */
>>> @@ -194,6 +205,7 @@ struct rte_bus {
>>>      rte_bus_unplug_t unplug;     /**< Remove single device from driver */
>>>      rte_bus_parse_t parse;       /**< Parse a device name */
>>>      struct rte_bus_conf conf;    /**< Bus configuration */
>>> +    rte_bus_get_iommu_class_t get_iommu_class; /**< Get iommu class */
>>>  };
>>>
>>>  /**
>>> @@ -293,6 +305,16 @@ struct rte_bus *rte_bus_find_by_device(const struct rte_device *dev);
>>>   */
>>>  struct rte_bus *rte_bus_find_by_name(const char *busname);
>>>
>>> +
>>> +/**
>>> + * Get iommu class of devices on the bus.
>>> + * Check that those devices are attached to iommu driver.
>>> + *
>>> + * @return
>>> + *     enum rte_iova_mode value.
>>> + */
>>> +enum rte_iova_mode rte_bus_get_iommu_class(void);
>>> +
>>>  /**
>>>   * Helper for Bus registration.
>>>   * The constructor has higher priority than PMD constructors.
>>> diff --git a/lib/librte_eal/linuxapp/eal/rte_eal_version.map b/lib/librte_eal/linuxapp/eal/rte_eal_version.map
>>> index 044f89c7c..186c7b0fd 100644
>>> --- a/lib/librte_eal/linuxapp/eal/rte_eal_version.map
>>> +++ b/lib/librte_eal/linuxapp/eal/rte_eal_version.map
>>> @@ -207,6 +207,7 @@ DPDK_17.08 {
>>>      rte_bus_find_by_name;
>>>      rte_pci_match;
>>>      rte_pci_get_iommu_class;
>>> +    rte_bus_get_iommu_class;
>>>
>>>  } DPDK_17.05;
>>>
>>>
>>
>>
>
>