diff mbox

[dpdk-dev,v3,1/2] eal/pci: introduce a PCI driver flag

Message ID 20170609102746.13828-1-jerin.jacob@caviumnetworks.com (mailing list archive)
State Accepted, archived
Delegated to: Ferruh Yigit
Headers

Checks

Context Check Description
ci/checkpatch success coding style OK
ci/Intel-compilation success Compilation OK

Commit Message

Jerin Jacob June 9, 2017, 10:27 a.m. UTC 
  Some ethdev devices like nicvf thunderx PMD need special treatment for
Secondary queue set(SQS) PCIe VF devices, where, it expects to not unmap
or free the memory without registering the ethdev subsystem.

Introducing a new RTE_PCI_DRV_KEEP_MAPPED_RES
PCI driver flag to request PCI subsystem to not unmap the mapped PCI
resources(PCI BAR address) if unsupported device detected.

Suggested-by: Ferruh Yigit <ferruh.yigit@intel.com>
Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
---
v3:

Remove the use of RTE_PCI_DRV_KEEP_MAPPED_RES in detach()(Ferruh)
http://dpdk.org/ml/archives/dev/2017-June/067691.html

v2:
Introduced RTE_PCI_DRV_KEEP_MAPPED_RES flag scheme(Ferruh),
Based on the discussion in the following thread.
http://dpdk.org/ml/archives/dev/2017-June/067091.html
---
 lib/librte_eal/common/eal_common_pci.c  | 7 ++++++-
 lib/librte_eal/common/include/rte_pci.h | 2 ++
 2 files changed, 8 insertions(+), 1 deletion(-)

Comments

Ferruh Yigit June 9, 2017, 10:46 a.m. UTC | #1 
On 6/9/2017 11:27 AM, Jerin Jacob wrote:
> Some ethdev devices like nicvf thunderx PMD need special treatment for
> Secondary queue set(SQS) PCIe VF devices, where, it expects to not unmap
> or free the memory without registering the ethdev subsystem.
> 
> Introducing a new RTE_PCI_DRV_KEEP_MAPPED_RES
> PCI driver flag to request PCI subsystem to not unmap the mapped PCI
> resources(PCI BAR address) if unsupported device detected.
> 
> Suggested-by: Ferruh Yigit <ferruh.yigit@intel.com>
> Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>

Series Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
Ferruh Yigit June 12, 2017, 4:02 p.m. UTC | #2 
On 6/9/2017 11:46 AM, Ferruh Yigit wrote:
> On 6/9/2017 11:27 AM, Jerin Jacob wrote:
>> Some ethdev devices like nicvf thunderx PMD need special treatment for
>> Secondary queue set(SQS) PCIe VF devices, where, it expects to not unmap
>> or free the memory without registering the ethdev subsystem.
>>
>> Introducing a new RTE_PCI_DRV_KEEP_MAPPED_RES
>> PCI driver flag to request PCI subsystem to not unmap the mapped PCI
>> resources(PCI BAR address) if unsupported device detected.
>>
>> Suggested-by: Ferruh Yigit <ferruh.yigit@intel.com>
>> Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> 
> Series Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>

Series applied to dpdk-next-net/master, thanks.
Thomas Monjalon June 12, 2017, 4:21 p.m. UTC | #3 
09/06/2017 12:27, Jerin Jacob:
> Some ethdev devices like nicvf thunderx PMD need special treatment for
> Secondary queue set(SQS) PCIe VF devices, where, it expects to not unmap
> or free the memory without registering the ethdev subsystem.
> 
> Introducing a new RTE_PCI_DRV_KEEP_MAPPED_RES
> PCI driver flag to request PCI subsystem to not unmap the mapped PCI
> resources(PCI BAR address) if unsupported device detected.
> 
> Suggested-by: Ferruh Yigit <ferruh.yigit@intel.com>
> Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
[...]
> --- a/lib/librte_eal/common/eal_common_pci.c
> +++ b/lib/librte_eal/common/eal_common_pci.c
> @@ -221,7 +221,12 @@ rte_pci_probe_one_driver(struct rte_pci_driver *dr,
>  	ret = dr->probe(dr, dev);
>  	if (ret) {
>  		dev->driver = NULL;
> -		if (dr->drv_flags & RTE_PCI_DRV_NEED_MAPPING)
> +		if ((dr->drv_flags & RTE_PCI_DRV_NEED_MAPPING) &&
> +			/* Don't unmap if device is unsupported and
> +			 * driver needs mapped resources.
> +			 */
> +			!(ret > 0 &&
> +				(dr->drv_flags & RTE_PCI_DRV_KEEP_MAPPED_RES)))
>  			rte_pci_unmap_device(dev);
>  	}
>  
> --- a/lib/librte_eal/common/include/rte_pci.h
> +++ b/lib/librte_eal/common/include/rte_pci.h
> +/** Device driver needs to keep mapped resources if unsupported dev detected */
> +#define RTE_PCI_DRV_KEEP_MAPPED_RES 0x0020

If I understand well, you want to map resources but not probe it?
Shouldn't it be less hacky to probe it as a (new) null class?
Jerin Jacob June 13, 2017, 4:43 a.m. UTC | #4 
-----Original Message-----
> Date: Mon, 12 Jun 2017 18:21:33 +0200
> From: Thomas Monjalon <thomas@monjalon.net>
> To: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> Cc: dev@dpdk.org, ferruh.yigit@intel.com
> Subject: Re: [dpdk-dev]  [PATCH v3 1/2] eal/pci: introduce a PCI driver flag
> 
> 09/06/2017 12:27, Jerin Jacob:
> > Some ethdev devices like nicvf thunderx PMD need special treatment for
> > Secondary queue set(SQS) PCIe VF devices, where, it expects to not unmap
> > or free the memory without registering the ethdev subsystem.
> > 
> > Introducing a new RTE_PCI_DRV_KEEP_MAPPED_RES
> > PCI driver flag to request PCI subsystem to not unmap the mapped PCI
> > resources(PCI BAR address) if unsupported device detected.
> > 
> > Suggested-by: Ferruh Yigit <ferruh.yigit@intel.com>
> > Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> [...]
> > --- a/lib/librte_eal/common/eal_common_pci.c
> > +++ b/lib/librte_eal/common/eal_common_pci.c
> > @@ -221,7 +221,12 @@ rte_pci_probe_one_driver(struct rte_pci_driver *dr,
> >  	ret = dr->probe(dr, dev);
> >  	if (ret) {
> >  		dev->driver = NULL;
> > -		if (dr->drv_flags & RTE_PCI_DRV_NEED_MAPPING)
> > +		if ((dr->drv_flags & RTE_PCI_DRV_NEED_MAPPING) &&
> > +			/* Don't unmap if device is unsupported and
> > +			 * driver needs mapped resources.
> > +			 */
> > +			!(ret > 0 &&
> > +				(dr->drv_flags & RTE_PCI_DRV_KEEP_MAPPED_RES)))
> >  			rte_pci_unmap_device(dev);
> >  	}
> >  
> > --- a/lib/librte_eal/common/include/rte_pci.h
> > +++ b/lib/librte_eal/common/include/rte_pci.h
> > +/** Device driver needs to keep mapped resources if unsupported dev detected */
> > +#define RTE_PCI_DRV_KEEP_MAPPED_RES 0x0020
> 
> If I understand well, you want to map resources but not probe it?

Yes.

> Shouldn't it be less hacky to probe it as a (new) null class?

The Vendor and Class ID is same for those device too so we need to map
the PCI bar and have access to know the class of device. If you are concerned about
if it an common code change, My first version was without common code change.
http://dpdk.org/dev/patchwork/patch/24983/

Ferruh would like to have flag scheme, I think it make sense for
PMD maintenance perspective.
Thomas Monjalon June 13, 2017, 7:03 a.m. UTC | #5 
13/06/2017 06:43, Jerin Jacob:
> -----Original Message-----
> > Date: Mon, 12 Jun 2017 18:21:33 +0200
> > From: Thomas Monjalon <thomas@monjalon.net>
> > To: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> > Cc: dev@dpdk.org, ferruh.yigit@intel.com
> > Subject: Re: [dpdk-dev]  [PATCH v3 1/2] eal/pci: introduce a PCI driver flag
> > 
> > 09/06/2017 12:27, Jerin Jacob:
> > > Some ethdev devices like nicvf thunderx PMD need special treatment for
> > > Secondary queue set(SQS) PCIe VF devices, where, it expects to not unmap
> > > or free the memory without registering the ethdev subsystem.
> > > 
> > > Introducing a new RTE_PCI_DRV_KEEP_MAPPED_RES
> > > PCI driver flag to request PCI subsystem to not unmap the mapped PCI
> > > resources(PCI BAR address) if unsupported device detected.
> > > 
> > > Suggested-by: Ferruh Yigit <ferruh.yigit@intel.com>
> > > Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> > [...]
> > > --- a/lib/librte_eal/common/eal_common_pci.c
> > > +++ b/lib/librte_eal/common/eal_common_pci.c
> > > @@ -221,7 +221,12 @@ rte_pci_probe_one_driver(struct rte_pci_driver *dr,
> > >  	ret = dr->probe(dr, dev);
> > >  	if (ret) {
> > >  		dev->driver = NULL;
> > > -		if (dr->drv_flags & RTE_PCI_DRV_NEED_MAPPING)
> > > +		if ((dr->drv_flags & RTE_PCI_DRV_NEED_MAPPING) &&
> > > +			/* Don't unmap if device is unsupported and
> > > +			 * driver needs mapped resources.
> > > +			 */
> > > +			!(ret > 0 &&
> > > +				(dr->drv_flags & RTE_PCI_DRV_KEEP_MAPPED_RES)))
> > >  			rte_pci_unmap_device(dev);
> > >  	}
> > >  
> > > --- a/lib/librte_eal/common/include/rte_pci.h
> > > +++ b/lib/librte_eal/common/include/rte_pci.h
> > > +/** Device driver needs to keep mapped resources if unsupported dev detected */
> > > +#define RTE_PCI_DRV_KEEP_MAPPED_RES 0x0020
> > 
> > If I understand well, you want to map resources but not probe it?
> 
> Yes.
> 
> > Shouldn't it be less hacky to probe it as a (new) null class?
> 
> The Vendor and Class ID is same for those device too so we need to map
> the PCI bar and have access to know the class of device. If you are concerned about
> if it an common code change, My first version was without common code change.
> http://dpdk.org/dev/patchwork/patch/24983/
> 
> Ferruh would like to have flag scheme, I think it make sense for
> PMD maintenance perspective.

Yes

My idea was to have a new class of device interface to reserve those
resources, so the probe function would succeed.
Do you think it would be a good idea?
Jerin Jacob June 13, 2017, 7:24 a.m. UTC | #6 
-----Original Message-----
> Date: Tue, 13 Jun 2017 09:03:56 +0200
> From: Thomas Monjalon <thomas@monjalon.net>
> To: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> Cc: dev@dpdk.org, ferruh.yigit@intel.com
> Subject: Re: [dpdk-dev]  [PATCH v3 1/2] eal/pci: introduce a PCI driver flag
> 
> 13/06/2017 06:43, Jerin Jacob:
> > -----Original Message-----
> > > Date: Mon, 12 Jun 2017 18:21:33 +0200
> > > From: Thomas Monjalon <thomas@monjalon.net>
> > > To: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> > > Cc: dev@dpdk.org, ferruh.yigit@intel.com
> > > Subject: Re: [dpdk-dev]  [PATCH v3 1/2] eal/pci: introduce a PCI driver flag
> > > 
> > > 09/06/2017 12:27, Jerin Jacob:
> > > > Some ethdev devices like nicvf thunderx PMD need special treatment for
> > > > Secondary queue set(SQS) PCIe VF devices, where, it expects to not unmap
> > > > or free the memory without registering the ethdev subsystem.
> > > > 
> > > > Introducing a new RTE_PCI_DRV_KEEP_MAPPED_RES
> > > > PCI driver flag to request PCI subsystem to not unmap the mapped PCI
> > > > resources(PCI BAR address) if unsupported device detected.
> > > > 
> > > > Suggested-by: Ferruh Yigit <ferruh.yigit@intel.com>
> > > > Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> > > [...]
> > > > --- a/lib/librte_eal/common/eal_common_pci.c
> > > > +++ b/lib/librte_eal/common/eal_common_pci.c
> > > > @@ -221,7 +221,12 @@ rte_pci_probe_one_driver(struct rte_pci_driver *dr,
> > > >  	ret = dr->probe(dr, dev);
> > > >  	if (ret) {
> > > >  		dev->driver = NULL;
> > > > -		if (dr->drv_flags & RTE_PCI_DRV_NEED_MAPPING)
> > > > +		if ((dr->drv_flags & RTE_PCI_DRV_NEED_MAPPING) &&
> > > > +			/* Don't unmap if device is unsupported and
> > > > +			 * driver needs mapped resources.
> > > > +			 */
> > > > +			!(ret > 0 &&
> > > > +				(dr->drv_flags & RTE_PCI_DRV_KEEP_MAPPED_RES)))
> > > >  			rte_pci_unmap_device(dev);
> > > >  	}
> > > >  
> > > > --- a/lib/librte_eal/common/include/rte_pci.h
> > > > +++ b/lib/librte_eal/common/include/rte_pci.h
> > > > +/** Device driver needs to keep mapped resources if unsupported dev detected */
> > > > +#define RTE_PCI_DRV_KEEP_MAPPED_RES 0x0020
> > > 
> > > If I understand well, you want to map resources but not probe it?
> > 
> > Yes.
> > 
> > > Shouldn't it be less hacky to probe it as a (new) null class?
> > 
> > The Vendor and Class ID is same for those device too so we need to map
> > the PCI bar and have access to know the class of device. If you are concerned about
> > if it an common code change, My first version was without common code change.
> > http://dpdk.org/dev/patchwork/patch/24983/
> > 
> > Ferruh would like to have flag scheme, I think it make sense for
> > PMD maintenance perspective.
> 
> Yes
> 
> My idea was to have a new class of device interface to reserve those
> resources, so the probe function would succeed.
> Do you think it would be a good idea?

Currently Kernel PF code creates 12 SRIOV VF devices per port(one VF device has
8 queues === 96 queues(12VFs) for 96 cores(thunderx max cores)), out of that 1 VF
device is _primary_ which mapped to dpdk ethdev port. If probe succeeds for
another 11 VFs then too may ethdev NULL ports show up. We can support
up to 12 ports(12*12 VF = 144 ports). I think, it is not good from
end user perceptive. We already have unsupported device concept in eal device
framework(when probe returns > 0). IMHO, it OK to keep as it for
simplicity.
Thomas Monjalon June 13, 2017, 8:18 a.m. UTC | #7 
13/06/2017 09:24, Jerin Jacob:
> From: Thomas Monjalon <thomas@monjalon.net>
> > 13/06/2017 06:43, Jerin Jacob:
> > > From: Thomas Monjalon <thomas@monjalon.net>
> > > > 09/06/2017 12:27, Jerin Jacob:
> > > > > Some ethdev devices like nicvf thunderx PMD need special treatment for
> > > > > Secondary queue set(SQS) PCIe VF devices, where, it expects to not unmap
> > > > > or free the memory without registering the ethdev subsystem.
> > > > > 
> > > > > Introducing a new RTE_PCI_DRV_KEEP_MAPPED_RES
> > > > > PCI driver flag to request PCI subsystem to not unmap the mapped PCI
> > > > > resources(PCI BAR address) if unsupported device detected.
> > > > > 
> > > > > Suggested-by: Ferruh Yigit <ferruh.yigit@intel.com>
> > > > > Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> > > > [...]
> > > > > --- a/lib/librte_eal/common/eal_common_pci.c
> > > > > +++ b/lib/librte_eal/common/eal_common_pci.c
> > > > > @@ -221,7 +221,12 @@ rte_pci_probe_one_driver(struct rte_pci_driver *dr,
> > > > >  	ret = dr->probe(dr, dev);
> > > > >  	if (ret) {
> > > > >  		dev->driver = NULL;
> > > > > -		if (dr->drv_flags & RTE_PCI_DRV_NEED_MAPPING)
> > > > > +		if ((dr->drv_flags & RTE_PCI_DRV_NEED_MAPPING) &&
> > > > > +			/* Don't unmap if device is unsupported and
> > > > > +			 * driver needs mapped resources.
> > > > > +			 */
> > > > > +			!(ret > 0 &&
> > > > > +				(dr->drv_flags & RTE_PCI_DRV_KEEP_MAPPED_RES)))
> > > > >  			rte_pci_unmap_device(dev);
> > > > >  	}
> > > > >  
> > > > > --- a/lib/librte_eal/common/include/rte_pci.h
> > > > > +++ b/lib/librte_eal/common/include/rte_pci.h
> > > > > +/** Device driver needs to keep mapped resources if unsupported dev detected */
> > > > > +#define RTE_PCI_DRV_KEEP_MAPPED_RES 0x0020
> > > > 
> > > > If I understand well, you want to map resources but not probe it?
> > > 
> > > Yes.
> > > 
> > > > Shouldn't it be less hacky to probe it as a (new) null class?
> > > 
> > > The Vendor and Class ID is same for those device too so we need to map
> > > the PCI bar and have access to know the class of device. If you are concerned about
> > > if it an common code change, My first version was without common code change.
> > > http://dpdk.org/dev/patchwork/patch/24983/
> > > 
> > > Ferruh would like to have flag scheme, I think it make sense for
> > > PMD maintenance perspective.
> > 
> > Yes
> > 
> > My idea was to have a new class of device interface to reserve those
> > resources, so the probe function would succeed.
> > Do you think it would be a good idea?
> 
> Currently Kernel PF code creates 12 SRIOV VF devices per port(one VF device has
> 8 queues === 96 queues(12VFs) for 96 cores(thunderx max cores)), out of that 1 VF
> device is _primary_ which mapped to dpdk ethdev port. If probe succeeds for
> another 11 VFs then too may ethdev NULL ports show up. We can support
> up to 12 ports(12*12 VF = 144 ports). I think, it is not good from
> end user perceptive. We already have unsupported device concept in eal device
> framework(when probe returns > 0). IMHO, it OK to keep as it for
> simplicity.

Thanks for the explanation.
So we have a flag RTE_PCI_DRV_KEEP_MAPPED_RES for this kind of device.
Maybe someone else will think to another usage of a null interface,
so I would like to keep this idea floating in the air, just in case :)
diff mbox

Patch

diff --git a/lib/librte_eal/common/eal_common_pci.c b/lib/librte_eal/common/eal_common_pci.c
index 5ae520186..78b097e61 100644
--- a/lib/librte_eal/common/eal_common_pci.c
+++ b/lib/librte_eal/common/eal_common_pci.c
@@ -221,7 +221,12 @@  rte_pci_probe_one_driver(struct rte_pci_driver *dr,
 	ret = dr->probe(dr, dev);
 	if (ret) {
 		dev->driver = NULL;
-		if (dr->drv_flags & RTE_PCI_DRV_NEED_MAPPING)
+		if ((dr->drv_flags & RTE_PCI_DRV_NEED_MAPPING) &&
+			/* Don't unmap if device is unsupported and
+			 * driver needs mapped resources.
+			 */
+			!(ret > 0 &&
+				(dr->drv_flags & RTE_PCI_DRV_KEEP_MAPPED_RES)))
 			rte_pci_unmap_device(dev);
 	}
 
diff --git a/lib/librte_eal/common/include/rte_pci.h b/lib/librte_eal/common/include/rte_pci.h
index b82ab9e79..0284a6208 100644
--- a/lib/librte_eal/common/include/rte_pci.h
+++ b/lib/librte_eal/common/include/rte_pci.h
@@ -212,6 +212,8 @@  struct rte_pci_bus {
 #define RTE_PCI_DRV_INTR_LSC	0x0008
 /** Device driver supports device removal interrupt */
 #define RTE_PCI_DRV_INTR_RMV 0x0010
+/** Device driver needs to keep mapped resources if unsupported dev detected */
+#define RTE_PCI_DRV_KEEP_MAPPED_RES 0x0020
 
 /**
  * A structure describing a PCI mapping.