[dpdk-dev,2/7] net/mlx4: fix Rx interrupts with multiple ports
Checks
Commit Message
Several Ethernet device structures are allocated on top of a common PCI
device for mlx4 adapters with multiple ports. These inherit a common
interrupt handle from their parent PCI device, which prevents Rx interrupts
from working properly on all ports as their configuration is overwritten.
Use a local interrupt handle to address this issue.
Fixes: 9f05a4b81809 ("net/mlx4: support user space Rx interrupt event")
Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com>
Acked-by: Moti Haimovsky <motih@mellanox.com>
---
drivers/net/mlx4/mlx4.c | 9 +++++++++
drivers/net/mlx4/mlx4.h | 1 +
2 files changed, 10 insertions(+)
Comments
On 6/14/2017 12:49 PM, Adrien Mazarguil wrote:
> Several Ethernet device structures are allocated on top of a common PCI
> device for mlx4 adapters with multiple ports. These inherit a common
> interrupt handle from their parent PCI device, which prevents Rx interrupts
> from working properly on all ports as their configuration is overwritten.
>
> Use a local interrupt handle to address this issue.
Hi Adrien,
I am not clear why local copy required, and main concern from my point
of view is if this is a common problem for all PMDs and should be
addressed in higher level?
The variable is already per eth_dev, but this patch moves it the private
data. What overwrites it within eth_dev?
Thanks,
ferruh
>
> Fixes: 9f05a4b81809 ("net/mlx4: support user space Rx interrupt event")
>
> Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com>
> Acked-by: Moti Haimovsky <motih@mellanox.com>
> ---
> drivers/net/mlx4/mlx4.c | 9 +++++++++
> drivers/net/mlx4/mlx4.h | 1 +
> 2 files changed, 10 insertions(+)
>
> diff --git a/drivers/net/mlx4/mlx4.c b/drivers/net/mlx4/mlx4.c
> index 178562e..2b4722f 100644
> --- a/drivers/net/mlx4/mlx4.c
> +++ b/drivers/net/mlx4/mlx4.c
> @@ -6207,6 +6207,15 @@ mlx4_pci_probe(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_dev)
>
> eth_dev->device->driver = &mlx4_driver.driver;
>
> + /*
> + * Copy and override interrupt handle to prevent it from
> + * being shared between all ethdev instances of a given PCI
> + * device. This is required to properly handle Rx interrupts
> + * on all ports.
> + */
> + priv->intr_handle_dev = *eth_dev->intr_handle;
> + eth_dev->intr_handle = &priv->intr_handle_dev;
> +
> priv->dev = eth_dev;
> eth_dev->dev_ops = &mlx4_dev_ops;
>
> diff --git a/drivers/net/mlx4/mlx4.h b/drivers/net/mlx4/mlx4.h
> index c46fc23..b74fbf8 100644
> --- a/drivers/net/mlx4/mlx4.h
> +++ b/drivers/net/mlx4/mlx4.h
> @@ -345,6 +345,7 @@ struct priv {
> unsigned int txqs_n; /* TX queues array size. */
> struct rxq *(*rxqs)[]; /* RX queues. */
> struct txq *(*txqs)[]; /* TX queues. */
> + struct rte_intr_handle intr_handle_dev; /* Device interrupt handler. */
> struct rte_intr_handle intr_handle; /* Interrupt handler. */
> struct rte_flow_drop *flow_drop_queue; /* Flow drop queue. */
> LIST_HEAD(mlx4_flows, rte_flow) flows;
>
On Fri, Jun 16, 2017 at 02:07:54PM +0100, Ferruh Yigit wrote:
> On 6/14/2017 12:49 PM, Adrien Mazarguil wrote:
> > Several Ethernet device structures are allocated on top of a common PCI
> > device for mlx4 adapters with multiple ports. These inherit a common
> > interrupt handle from their parent PCI device, which prevents Rx interrupts
> > from working properly on all ports as their configuration is overwritten.
> >
> > Use a local interrupt handle to address this issue.
>
> Hi Adrien,
>
> I am not clear why local copy required, and main concern from my point
> of view is if this is a common problem for all PMDs and should be
> addressed in higher level?
This issue only affects PMDs that handle multiple Ethernet ports through a
single PCI device. Such PMDs (like mlx4) identify themselves as PCI drivers
that manually have to register multiple rte_eth_dev instances through
rte_eth_dev_allocate(), which they then have to initialize.
> The variable is already per eth_dev, but this patch moves it the private
> data. What overwrites it within eth_dev?
Calling rte_eth_copy_pci_info() makes the rte_eth_dev structure inherit the
default interrupt handle of the underlying PCI device. By "inherit", I mean
eth_dev->intr_handle points to it, in that sense it's not per eth_dev but
per PCI device.
mlx4 Rx interrupts are associated with a given Verbs context, and each port
has its own Verbs context, so they cannot be shared, while other PMDs using
other methods for catching interrupts may be perfectly fine with a single
vector associated with the PCI device. It depends on the PMD, for instance
there is no such problem with mlx5 as exactly one PCI device is associated
with a given port.
This patch merely allocates a specific interrupt handle associated with the
eth_dev itself and makes the eth_dev handle point to that instead of the
default PCI handle. This "local" handle is initialized using the PCI handle
as a template before modifying the pointer. It's completely safe.
> > Fixes: 9f05a4b81809 ("net/mlx4: support user space Rx interrupt event")
> >
> > Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com>
> > Acked-by: Moti Haimovsky <motih@mellanox.com>
> > ---
> > drivers/net/mlx4/mlx4.c | 9 +++++++++
> > drivers/net/mlx4/mlx4.h | 1 +
> > 2 files changed, 10 insertions(+)
> >
> > diff --git a/drivers/net/mlx4/mlx4.c b/drivers/net/mlx4/mlx4.c
> > index 178562e..2b4722f 100644
> > --- a/drivers/net/mlx4/mlx4.c
> > +++ b/drivers/net/mlx4/mlx4.c
> > @@ -6207,6 +6207,15 @@ mlx4_pci_probe(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_dev)
> >
> > eth_dev->device->driver = &mlx4_driver.driver;
> >
> > + /*
> > + * Copy and override interrupt handle to prevent it from
> > + * being shared between all ethdev instances of a given PCI
> > + * device. This is required to properly handle Rx interrupts
> > + * on all ports.
> > + */
> > + priv->intr_handle_dev = *eth_dev->intr_handle;
> > + eth_dev->intr_handle = &priv->intr_handle_dev;
> > +
> > priv->dev = eth_dev;
> > eth_dev->dev_ops = &mlx4_dev_ops;
> >
> > diff --git a/drivers/net/mlx4/mlx4.h b/drivers/net/mlx4/mlx4.h
> > index c46fc23..b74fbf8 100644
> > --- a/drivers/net/mlx4/mlx4.h
> > +++ b/drivers/net/mlx4/mlx4.h
> > @@ -345,6 +345,7 @@ struct priv {
> > unsigned int txqs_n; /* TX queues array size. */
> > struct rxq *(*rxqs)[]; /* RX queues. */
> > struct txq *(*txqs)[]; /* TX queues. */
> > + struct rte_intr_handle intr_handle_dev; /* Device interrupt handler. */
> > struct rte_intr_handle intr_handle; /* Interrupt handler. */
> > struct rte_flow_drop *flow_drop_queue; /* Flow drop queue. */
> > LIST_HEAD(mlx4_flows, rte_flow) flows;
> >
>
On 6/16/2017 2:39 PM, Adrien Mazarguil wrote:
> On Fri, Jun 16, 2017 at 02:07:54PM +0100, Ferruh Yigit wrote:
>> On 6/14/2017 12:49 PM, Adrien Mazarguil wrote:
>>> Several Ethernet device structures are allocated on top of a common PCI
>>> device for mlx4 adapters with multiple ports. These inherit a common
>>> interrupt handle from their parent PCI device, which prevents Rx interrupts
>>> from working properly on all ports as their configuration is overwritten.
>>>
>>> Use a local interrupt handle to address this issue.
>>
>> Hi Adrien,
>>
>> I am not clear why local copy required, and main concern from my point
>> of view is if this is a common problem for all PMDs and should be
>> addressed in higher level?
>
> This issue only affects PMDs that handle multiple Ethernet ports through a
> single PCI device. Such PMDs (like mlx4) identify themselves as PCI drivers
> that manually have to register multiple rte_eth_dev instances through
> rte_eth_dev_allocate(), which they then have to initialize.
>
>> The variable is already per eth_dev, but this patch moves it the private
>> data. What overwrites it within eth_dev?
>
> Calling rte_eth_copy_pci_info() makes the rte_eth_dev structure inherit the
> default interrupt handle of the underlying PCI device. By "inherit", I mean
> eth_dev->intr_handle points to it, in that sense it's not per eth_dev but
> per PCI device.
>
> mlx4 Rx interrupts are associated with a given Verbs context, and each port
> has its own Verbs context, so they cannot be shared, while other PMDs using
> other methods for catching interrupts may be perfectly fine with a single
> vector associated with the PCI device. It depends on the PMD, for instance
> there is no such problem with mlx5 as exactly one PCI device is associated
> with a given port.
>
> This patch merely allocates a specific interrupt handle associated with the
> eth_dev itself and makes the eth_dev handle point to that instead of the
> default PCI handle. This "local" handle is initialized using the PCI handle
> as a template before modifying the pointer. It's completely safe.
Thanks for clarification.
>
>>> Fixes: 9f05a4b81809 ("net/mlx4: support user space Rx interrupt event")
>>>
>>> Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com>
>>> Acked-by: Moti Haimovsky <motih@mellanox.com>
>>> ---
>>> drivers/net/mlx4/mlx4.c | 9 +++++++++
>>> drivers/net/mlx4/mlx4.h | 1 +
>>> 2 files changed, 10 insertions(+)
>>>
>>> diff --git a/drivers/net/mlx4/mlx4.c b/drivers/net/mlx4/mlx4.c
>>> index 178562e..2b4722f 100644
>>> --- a/drivers/net/mlx4/mlx4.c
>>> +++ b/drivers/net/mlx4/mlx4.c
>>> @@ -6207,6 +6207,15 @@ mlx4_pci_probe(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_dev)
>>>
>>> eth_dev->device->driver = &mlx4_driver.driver;
>>>
>>> + /*
>>> + * Copy and override interrupt handle to prevent it from
>>> + * being shared between all ethdev instances of a given PCI
>>> + * device. This is required to properly handle Rx interrupts
>>> + * on all ports.
>>> + */
>>> + priv->intr_handle_dev = *eth_dev->intr_handle;
>>> + eth_dev->intr_handle = &priv->intr_handle_dev;
>>> +
>>> priv->dev = eth_dev;
>>> eth_dev->dev_ops = &mlx4_dev_ops;
>>>
>>> diff --git a/drivers/net/mlx4/mlx4.h b/drivers/net/mlx4/mlx4.h
>>> index c46fc23..b74fbf8 100644
>>> --- a/drivers/net/mlx4/mlx4.h
>>> +++ b/drivers/net/mlx4/mlx4.h
>>> @@ -345,6 +345,7 @@ struct priv {
>>> unsigned int txqs_n; /* TX queues array size. */
>>> struct rxq *(*rxqs)[]; /* RX queues. */
>>> struct txq *(*txqs)[]; /* TX queues. */
>>> + struct rte_intr_handle intr_handle_dev; /* Device interrupt handler. */
>>> struct rte_intr_handle intr_handle; /* Interrupt handler. */
>>> struct rte_flow_drop *flow_drop_queue; /* Flow drop queue. */
>>> LIST_HEAD(mlx4_flows, rte_flow) flows;
>>>
>>
>
@@ -6207,6 +6207,15 @@ mlx4_pci_probe(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_dev)
eth_dev->device->driver = &mlx4_driver.driver;
+ /*
+ * Copy and override interrupt handle to prevent it from
+ * being shared between all ethdev instances of a given PCI
+ * device. This is required to properly handle Rx interrupts
+ * on all ports.
+ */
+ priv->intr_handle_dev = *eth_dev->intr_handle;
+ eth_dev->intr_handle = &priv->intr_handle_dev;
+
priv->dev = eth_dev;
eth_dev->dev_ops = &mlx4_dev_ops;
@@ -345,6 +345,7 @@ struct priv {
unsigned int txqs_n; /* TX queues array size. */
struct rxq *(*rxqs)[]; /* RX queues. */
struct txq *(*txqs)[]; /* TX queues. */
+ struct rte_intr_handle intr_handle_dev; /* Device interrupt handler. */
struct rte_intr_handle intr_handle; /* Interrupt handler. */
struct rte_flow_drop *flow_drop_queue; /* Flow drop queue. */
LIST_HEAD(mlx4_flows, rte_flow) flows;