[dpdk-dev,v4] net/tap: fix device removal when no queues exist
Checks
Commit Message
TAP device is created following its first queue creation. Multiple
queues can be added or removed over time. In Linux terminology those
are file descriptors which are opened or closed over time. As long as
the number of opened file descriptors is positive - TAP device will
appear as a Linux device. In case all queues are released (the
equivalent of all file descriptors being closed) the TAP device will
be removed. This can lead to abnormalities in different scenarios
where the TAP device should exist even if all its queues are released.
In order to make TAP existence independent of its number of queues -
an extra file descriptor is opened on TAP creation and is closed on
TAP closure. Its only purpose is to serve as a keep-alive mechanism
for the TAP device.
Fixes: bf7b7f437b49 ("net/tap: create netdevice during probing")
Cc: stable@dpdk.org
Signed-off-by: Ophir Munk <ophirmu@mellanox.com>
---
v1:
Initial release
v2:
Reword commit message (a fixing patch)
v3:
Following review comments (return value of ka_fd)
and commit message typo fixing
v4:
Explicit setting pmd->ka_fd = -1 in eth_dev_tap_create()
drivers/net/tap/rte_eth_tap.c | 31 ++++++++++++++++++++++++-------
drivers/net/tap/rte_eth_tap.h | 1 +
2 files changed, 25 insertions(+), 7 deletions(-)
Comments
> On May 21, 2018, at 2:54 AM, Ophir Munk <ophirmu@mellanox.com> wrote:
>
> TAP device is created following its first queue creation. Multiple
> queues can be added or removed over time. In Linux terminology those
> are file descriptors which are opened or closed over time. As long as
> the number of opened file descriptors is positive - TAP device will
> appear as a Linux device. In case all queues are released (the
> equivalent of all file descriptors being closed) the TAP device will
> be removed. This can lead to abnormalities in different scenarios
> where the TAP device should exist even if all its queues are released.
> In order to make TAP existence independent of its number of queues -
> an extra file descriptor is opened on TAP creation and is closed on
> TAP closure. Its only purpose is to serve as a keep-alive mechanism
> for the TAP device.
>
> Fixes: bf7b7f437b49 ("net/tap: create netdevice during probing")
> Cc: stable@dpdk.org
>
> Signed-off-by: Ophir Munk <ophirmu@mellanox.com>
> ---
> v1:
> Initial release
> v2:
> Reword commit message (a fixing patch)
> v3:
> Following review comments (return value of ka_fd)
> and commit message typo fixing
> v4:
> Explicit setting pmd->ka_fd = -1 in eth_dev_tap_create()
>
Acked by: Keith Wiles <keith.wiles@intel.com>
Regards,
Keith
On 5/21/2018 1:52 PM, Wiles, Keith wrote:
>
>
>> On May 21, 2018, at 2:54 AM, Ophir Munk <ophirmu@mellanox.com> wrote:
>>
>> TAP device is created following its first queue creation. Multiple
>> queues can be added or removed over time. In Linux terminology those
>> are file descriptors which are opened or closed over time. As long as
>> the number of opened file descriptors is positive - TAP device will
>> appear as a Linux device. In case all queues are released (the
>> equivalent of all file descriptors being closed) the TAP device will
>> be removed. This can lead to abnormalities in different scenarios
>> where the TAP device should exist even if all its queues are released.
>> In order to make TAP existence independent of its number of queues -
>> an extra file descriptor is opened on TAP creation and is closed on
>> TAP closure. Its only purpose is to serve as a keep-alive mechanism
>> for the TAP device.
>>
>> Fixes: bf7b7f437b49 ("net/tap: create netdevice during probing")
>> Cc: stable@dpdk.org
>>
>> Signed-off-by: Ophir Munk <ophirmu@mellanox.com>
>> ---
>> v1:
>> Initial release
>> v2:
>> Reword commit message (a fixing patch)
>> v3:
>> Following review comments (return value of ka_fd)
>> and commit message typo fixing
>> v4:
>> Explicit setting pmd->ka_fd = -1 in eth_dev_tap_create()
>>
>
> Acked by: Keith Wiles <keith.wiles@intel.com>
Applied to dpdk-next-net/master, thanks.
Hi Ophir,
One suggestion shared inline to email
<Snipped>
>
> diff --git a/drivers/net/tap/rte_eth_tap.c b/drivers/net/tap/rte_eth_tap.c index
> c006d07..52ef799 100644
> --- a/drivers/net/tap/rte_eth_tap.c
> +++ b/drivers/net/tap/rte_eth_tap.c
> @@ -929,6 +929,15 @@ tap_dev_close(struct rte_eth_dev *dev)
> ioctl(internals->ioctl_sock, SIOCSIFFLAGS,
> &internals->remote_initial_flags);
> }
> +
> + if (internals->ka_fd != -1) {
> + close(internals->ka_fd);
Do we need to notify the user which fd is been closed via LOG DEBUG?
> + internals->ka_fd = -1;
> + }
> + /*
> + * Since TUN device has no more opened file descriptors
> + * it will be removed from kernel
> + */
> }
>
> static void
> @@ -1549,6 +1558,7 @@ eth_dev_tap_create(struct rte_vdev_device *vdev,
> char *tap_name,
> dev->intr_handle = &pmd->intr_handle;
>
> /* Presetup the fds to -1 as being not valid */
> + pmd->ka_fd = -1;
> for (i = 0; i < RTE_PMD_TAP_MAX_QUEUES; i++) {
> pmd->rxq[i].fd = -1;
> pmd->txq[i].fd = -1;
> @@ -1561,13 +1571,17 @@ eth_dev_tap_create(struct rte_vdev_device *vdev,
> char *tap_name,
> rte_memcpy(&pmd->eth_addr, mac_addr,
> sizeof(*mac_addr));
> }
>
> - /* Immediately create the netdevice (this will create the 1st queue). */
> - /* rx queue */
> - if (tap_setup_queue(dev, pmd, 0, 1) == -1)
> - goto error_exit;
> - /* tx queue */
> - if (tap_setup_queue(dev, pmd, 0, 0) == -1)
> + /*
> + * Allocate a TUN device keep-alive file descriptor that will only be
> + * closed when the TUN device itself is closed or removed.
> + * This keep-alive file descriptor will guarantee that the TUN device
> + * exists even when all of its queues are closed
> + */
> + pmd->ka_fd = tun_alloc(pmd);
> + if (pmd->ka_fd == -1) {
> + TAP_LOG(ERR, "Unable to create %s interface", tuntap_name);
> goto error_exit;
> + }
>
> ifr.ifr_mtu = dev->data->mtu;
> if (tap_ioctl(pmd, SIOCSIFMTU, &ifr, 1, LOCAL_AND_REMOTE) < 0) @@
> -1961,9 +1975,12 @@ rte_pmd_tap_remove(struct rte_vdev_device *dev)
>
> close(internals->ioctl_sock);
> rte_free(eth_dev->data->dev_private);
> -
> rte_eth_dev_release_port(eth_dev);
>
> + if (internals->ka_fd != -1) {
> + close(internals->ka_fd);
> + internals->ka_fd = -1;
> + }
> return 0;
> }
>
> diff --git a/drivers/net/tap/rte_eth_tap.h b/drivers/net/tap/rte_eth_tap.h index
> babe42d..575dce4 100644
> --- a/drivers/net/tap/rte_eth_tap.h
> +++ b/drivers/net/tap/rte_eth_tap.h
> @@ -81,6 +81,7 @@ struct pmd_internals {
> struct rx_queue rxq[RTE_PMD_TAP_MAX_QUEUES]; /* List of RX
> queues */
> struct tx_queue txq[RTE_PMD_TAP_MAX_QUEUES]; /* List of TX queues
> */
> struct rte_intr_handle intr_handle; /* LSC interrupt handle. */
> + int ka_fd; /* keep-alive file descriptor */
> };
>
> /* tap_intr.c */
> --
> 2.7.4
> On May 22, 2018, at 11:50 PM, Varghese, Vipin <vipin.varghese@intel.com> wrote:
>
> Hi Ophir,
>
> One suggestion shared inline to email
>
> <Snipped>
>
>>
>> diff --git a/drivers/net/tap/rte_eth_tap.c b/drivers/net/tap/rte_eth_tap.c index
>> c006d07..52ef799 100644
>> --- a/drivers/net/tap/rte_eth_tap.c
>> +++ b/drivers/net/tap/rte_eth_tap.c
>> @@ -929,6 +929,15 @@ tap_dev_close(struct rte_eth_dev *dev)
>> ioctl(internals->ioctl_sock, SIOCSIFFLAGS,
>> &internals->remote_initial_flags);
>> }
>> +
>> + if (internals->ka_fd != -1) {
>> + close(internals->ka_fd);
>
> Do we need to notify the user which fd is been closed via LOG DEBUG?
Why would we want to have a LOG DEBUG here, it would make the debug output a bit chatty IMO. I mean you could have one, but it seems ok as it is to me.
>
>> + internals->ka_fd = -1;
>> + }
>> + /*
>> + * Since TUN device has no more opened file descriptors
>> + * it will be removed from kernel
>> + */
>> }
>>
>> static void
>> @@ -1549,6 +1558,7 @@ eth_dev_tap_create(struct rte_vdev_device *vdev,
>> char *tap_name,
>> dev->intr_handle = &pmd->intr_handle;
>>
>> /* Presetup the fds to -1 as being not valid */
>> + pmd->ka_fd = -1;
>> for (i = 0; i < RTE_PMD_TAP_MAX_QUEUES; i++) {
>> pmd->rxq[i].fd = -1;
>> pmd->txq[i].fd = -1;
>> @@ -1561,13 +1571,17 @@ eth_dev_tap_create(struct rte_vdev_device *vdev,
>> char *tap_name,
>> rte_memcpy(&pmd->eth_addr, mac_addr,
>> sizeof(*mac_addr));
>> }
>>
>> - /* Immediately create the netdevice (this will create the 1st queue). */
>> - /* rx queue */
>> - if (tap_setup_queue(dev, pmd, 0, 1) == -1)
>> - goto error_exit;
>> - /* tx queue */
>> - if (tap_setup_queue(dev, pmd, 0, 0) == -1)
>> + /*
>> + * Allocate a TUN device keep-alive file descriptor that will only be
>> + * closed when the TUN device itself is closed or removed.
>> + * This keep-alive file descriptor will guarantee that the TUN device
>> + * exists even when all of its queues are closed
>> + */
>> + pmd->ka_fd = tun_alloc(pmd);
>> + if (pmd->ka_fd == -1) {
>> + TAP_LOG(ERR, "Unable to create %s interface", tuntap_name);
>> goto error_exit;
>> + }
>>
>> ifr.ifr_mtu = dev->data->mtu;
>> if (tap_ioctl(pmd, SIOCSIFMTU, &ifr, 1, LOCAL_AND_REMOTE) < 0) @@
>> -1961,9 +1975,12 @@ rte_pmd_tap_remove(struct rte_vdev_device *dev)
>>
>> close(internals->ioctl_sock);
>> rte_free(eth_dev->data->dev_private);
>> -
>> rte_eth_dev_release_port(eth_dev);
>>
>> + if (internals->ka_fd != -1) {
>> + close(internals->ka_fd);
>> + internals->ka_fd = -1;
>> + }
>> return 0;
>> }
>>
>> diff --git a/drivers/net/tap/rte_eth_tap.h b/drivers/net/tap/rte_eth_tap.h index
>> babe42d..575dce4 100644
>> --- a/drivers/net/tap/rte_eth_tap.h
>> +++ b/drivers/net/tap/rte_eth_tap.h
>> @@ -81,6 +81,7 @@ struct pmd_internals {
>> struct rx_queue rxq[RTE_PMD_TAP_MAX_QUEUES]; /* List of RX
>> queues */
>> struct tx_queue txq[RTE_PMD_TAP_MAX_QUEUES]; /* List of TX queues
>> */
>> struct rte_intr_handle intr_handle; /* LSC interrupt handle. */
>> + int ka_fd; /* keep-alive file descriptor */
>> };
>>
>> /* tap_intr.c */
>> --
>> 2.7.4
>
Regards,
Keith
Sure, shared a suggestion. If not required can drop the same.
> -----Original Message-----
> From: Wiles, Keith
> Sent: Wednesday, May 23, 2018 10:24 AM
> To: Varghese, Vipin <vipin.varghese@intel.com>
> Cc: Ophir Munk <ophirmu@mellanox.com>; dev@dpdk.org; Pascal Mazon
> <pascal.mazon@6wind.com>; Thomas Monjalon <thomas@monjalon.net>;
> Olga Shern <olgas@mellanox.com>; Shahaf Shuler <shahafs@mellanox.com>;
> stable@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH v4] net/tap: fix device removal when no queues
> exist
>
>
>
> > On May 22, 2018, at 11:50 PM, Varghese, Vipin <vipin.varghese@intel.com>
> wrote:
> >
> > Hi Ophir,
> >
> > One suggestion shared inline to email
> >
> > <Snipped>
> >
> >>
> >> diff --git a/drivers/net/tap/rte_eth_tap.c
> >> b/drivers/net/tap/rte_eth_tap.c index
> >> c006d07..52ef799 100644
> >> --- a/drivers/net/tap/rte_eth_tap.c
> >> +++ b/drivers/net/tap/rte_eth_tap.c
> >> @@ -929,6 +929,15 @@ tap_dev_close(struct rte_eth_dev *dev)
> >> ioctl(internals->ioctl_sock, SIOCSIFFLAGS,
> >> &internals->remote_initial_flags);
> >> }
> >> +
> >> + if (internals->ka_fd != -1) {
> >> + close(internals->ka_fd);
> >
> > Do we need to notify the user which fd is been closed via LOG DEBUG?
>
> Why would we want to have a LOG DEBUG here, it would make the debug
> output a bit chatty IMO. I mean you could have one, but it seems ok as it is to
> me.
>
> >
> >> + internals->ka_fd = -1;
> >> + }
> >> + /*
> >> + * Since TUN device has no more opened file descriptors
> >> + * it will be removed from kernel
> >> + */
> >> }
> >>
> >> static void
> >> @@ -1549,6 +1558,7 @@ eth_dev_tap_create(struct rte_vdev_device
> >> *vdev, char *tap_name,
> >> dev->intr_handle = &pmd->intr_handle;
> >>
> >> /* Presetup the fds to -1 as being not valid */
> >> + pmd->ka_fd = -1;
> >> for (i = 0; i < RTE_PMD_TAP_MAX_QUEUES; i++) {
> >> pmd->rxq[i].fd = -1;
> >> pmd->txq[i].fd = -1;
> >> @@ -1561,13 +1571,17 @@ eth_dev_tap_create(struct rte_vdev_device
> >> *vdev, char *tap_name,
> >> rte_memcpy(&pmd->eth_addr, mac_addr,
> sizeof(*mac_addr));
> >> }
> >>
> >> - /* Immediately create the netdevice (this will create the 1st queue). */
> >> - /* rx queue */
> >> - if (tap_setup_queue(dev, pmd, 0, 1) == -1)
> >> - goto error_exit;
> >> - /* tx queue */
> >> - if (tap_setup_queue(dev, pmd, 0, 0) == -1)
> >> + /*
> >> + * Allocate a TUN device keep-alive file descriptor that will only be
> >> + * closed when the TUN device itself is closed or removed.
> >> + * This keep-alive file descriptor will guarantee that the TUN device
> >> + * exists even when all of its queues are closed
> >> + */
> >> + pmd->ka_fd = tun_alloc(pmd);
> >> + if (pmd->ka_fd == -1) {
> >> + TAP_LOG(ERR, "Unable to create %s interface", tuntap_name);
> >> goto error_exit;
> >> + }
> >>
> >> ifr.ifr_mtu = dev->data->mtu;
> >> if (tap_ioctl(pmd, SIOCSIFMTU, &ifr, 1, LOCAL_AND_REMOTE) < 0) @@
> >> -1961,9 +1975,12 @@ rte_pmd_tap_remove(struct rte_vdev_device *dev)
> >>
> >> close(internals->ioctl_sock);
> >> rte_free(eth_dev->data->dev_private);
> >> -
> >> rte_eth_dev_release_port(eth_dev);
> >>
> >> + if (internals->ka_fd != -1) {
> >> + close(internals->ka_fd);
> >> + internals->ka_fd = -1;
> >> + }
> >> return 0;
> >> }
> >>
> >> diff --git a/drivers/net/tap/rte_eth_tap.h
> >> b/drivers/net/tap/rte_eth_tap.h index
> >> babe42d..575dce4 100644
> >> --- a/drivers/net/tap/rte_eth_tap.h
> >> +++ b/drivers/net/tap/rte_eth_tap.h
> >> @@ -81,6 +81,7 @@ struct pmd_internals {
> >> struct rx_queue rxq[RTE_PMD_TAP_MAX_QUEUES]; /* List of RX
> queues */
> >> struct tx_queue txq[RTE_PMD_TAP_MAX_QUEUES]; /* List of TX queues
> */
> >> struct rte_intr_handle intr_handle; /* LSC interrupt handle. */
> >> + int ka_fd; /* keep-alive file descriptor */
> >> };
> >>
> >> /* tap_intr.c */
> >> --
> >> 2.7.4
> >
>
> Regards,
> Keith
@@ -929,6 +929,15 @@ tap_dev_close(struct rte_eth_dev *dev)
ioctl(internals->ioctl_sock, SIOCSIFFLAGS,
&internals->remote_initial_flags);
}
+
+ if (internals->ka_fd != -1) {
+ close(internals->ka_fd);
+ internals->ka_fd = -1;
+ }
+ /*
+ * Since TUN device has no more opened file descriptors
+ * it will be removed from kernel
+ */
}
static void
@@ -1549,6 +1558,7 @@ eth_dev_tap_create(struct rte_vdev_device *vdev, char *tap_name,
dev->intr_handle = &pmd->intr_handle;
/* Presetup the fds to -1 as being not valid */
+ pmd->ka_fd = -1;
for (i = 0; i < RTE_PMD_TAP_MAX_QUEUES; i++) {
pmd->rxq[i].fd = -1;
pmd->txq[i].fd = -1;
@@ -1561,13 +1571,17 @@ eth_dev_tap_create(struct rte_vdev_device *vdev, char *tap_name,
rte_memcpy(&pmd->eth_addr, mac_addr, sizeof(*mac_addr));
}
- /* Immediately create the netdevice (this will create the 1st queue). */
- /* rx queue */
- if (tap_setup_queue(dev, pmd, 0, 1) == -1)
- goto error_exit;
- /* tx queue */
- if (tap_setup_queue(dev, pmd, 0, 0) == -1)
+ /*
+ * Allocate a TUN device keep-alive file descriptor that will only be
+ * closed when the TUN device itself is closed or removed.
+ * This keep-alive file descriptor will guarantee that the TUN device
+ * exists even when all of its queues are closed
+ */
+ pmd->ka_fd = tun_alloc(pmd);
+ if (pmd->ka_fd == -1) {
+ TAP_LOG(ERR, "Unable to create %s interface", tuntap_name);
goto error_exit;
+ }
ifr.ifr_mtu = dev->data->mtu;
if (tap_ioctl(pmd, SIOCSIFMTU, &ifr, 1, LOCAL_AND_REMOTE) < 0)
@@ -1961,9 +1975,12 @@ rte_pmd_tap_remove(struct rte_vdev_device *dev)
close(internals->ioctl_sock);
rte_free(eth_dev->data->dev_private);
-
rte_eth_dev_release_port(eth_dev);
+ if (internals->ka_fd != -1) {
+ close(internals->ka_fd);
+ internals->ka_fd = -1;
+ }
return 0;
}
@@ -81,6 +81,7 @@ struct pmd_internals {
struct rx_queue rxq[RTE_PMD_TAP_MAX_QUEUES]; /* List of RX queues */
struct tx_queue txq[RTE_PMD_TAP_MAX_QUEUES]; /* List of TX queues */
struct rte_intr_handle intr_handle; /* LSC interrupt handle. */
+ int ka_fd; /* keep-alive file descriptor */
};
/* tap_intr.c */