[dpdk-dev,v2] net/tap: fix device removal when no queues exist

  TAP device is created following its first queue creation. Multiple
queues can be added or removed over time. In Linux terminology those
are file descriptors which are opened or closed over time. As long as
the number of opened file descriptors is positive - TAP device will
appear as a Linux device. In case all queues are released (the
equivalent of all file descriptors being closed) the TAP device will
be removed. This can lead to abnormalities in different scenarios
where the TAP device should exist even if all its queues are released.
In order to make TAP existence independent of its number of queues -
an extra file descriptor is opened on TAP creation and is closed on
TAP closure. It's only purpose is to serve as a keep-alive mechanism
for the TAP device.

Fixes: bf7b7f437b49 ("net/tap: create netdevice during probing")
Cc: stable@dpdk.org

Signed-off-by: Ophir Munk <ophirmu@mellanox.com>
---
v1:
Initial release
v2:
Reword commit message (a fixing patch)

 drivers/net/tap/rte_eth_tap.c | 31 ++++++++++++++++++++++++-------
 drivers/net/tap/rte_eth_tap.h |  1 +
 2 files changed, 25 insertions(+), 7 deletions(-)
  

> On May 17, 2018, at 2:47 AM, Ophir Munk <ophirmu@mellanox.com> wrote:
> 
> TAP device is created following its first queue creation. Multiple
> queues can be added or removed over time. In Linux terminology those
> are file descriptors which are opened or closed over time. As long as
> the number of opened file descriptors is positive - TAP device will
> appear as a Linux device. In case all queues are released (the
> equivalent of all file descriptors being closed) the TAP device will
> be removed. This can lead to abnormalities in different scenarios
> where the TAP device should exist even if all its queues are released.
> In order to make TAP existence independent of its number of queues -
> an extra file descriptor is opened on TAP creation and is closed on
> TAP closure. It's only purpose is to serve as a keep-alive mechanism
> for the TAP device.
> 
> Fixes: bf7b7f437b49 ("net/tap: create netdevice during probing")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Ophir Munk <ophirmu@mellanox.com>
> ---
> v1:
> Initial release
> v2:
> Reword commit message (a fixing patch)
> 
> drivers/net/tap/rte_eth_tap.c | 31 ++++++++++++++++++++++++-------
> drivers/net/tap/rte_eth_tap.h |  1 +
> 2 files changed, 25 insertions(+), 7 deletions(-)

I did not see where ka_fd is set to -1 at startup, just in case we fail before the first open attempt and possible hit the close code in the remove routine. I did not look at the complete driver, but I think it maybe reasonable to add that initial variable setup.

> 
> diff --git a/drivers/net/tap/rte_eth_tap.c b/drivers/net/tap/rte_eth_tap.c
> index c006d07..6901edc 100644
> --- a/drivers/net/tap/rte_eth_tap.c
> +++ b/drivers/net/tap/rte_eth_tap.c
> @@ -929,6 +929,15 @@ tap_dev_close(struct rte_eth_dev *dev)
> 		ioctl(internals->ioctl_sock, SIOCSIFFLAGS,
> 				&internals->remote_initial_flags);
> 	}
> +
> +	if (internals->ka_fd != -1) {
> +		close(internals->ka_fd);
> +		internals->ka_fd = -1;
> +	}
> +	/*
> +	 * Since TUN device has no more opened file descriptors
> +	 * it will be removed from kernel
> +	 */
> }
> 
> static void
> @@ -1561,13 +1570,18 @@ eth_dev_tap_create(struct rte_vdev_device *vdev, char *tap_name,
> 			rte_memcpy(&pmd->eth_addr, mac_addr, sizeof(*mac_addr));
> 	}
> 
> -	/* Immediately create the netdevice (this will create the 1st queue). */
> -	/* rx queue */
> -	if (tap_setup_queue(dev, pmd, 0, 1) == -1)
> -		goto error_exit;
> -	/* tx queue */
> -	if (tap_setup_queue(dev, pmd, 0, 0) == -1)
> +	/*
> +	 * Allocate a TUN device keep-alive file descriptor that will only be
> +	 * closed when the TUN device itself is closed or removed.
> +	 * This keep-alive file descriptor will guarantee that the TUN device
> +	 * exists even when all of its queues are closed
> +	 */
> +	pmd->ka_fd = tun_alloc(pmd);
> +	if (pmd->ka_fd < 0) {
> +		pmd->ka_fd = -1;
> +		TAP_LOG(ERR, "Unable to create %s interface", tuntap_name);
> 		goto error_exit;
> +	}
> 
> 	ifr.ifr_mtu = dev->data->mtu;
> 	if (tap_ioctl(pmd, SIOCSIFMTU, &ifr, 1, LOCAL_AND_REMOTE) < 0)
> @@ -1961,9 +1975,12 @@ rte_pmd_tap_remove(struct rte_vdev_device *dev)
> 
> 	close(internals->ioctl_sock);
> 	rte_free(eth_dev->data->dev_private);
> -
> 	rte_eth_dev_release_port(eth_dev);
> 
> +	if (internals->ka_fd != -1) {
> +		close(internals->ka_fd);
> +		internals->ka_fd = -1;
> +	}
> 	return 0;
> }
> 
> diff --git a/drivers/net/tap/rte_eth_tap.h b/drivers/net/tap/rte_eth_tap.h
> index babe42d..575dce4 100644
> --- a/drivers/net/tap/rte_eth_tap.h
> +++ b/drivers/net/tap/rte_eth_tap.h
> @@ -81,6 +81,7 @@ struct pmd_internals {
> 	struct rx_queue rxq[RTE_PMD_TAP_MAX_QUEUES]; /* List of RX queues */
> 	struct tx_queue txq[RTE_PMD_TAP_MAX_QUEUES]; /* List of TX queues */
> 	struct rte_intr_handle intr_handle;          /* LSC interrupt handle. */
> +	int ka_fd;                        /* keep-alive file descriptor */
> };
> 
> /* tap_intr.c */
> -- 
> 2.7.4
> 

Regards,
Keith
  

Hi Keith,
Please find comments inline

> -----Original Message-----
> From: Wiles, Keith [mailto:keith.wiles@intel.com]
> Sent: Thursday, May 17, 2018 4:00 PM
> To: Ophir Munk <ophirmu@mellanox.com>
> Cc: dev@dpdk.org; Pascal Mazon <pascal.mazon@6wind.com>; Thomas
> Monjalon <thomas@monjalon.net>; Olga Shern <olgas@mellanox.com>;
> Shahaf Shuler <shahafs@mellanox.com>; stable@dpdk.org
> Subject: Re: [PATCH v2] net/tap: fix device removal when no queues exist
> 
> 
> 
> > On May 17, 2018, at 2:47 AM, Ophir Munk <ophirmu@mellanox.com>
> wrote:
> >
> > TAP device is created following its first queue creation. Multiple
> > queues can be added or removed over time. In Linux terminology those
> > are file descriptors which are opened or closed over time. As long as
> > the number of opened file descriptors is positive - TAP device will
> > appear as a Linux device. In case all queues are released (the
> > equivalent of all file descriptors being closed) the TAP device will
> > be removed. This can lead to abnormalities in different scenarios
> > where the TAP device should exist even if all its queues are released.
> > In order to make TAP existence independent of its number of queues -
> > an extra file descriptor is opened on TAP creation and is closed on
> > TAP closure. It's only purpose is to serve as a keep-alive mechanism
> > for the TAP device.
> >
> > Fixes: bf7b7f437b49 ("net/tap: create netdevice during probing")
> > Cc: stable@dpdk.org
> >
> > Signed-off-by: Ophir Munk <ophirmu@mellanox.com>
> > ---
> > v1:
> > Initial release
> > v2:
> > Reword commit message (a fixing patch)
> >
> > drivers/net/tap/rte_eth_tap.c | 31 ++++++++++++++++++++++++-------
> > drivers/net/tap/rte_eth_tap.h |  1 +
> > 2 files changed, 25 insertions(+), 7 deletions(-)
> 
> I did not see where ka_fd is set to -1 at startup, just in case we fail before the
> first open attempt and possible hit the close code in the remove routine. I did
> not look at the complete driver, but I think it maybe reasonable to add that
> initial variable setup.
> 

Your concern is in place. However, please note that ka_fd is allocated and initialized to 0 as part of private device structure initialization during the call to:
dev = rte_eth_vdev_allocate(vdev, sizeof(*pmd));

Therefore it is guaranteed to be 0 in case we fail before the first open attempt and if we hit the close code we will not be closing anything wrong (closing a 0 fd has no effect). 

While reading the code again I have noticed that tun_alloc() always return -1 so no need to reassign ka_fd to -1 after calling tun_alloc. Also I have fixed a typo in the commit message: It's --> Its.
I have sent v3 with the above small changes.

> >
> > diff --git a/drivers/net/tap/rte_eth_tap.c
> > b/drivers/net/tap/rte_eth_tap.c index c006d07..6901edc 100644
> > --- a/drivers/net/tap/rte_eth_tap.c
> > +++ b/drivers/net/tap/rte_eth_tap.c
> > @@ -929,6 +929,15 @@ tap_dev_close(struct rte_eth_dev *dev)
> > 		ioctl(internals->ioctl_sock, SIOCSIFFLAGS,
> > 				&internals->remote_initial_flags);
> > 	}
> > +
> > +	if (internals->ka_fd != -1) {
> > +		close(internals->ka_fd);
> > +		internals->ka_fd = -1;
> > +	}
> > +	/*
> > +	 * Since TUN device has no more opened file descriptors
> > +	 * it will be removed from kernel
> > +	 */
> > }
> >
> > static void
> > @@ -1561,13 +1570,18 @@ eth_dev_tap_create(struct rte_vdev_device
> *vdev, char *tap_name,
> > 			rte_memcpy(&pmd->eth_addr, mac_addr,
> sizeof(*mac_addr));
> > 	}
> >
> > -	/* Immediately create the netdevice (this will create the 1st queue).
> */
> > -	/* rx queue */
> > -	if (tap_setup_queue(dev, pmd, 0, 1) == -1)
> > -		goto error_exit;
> > -	/* tx queue */
> > -	if (tap_setup_queue(dev, pmd, 0, 0) == -1)
> > +	/*
> > +	 * Allocate a TUN device keep-alive file descriptor that will only be
> > +	 * closed when the TUN device itself is closed or removed.
> > +	 * This keep-alive file descriptor will guarantee that the TUN device
> > +	 * exists even when all of its queues are closed
> > +	 */
> > +	pmd->ka_fd = tun_alloc(pmd);
> > +	if (pmd->ka_fd < 0) {
> > +		pmd->ka_fd = -1;
> > +		TAP_LOG(ERR, "Unable to create %s interface",
> tuntap_name);
> > 		goto error_exit;
> > +	}
> >
> > 	ifr.ifr_mtu = dev->data->mtu;
> > 	if (tap_ioctl(pmd, SIOCSIFMTU, &ifr, 1, LOCAL_AND_REMOTE) < 0)
> @@
> > -1961,9 +1975,12 @@ rte_pmd_tap_remove(struct rte_vdev_device *dev)
> >
> > 	close(internals->ioctl_sock);
> > 	rte_free(eth_dev->data->dev_private);
> > -
> > 	rte_eth_dev_release_port(eth_dev);
> >
> > +	if (internals->ka_fd != -1) {
> > +		close(internals->ka_fd);
> > +		internals->ka_fd = -1;
> > +	}
> > 	return 0;
> > }
> >
> > diff --git a/drivers/net/tap/rte_eth_tap.h
> > b/drivers/net/tap/rte_eth_tap.h index babe42d..575dce4 100644
> > --- a/drivers/net/tap/rte_eth_tap.h
> > +++ b/drivers/net/tap/rte_eth_tap.h
> > @@ -81,6 +81,7 @@ struct pmd_internals {
> > 	struct rx_queue rxq[RTE_PMD_TAP_MAX_QUEUES]; /* List of RX
> queues */
> > 	struct tx_queue txq[RTE_PMD_TAP_MAX_QUEUES]; /* List of TX
> queues */
> > 	struct rte_intr_handle intr_handle;          /* LSC interrupt handle. */
> > +	int ka_fd;                        /* keep-alive file descriptor */
> > };
> >
> > /* tap_intr.c */
> > --
> > 2.7.4
> >
> 
> Regards,
> Keith