[dpdk-dev] [PATCH v3 09/11] lib/librte_vhost: vhost user support

[Xie, Huawei]

On 2/24/2015 1:36 AM, Czesnowicz, Przemyslaw wrote:
> From: "Xie, Huawei" <huawei.xie at intel.com>
>
> In rte_vhost_driver_register(), vhost unix domain socket listener fd is created
> and added to polled(based on select) fdset.
>
> In rte_vhost_driver_session_start(), fds in the fdset are checked for
> processing. If there is new connection from qemu, connection fd accepted is
> added to polled fdset. The listener and connection fds in the fdset are
> then both checked. When there is message on the connection fd, its
> callback vserver_message_handler is called to process vhost-user messages.
>
> To support identifying which virtio is from which guest VM, we could call
> rte_vhost_driver_register with different socket path. Virtio devices from
> same VM will connect to VM specific socket. The socket path information is
> stored in the virtio_net structure.
>
> Signed-off-by: Huawei Xie <huawei.xie at intel.com>
> Signed-off-by: Przemyslaw Czesnowicz <przemyslaw.czesnowicz at intel.com>
> ---
>  lib/librte_vhost/Makefile                     |   8 +-
>  lib/librte_vhost/rte_virtio_net.h             |   2 +
>  lib/librte_vhost/vhost-net.h                  |   4 +-
>  lib/librte_vhost/vhost_user/vhost-net-user.c  | 457 ++++++++++++++++++++++++++
>  lib/librte_vhost/vhost_user/vhost-net-user.h  | 106 ++++++
>  lib/librte_vhost/vhost_user/virtio-net-user.c | 314 ++++++++++++++++++
>  lib/librte_vhost/vhost_user/virtio-net-user.h |  49 +++
>  lib/librte_vhost/virtio-net.c                 |  15 +-
>  8 files changed, 948 insertions(+), 7 deletions(-)
>  create mode 100644 lib/librte_vhost/vhost_user/vhost-net-user.c
>  create mode 100644 lib/librte_vhost/vhost_user/vhost-net-user.h
>  create mode 100644 lib/librte_vhost/vhost_user/virtio-net-user.c
>  create mode 100644 lib/librte_vhost/vhost_user/virtio-net-user.h
>
> diff --git a/lib/librte_vhost/Makefile b/lib/librte_vhost/Makefile
> index 298e4f8..cac943e 100644
> --- a/lib/librte_vhost/Makefile
> +++ b/lib/librte_vhost/Makefile

cut lines...

> +
> +int
> +user_set_mem_table(struct vhost_device_ctx ctx, struct VhostUserMsg *pmsg)
> +{
> +	struct VhostUserMemory memory = pmsg->payload.memory;
> +	struct virtio_memory_regions *pregion;
> +	uint64_t mapped_address, mapped_size;
> +	struct virtio_net *dev;
> +	unsigned int idx = 0;
> +	struct orig_region_map *pregion_orig;
> +	uint64_t alignment;
> +
> +	/* unmap old memory regions one by one*/
> +	dev = get_device(ctx);
> +	if (dev == NULL)
> +		return -1;
> +
> +	if (dev->mem) {
> +		free_mem_region(dev);
> +		free(dev->mem);
> +		dev->mem = NULL;
> +	}
> +
VHOST_SET_MEM_TABLE is sent and only sent to vhost backend in
vhost_dev_start.
In FC21, we don't receive VHOST_GET_VRING_BASE to stop the vhost
device(refer to later comment below). Need root cause the change of
guest virtio driver.
For that case, we could remove the vhost device from data plane(if it is
there) when we receive this message.
Will submit a fix.
So far everything works fine but apparently we need clear spec when to
stop and start vhost device.

> +	dev->mem = calloc(1,
cut lines....

+}
+
+
+/*
+ *  In vhost-user, when we receive kick message, will test whether virtio
+ *  device is ready for packet processing.
+ */
+void
+user_set_vring_kick(struct vhost_device_ctx ctx, struct VhostUserMsg *pmsg)
+{
+	struct vhost_vring_file file;
+	struct virtio_net *dev = get_device(ctx);
+
+	file.index = pmsg->payload.u64 & VHOST_USER_VRING_IDX_MASK;
+	if (pmsg->payload.u64 & VHOST_USER_VRING_NOFD_MASK)
+		file.fd = -1;
+	else
+		file.fd = pmsg->fds[0];
+	RTE_LOG(INFO, VHOST_CONFIG,
+		"vring kick idx:%d file:%d\n", file.index, file.fd);
+	ops->set_vring_kick(ctx, &file);
+
+	if (virtio_is_ready(dev) &&
+		!(dev->flags & VIRTIO_DEV_RUNNING))
+			notify_ops->new_device(dev);
+}

When vhost device is started, qemu will call
vhost_net_start_one->vhost_dev_start->vhost_virtqueue_start.
    1. vhost_dev_start send VHOST_SET_MEM_TABLE to vhost backend.
    2. kick fd is sent and only sent to vhost backend in
vhost_virtqueue_start for each queue, through VHOST_SET_VRING_KICK message.
    2. vhost_net_start_one will send VHOST_NET_SET_BACKEND to vhost
backend if type is TAP.
For vhost-user, as we don't receive SET_BACKEND message, it makes sense
to use VHOST_SET_VRING_KICK as the start signal for vhost device.

> +
> +/*
> + * when virtio is stopped, qemu will send us the GET_VRING_BASE message.
> + */
> +int
> +user_get_vring_base(struct vhost_device_ctx ctx,
> +	struct vhost_vring_state *state)
> +{
> +	struct virtio_net *dev = get_device(ctx);
> +
> +	/* We have to stop the queue (virtio) if it is running. */
> +	if (dev->flags & VIRTIO_DEV_RUNNING)
> +		notify_ops->destroy_device(dev);
> +
> +	/* Here we are safe to get the last used index */
> +	ops->get_vring_base(ctx, state->index, state);
> +
> +	RTE_LOG(INFO, VHOST_CONFIG,
> +		"vring base idx:%d file:%d\n", state->index, state->num);
> +	/*
> +	 * Based on current qemu vhost-user implementation, this message is
> +	 * sent and only sent in vhost_vring_stop.
> +	 * TODO: cleanup the vring, it isn't usable since here.
> +	 */
> +	if (((int)dev->virtqueue[VIRTIO_RXQ]->callfd) >= 0) {
> +		close(dev->virtqueue[VIRTIO_RXQ]->callfd);
> +		dev->virtqueue[VIRTIO_RXQ]->callfd = (eventfd_t)-1;
> +	}
> +	if (((int)dev->virtqueue[VIRTIO_TXQ]->callfd) >= 0) {
> +		close(dev->virtqueue[VIRTIO_TXQ]->callfd);
> +		dev->virtqueue[VIRTIO_TXQ]->callfd = (eventfd_t)-1;
> +	}
> +
> +	return 0;
> +}

When vhost device is to be stopped(mostly when virtio device status
register is written), vhost_net_stop_one->vhost_dev_stop-> is called.
1. VHOST_NET_SET_BACKEND message is sent to vhost backend if type is TAP.
2. VHOST_GET_VRING_BASE is sent in vhost_virtqueue_stop for each virt queue.
It makes sense to use VHOST_GET_VRING_BASE as the stop signal for vhost
device.

> +
> +void
> +user_destroy_device(struct vhost_device_ctx ctx)
> +{
> +	struct virtio_net *dev = get_device(ctx);
> +
> +	if (dev && (dev->flags & VIRTIO_DEV_RUNNING))
> +		notify_ops->destroy_device(dev);
> +
> +	if (dev && dev->mem) {
> +		free_mem_region(dev);
> +		free(dev->mem);
> +		dev->mem = NULL;
> +	}
> +}
> diff --git a/lib/librte_vhost/vhost_user/virtio-net-user.h b/lib/librte_vhost/vhost_user/virtio-net-user.h
> new file mode 100644
> index 0000000..df24860
> --- /dev/null
cut lines...