[dpdk-dev] [RFC 0/5] Port Representor for control and monitoring of VF devices

[Alejandro Lucero]

On Thu, Sep 7, 2017 at 2:13 PM, Declan Doherty <declan.doherty at intel.com>
wrote:

> On 07/09/2017 11:01 AM, Alejandro Lucero wrote:
>
>> I understand this is the representor idea suiting Intel cards but it does
>> not cover other possibilities.
>>
>> At least, Netronome and Mellanox require a representor not just for
>> controlling a VF, but also for sending and receiving packets through the
>> representor PMD.
>>
>>
> Hey Alejandro,
>
> What we've proposed here doesn't preclude the support of a data path on
> the representor pmd as the functionality which the PMD supports is
> dependent on how the parent device running the representor broker
> configures each representor instance. As you note in the case of our
> current generation of IO devices supporting a data path doesn't make sense,
> but I think this framework could support that functionality if required.
>
>
Hi Declan,

Tell me if I'm wrong, but this looks to me like kernel netdev ndo_set_vf_*
functions, maybe with the idea of having more possibilities than those
currently available with the kernel. The representor idea I refer to is
completely different, and it is related to using SRIOV with OVS offload. It
has other possibilities but this is the current main reason. Some
references:


https://www.slideshare.net/Netronome/using-agilio-smartnics-for-openstack-networking-acceleration


https://netdevconf.org/1.2/slides/oct6/04_gerlitz_efraim_introduction_to_switchdev_sriov_offloads.pdf

I sent an abstract for a presentation in next Dublin Users meeting, and
>> discussing the representor idea was in the agenda.
>>
>>
> I've also submitted an presentation which I got notification was accepted
> this morning which is based on the RFC, so maybe we should collaborate on a
> presentation if there is a large overlap?


Unfortunately, I did not get my abstract approved. Maybe you could add some
slide about this other view for the shake of discussion.


>
>
> By the way, I remember there was reticence about adding control plane to
>> DPDK. What is the current official DPDK position in this regard?
>>
>>
> I also remember there was concerns about adding control plane APIs to
> DPDK, hence the RFC, but I think the advantage of the representor port
> approach is that there are no new public API being introduced, with only
> back-end support in the PMDs which wish to support this functionality.
>
>
I'd like to know what is the opinion of the DPDK community regarding the
control path.


>
>
>>
>> On Thu, Sep 7, 2017 at 9:35 AM, Mohammad Abdul Awal <
>> mohammad.abdul.awal at intel.com> wrote:
>>
>> Signed-off-by: Mohammad Abdul Awal <mohammad.abdul.awal at intel.com>
>>> Signed-off-by: Remy Horton <remy.horton at intel.com>
>>> Signed-off-by: Declan Doherty <declan.doherty at intel.com>
>>> ---
>>> This RFC introduces a port representor based model for the control,
>>> management
>>> and monitoring of SR-IOV virtual function (VF) devices for control plane
>>> applications which have bound the devices physical function.
>>>
>>> Port Representors are virtual poll mode drivers (PMD) which provide a
>>> logical
>>> representation in DPDK for VF ports for control and monitoring. Each port
>>> representor PMD represents a single VF and is associated with it's parent
>>> physical function (PF) PMD which provides the back-end hooks for the
>>> representor
>>> device ops and defines the control domain to which that port belongs.This
>>> allows to use existing DPDK APIs to monitor and control the port without
>>> the
>>> need to create and maintain VF specific APIs.
>>>
>>>
>>> +-----------------------------+   +---------------+  +---------------+
>>> |        Control Plane        |   |   Data Plane  |  |   Data Plane  |
>>> |         Application         |   |   Application |  |   Application |
>>> +-----------------------------+   +---------------+  +---------------+
>>> |         eth dev api         |   |  eth dev api  |  |  eth dev api  |
>>> +-----------------------------+   +---------------+  +---------------+
>>> +-------+  +-------+  +-------+   +---------------+  +---------------+
>>> |  PF0  |  | Port  |  | Port  |   |    VF0 PMD    |  |    VF0 PMD    |
>>> |  PMD  <--+ Rep 0 |  | Rep 1 |   +---------------+  +------+--------+
>>> |       |  | PMD   |  | PMD   |                             |
>>> +---+--^+  +-------+  +-+-----+                             |
>>>     |  |                |  |                                |
>>>     |  +----------------+  |                                |
>>>     |                      |                                |
>>>     |                      |                                |
>>> +--------------------------------+                          |
>>> |   |  HW (logical view)   |     |                          |
>>> | --+------+ +-------+ +---+---+ |                          |
>>> | |   PF   | |  VF0  | |  VF1  | |                          |
>>> | |        | |       | |       +----------------------------+
>>> | +--------+ +-------+ +-------+ |
>>> | +----------------------------+ |
>>> | |        VEB                 | |
>>> | +----------------------------+ |
>>> | +--------+                     |
>>> | |  Port  |                     |
>>> | |   0    |                     |
>>> | +--------+                     |
>>> +--------------------------------+
>>>
>>> The figure above shows a deployment where the PF is bound to a DPDK
>>> control
>>> plane application which uses representor ports to manage the
>>> configuration
>>> and
>>> monitoring of it's VF ports. Each virtual function is represented in the
>>> application by a representor port PMD which enables control of the
>>> corresponding
>>> VF through eth dev APIs on the representor PMD such as:
>>>
>>> - void rte_eth_promiscuous_enable(uint8_t port_id);
>>> - void rte_eth_promiscuous_disable(uint8_t port_id);
>>> - void rte_eth_allmulticast_enable(uint8_t port_id);
>>> - void rte_eth_allmulticast_disable(uint8_t port_id);
>>> - int rte_eth_dev_mac_addr_add(uint8_t port, struct ether_addr
>>> *mac_addr,
>>>         uint32_t pool);
>>> - int rte_eth_dev_set_vlan_offload(uint8_t port_id, int offload_mask);
>>>
>>> as well as monitoring through API's like
>>>
>>> - void rte_eth_link_get(uint8_t port_id, struct rte_eth_link *link);
>>> - int rte_eth_stats_get(uint8_t port_id, struct rte_eth_stats *stats);
>>>
>>> The port representor infrastructure is enabled through a single common,
>>> device
>>> independent, virtual PMD whos context is initialized and enabled through
>>> a
>>> broker instance running within the context of the physical function
>>> device
>>> driver.
>>>
>>> +-------------------------+       +-------------------------+
>>> |        rte_ethdev       |       |       rte_ethdev        |
>>> +-------------------------+       +-------------------------+
>>> |  Physical Function PMD  |       |  Port Reperesentor PMD  |
>>> |         +-------------+ |       | +---------+ +---------+ |
>>> |         | Representor | |       | | dev_data| | dev_ops | |
>>> |         |    Broker   | |       | +----+----+ +----+----+ |
>>> |         | +---------+ | |       +------|-----------|------+
>>> |         | | VF Port | | |              |           |
>>> |         | | Context +------------------+           |
>>> |         | +---------+ | |                          |
>>> |         | +---------+ | |                          |
>>> |         | | Handler +------------------------------+
>>> |         | |   Ops   | | |
>>> |         | +---------+ | |
>>> |         +-------------+ |
>>> +-------------------------+
>>>
>>> Creation of representor ports can be achieved either through the --vdev
>>> EAL
>>> option or through the rte_vdev_init() API. Each port representor requires
>>> the
>>> BDF of it's parent PF and the Virtual Function ID of the port which the
>>> representor will support. During initialization of the representor PMD,
>>> it
>>> calls
>>> the broker API to register itself with the PF PMD and to get it's context
>>> configured which includes the setting up of it's context and ops function
>>> handlers.
>>>
>>>
>>> As the port representor model is based around the paradigm of using
>>> standard
>>> port based APIs, it will allow future expansion of functionality without
>>> the
>>> need to add new APIs. For example it should be possible to support
>>> configuration
>>> of egress QoS parameters using existing TM APIs by extending the port
>>> representor PMD/broker infrastructure.
>>>
>>> Mohammad Abdul Awal (5):
>>>   Implemented port representor broker infrastructure, created BDF to
>>>     port     function.
>>>   added --enable-representor command line argument in EAL to load
>>>     representor broker infrastructure.
>>>   Implement port representor PMD
>>>   Enable port representor PMD and broker for fortville PMD driver
>>>   Enable port representor PMD and broker for ixgbe PMD driver.
>>>
>>>  config/common_base                                 |   5 +
>>>  drivers/net/Makefile                               |   2 +
>>>  drivers/net/i40e/Makefile                          |   1 +
>>>  drivers/net/i40e/i40e_ethdev.c                     |  17 +
>>>  drivers/net/i40e/i40e_prep_ops.c                   | 402 +++++++++
>>>  drivers/net/i40e/i40e_prep_ops.h                   |  41 +
>>>  drivers/net/i40e/rte_pmd_i40e.c                    |  47 +
>>>  drivers/net/i40e/rte_pmd_i40e.h                    |  18 +
>>>  drivers/net/ixgbe/Makefile                         |   2 +-
>>>  drivers/net/ixgbe/ixgbe_ethdev.c                   |  33 +-
>>>  drivers/net/ixgbe/ixgbe_ethdev.h                   |  11 +
>>>  drivers/net/ixgbe/ixgbe_prep_ops.c                 | 279 ++++++
>>>  drivers/net/ixgbe/ixgbe_prep_ops.h                 |  41 +
>>>  drivers/net/representor/Makefile                   |  51 ++
>>>  drivers/net/representor/rte_eth_representor.c      | 973
>>> +++++++++++++++++++++
>>>  .../representor/rte_pmd_representor_version.map    |   4 +
>>>  lib/librte_eal/bsdapp/eal/eal.c                    |   6 +
>>>  lib/librte_eal/common/eal_common_options.c         |   1 +
>>>  lib/librte_eal/common/eal_internal_cfg.h           |   2 +
>>>  lib/librte_eal/common/eal_options.h                |   2 +
>>>  lib/librte_eal/common/include/rte_eal.h            |   8 +
>>>  lib/librte_eal/linuxapp/eal/eal.c                  |   9 +
>>>  lib/librte_ether/Makefile                          |   2 +
>>>  lib/librte_ether/rte_ethdev.c                      |  93 ++
>>>  lib/librte_ether/rte_ethdev.h                      |  26 +
>>>  lib/librte_ether/rte_ethdev_vdev.h                 |  37 +-
>>>  lib/librte_ether/rte_ether_version.map             |   9 +
>>>  lib/librte_ether/rte_port_representor.c            | 160 ++++
>>>  lib/librte_ether/rte_port_representor.h            | 289 ++++++
>>>  mk/rte.app.mk                                      |   1 +
>>>  30 files changed, 2556 insertions(+), 16 deletions(-)
>>>  create mode 100644 drivers/net/i40e/i40e_prep_ops.c
>>>  create mode 100644 drivers/net/i40e/i40e_prep_ops.h
>>>  create mode 100644 drivers/net/ixgbe/ixgbe_prep_ops.c
>>>  create mode 100644 drivers/net/ixgbe/ixgbe_prep_ops.h
>>>  create mode 100644 drivers/net/representor/Makefile
>>>  create mode 100644 drivers/net/representor/rte_eth_representor.c
>>>  create mode 100644 drivers/net/representor/rte_
>>> pmd_representor_version.map
>>>  create mode 100644 lib/librte_ether/rte_port_representor.c
>>>  create mode 100644 lib/librte_ether/rte_port_representor.h
>>>
>>> --
>>> 2.7.4
>>>
>>>
>>>
>>
>