[dpdk-dev] [RFC PATCH 0/7] RFC:EventDev OPDL PMD

[Ma, Liang]

Hi Jerin,
   Many thanks for your comments. Please check my comment below. 

On 25 Nov 02:25, Jerin Jacob wrote:
> -----Original Message-----
> > Date: Fri, 24 Nov 2017 11:23:45 +0000
> > From: liang.j.ma at intel.com
> > To: jerin.jacob at caviumnetworks.com
> > CC: dev at dpdk.org, harry.van.haaren at intel.com, bruce.richardson at intel.com,
> >  deepak.k.jain at intel.com, john.geary at intel.com
> > Subject: [RFC PATCH 0/7] RFC:EventDev OPDL PMD
> > X-Mailer: git-send-email 2.7.5
> > 
> > From: Liang Ma <liang.j.ma at intel.com>
> 
> 
> Thanks Liang Ma for the RFC.
> 
> > 
> > The OPDL (Ordered Packet Distribution Library) eventdev is a specific
> > implementation of the eventdev API. It is particularly suited to packet
> > processing workloads that have high throughput and low latency 
> > requirements. All packets follow the same path through the device.
> > The order which packets  follow is determinted by the order in which
> > queues are set up. Packets are left on the ring until they are transmitted.
> > As a result packets do not go out of order.
> > 
> > Features:
> > 
> > The OPDL eventdev implements a subset of features of the eventdev API;
> > 
> > Queues
> >  * Atomic
> >  * Ordered (Parallel is supported as parallel is a subset of Ordered)
> >  * Single-Link
> > 
> > Ports
> >  * Load balanced (for Atomic, Ordered, Parallel queues)
> >  * Single Link (for single-link queues)
> > 
> > Single Port Queue
> > 
> > It is possible to create a Single Port Queue 
> > RTE_EVENT_QUEUE_CFG_SINGLE_LINK. Packets dequeued from this queue do
> > not need to be re-enqueued (as is the case with an ordered queue). The 
> > purpose of this queue is to allow for asynchronous handling of packets in 
> > the middle of a pipeline. Ordered queues in the middle of a pipeline 
> > cannot delete packets.
> > 
> > 
> > Queue Dependencies
> > 
> > As stated the order in which packets travel through queues is static in
> > nature. They go through the queues in the order the queues are setup at
> > initialisation rte_event_queue_setup(). For example if an application
> > sets up 3 queues, Q0, Q1, Q2 and has 3 assoicated ports P0, P1, P2 and 
> > P3 then packets must be
> > 
> >  * Enqueued onto Q0 (typically through P0), then
> > 
> >  * Dequeued from Q0 (typically through P1), then
> > 
> >  * Enqueued onto Q1 (also through P1), then
> > 
> >  * Dequeued from Q2 (typically through P2),  then
> > 
> >  * Enqueued onto Q3 (also through P2), then
> > 
> >  * Dequeued from Q3 (typically through P3) and then transmitted on the 
> >    relevant eth port
> > 
> > 
> > Limitations
> > 
> > The opdl implementation has a number of limitations. These limitations are
> > due to the static nature of the underlying queues. It is because of this
> > that the implementation can achieve such high throughput and low latency
> > 
> > The following list is a comprehensive outline of the what is supported and
> > the limitations / restrictions imposed by the opdl pmd
> > 
> >  - The order in which packets moved between queues is static and fixed 
> >    (dynamic scheduling is not supported).
> > 
> >  - NEW, RELEASE op type are not explicitly supported. RX (first enqueue) 
> >    implicitly adds NEW event types, and TX (last dequeue) implicitly does
> >    RELEASE event types.
> > 
> >  - All packets follow the same path through device queues.
> > 
> >  - Flows within queues are NOT supported.
> > 
> >  - Event priority is NOT supported.
> > 
> >  - Once the device is stopped all inflight events are lost. Applications should 
> >    clear all inflight events before stopping it.
> > 
> >  - Each port can only be associated with one queue.
> > 
> >  - Each queue can have multiple ports associated with it.
> > 
> >  - Each worker core has to dequeue the maximum burst size for that port.
> > 
> >  - For performance, the rte_event flow_id should not be updated once 
> >     packet is enqueued on RX.
> 
> Some top-level comments,
> 
> # How does application knows this PMD has above limitations?
> 
> I think, We need to add more capability RTE_EVENT_DEV_CAP_*
> to depict these constraints. On the same note, I believe this
> PMD is "radically" different than other SW/HW PMD then anyway
> we cannot write the portable application using this PMD. So there
> is no point in abstracting it as eventdev PMD. Could you please
> work on the new capabilities are required to enable this PMD.
> If it needs more capability flags to express this PMD capability,
> we might have a different library for this as it defects the
> purpose of portable eventdev applications.
>
Agree with improve capability information with add more details with 
RTE_EVENT_DEV_CAP_*. While the OPDL is designed around a different 
load-balancing architecture, that of load-balancing across pipeline 
stages where a consumer is only working on a single stage, this does not 
necessarily mean that it is completely incompatible with other eventdev 
implementations. Although, it is true that an application written to use 
one of the existing eventdevs probably won't work nicely with the OPDL
eventdev, the converse situation should work ok. That is, an application
written as a pipeline using the OPDL eventdev for load balancing should 
work without changes with the generic SW implementation, and there should 
be no reason why it should not also work with other HW implementations 
in DPDK too. 
OPDL PMD implement a subset functionality of eventdev API. I demonstrate 
OPDL on this year PRC DPDK summit,  got some early feedback from potential
users. Most of them would like to use that under existing API(aka eventdev) 
rather than another new API/lib. That let potential user easier to swap to 
exist SW/HW eventdev PMD.

> # We should not add yet another "PMD" specific example application
> in example area like "examples/eventdev_pipeline_opdl_pmd". We are
> working on making examples/eventdev/pipeline_sw_pmd to make work
> on both HW and SW.
> 
We would agree here that we don't need a proliferation of example applications.
However this is a different architecture (not a dynamic packet scheduler rather
a static pipeline work distributer), and as such perhaps we should have a 
sample app that demonstrates each contrasting architecture.

> # We should not add new PMD specific test cases in
> test/test/test_eventdev_opdl.c area.I think existing PMD specific
> test case can be moved to respective driver area, and it can do 
> the self-test by passing some command line arguments to vdev.
> 
We simply followed the existing test structure here. Would it be confusing to 
have another variant of example test code, is this done anywhere else? 
Also would there be a chance that DTS would miss running tests or not like 
having to run them using a different method. However we would defer to the consensus here.
Could you elaborate on your concerns with having another test file in the test area ?

> # Do you have relative performance number with exiting SW PMD?
> Meaning how much it improves for any specific use case WRT exiting
> SW PMD. That should a metric to define the need for new PMD.
> 
Yes, we definitely has the number. Given the limitation(Ref cover letter), OPDL 
can achieve 3X-5X times schedule rate(on Xeon 2699 v4 platform) compare with the 
standard SW PMD and no need of schedule core. This is the core value of OPDL PMD.
For certain user case, "static pipeline"  "strong order ",  OPDL is very useful 
and efficient and generic to processor arch.

> # There could be another SW driver from another vendor like ARM.
> So, I think, it is important to define the need for another SW
> PMD and how much limitation/new capabilities it needs to define to
> fit into the eventdev framework,
>
Put a summary here, OPDL is designed for certain user case, performance is increase 
dramatically. Also OPDL can fallback to standard SW PMD seamless.
That definitely fit into the eventdev API

Liang

> Jerin
> 
> > Reference
> > General concept of event driven programming model
> > [http://dpdk.org/doc/guides/eventdevs/index.html]
> > 
> > Original Ordered Pipeline Design slides 
> > [https://dpdksummit.com/Archive/pdf/2017Asia/DPDK-China2017-Ma-OPDL.pdf]
> > 
> > 
> > Liang Ma (7):
> >   event/opdl:  add the opdl ring infrastructure library
> >   event/opdl: add the opdl pmd header and init helper function
> >   event/opdl: add the opdl pmd main body and xstats helper function
> >   event/opdl: update the build system to enable compilation of pmd
> >   test/eventdev: opdl eventdev pmd unit test func and makefiles
> >   examples/eventdev_pipeline_opdl: adding example
> >   doc: add eventdev opdl pmd docuement and example usage document
> > 
> >  config/common_base                                 |    6 +
> >  doc/guides/eventdevs/index.rst                     |    1 +
> >  doc/guides/eventdevs/opdl.rst                      |  162 +++
> >  .../sample_app_ug/eventdev_pipeline_opdl_pmd.rst   |  128 +++
> >  doc/guides/sample_app_ug/index.rst                 |    1 +
> >  drivers/event/Makefile                             |    1 +
> >  drivers/event/opdl/Makefile                        |   65 ++
> >  drivers/event/opdl/opdl_evdev.c                    |  714 ++++++++++++
> >  drivers/event/opdl/opdl_evdev.h                    |  353 ++++++
> >  drivers/event/opdl/opdl_evdev_init.c               |  945 ++++++++++++++++
> >  drivers/event/opdl/opdl_evdev_xstats.c             |  205 ++++
> >  drivers/event/opdl/opdl_ring.c                     | 1170 ++++++++++++++++++++
> >  drivers/event/opdl/opdl_ring.h                     |  578 ++++++++++
> >  drivers/event/opdl/rte_pmd_evdev_opdl_version.map  |    3 +
> >  examples/eventdev_pipeline_opdl_pmd/Makefile       |   49 +
> >  examples/eventdev_pipeline_opdl_pmd/main.c         |  766 +++++++++++++
> >  mk/rte.app.mk                                      |    1 +
> >  test/test/Makefile                                 |    1 +
> >  test/test/test_eventdev_opdl.c                     | 1089 ++++++++++++++++++
> >  19 files changed, 6238 insertions(+)
> >  create mode 100644 doc/guides/eventdevs/opdl.rst
> >  create mode 100644 doc/guides/sample_app_ug/eventdev_pipeline_opdl_pmd.rst
> >  create mode 100644 drivers/event/opdl/Makefile
> >  create mode 100644 drivers/event/opdl/opdl_evdev.c
> >  create mode 100644 drivers/event/opdl/opdl_evdev.h
> >  create mode 100644 drivers/event/opdl/opdl_evdev_init.c
> >  create mode 100644 drivers/event/opdl/opdl_evdev_xstats.c
> >  create mode 100644 drivers/event/opdl/opdl_ring.c
> >  create mode 100644 drivers/event/opdl/opdl_ring.h
> >  create mode 100644 drivers/event/opdl/rte_pmd_evdev_opdl_version.map
> >  create mode 100644 examples/eventdev_pipeline_opdl_pmd/Makefile
> >  create mode 100644 examples/eventdev_pipeline_opdl_pmd/main.c
> >  create mode 100644 test/test/test_eventdev_opdl.c
> > 
> > -- 
> > 2.7.5
> > 
> > --------------------------------------------------------------
> > Intel Research and Development Ireland Limited
> > Registered in Ireland
> > Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
> > Registered Number: 308263
> > 
> > 
> > This e-mail and any attachments may contain confidential material for the sole
> > use of the intended recipient(s). Any review or distribution by others is
> > strictly prohibited. If you are not the intended recipient, please contact the
> > sender and delete all copies.