[dpdk-dev] [PATCH v5 06/20] event/sw: add support for event queues

Van Haaren, Harry harry.van.haaren at intel.com
Wed Mar 29 10:28:12 CEST 2017


> From: Jerin Jacob [mailto:jerin.jacob at caviumnetworks.com]
> Sent: Tuesday, March 28, 2017 6:36 PM
> To: Van Haaren, Harry <harry.van.haaren at intel.com>
> Cc: dev at dpdk.org; Richardson, Bruce <bruce.richardson at intel.com>
> Subject: Re: [PATCH v5 06/20] event/sw: add support for event queues
> 

<snip IQ priority question>


> > > A few question on everyone benefit:
> > >
> > > 1) Does RTE_EVENT_QUEUE_CFG_SINGLE_LINK has any other meaning other than an
> > > event queue linked only to single port?  Based on the discussions, It was
> > > add in the header file so that SW PMD can know upfront only single port
> > > will be linked to the given event queue. It is added as an optimization for SW
> > > PMD. Does it has any functional expectation?
> >
> > In the context of the SW PMD, SINGLE_LINK means that a specific queue and port have a unique
> relationship in that there is only connection. This allows bypassing of Atomic, Ordering and
> Load-Balancing code. The result is a good performance increase, particularly if the worker port
> dequeue depth is large, as then large bursts of packets can be dequeued with little overhead.
> >
> > As a result, (ATOMIC | SINGLE_LINK) is not a supported combination for the sw pmd queue
> types.
> > To be more precise, a SINGLE_LINK is its own queue type, and can not be OR-ed with any other
> type.
> >
> >
> > > 2) Based on following topology given in documentation patch for queue
> > > based event pipelining,
> > >
> > >   rx_port    w1_port
> > > 	 \     /         \
> > > 	  qid0 - w2_port - qid1
> > > 	       \         /     \
> > > 		    w3_port        tx_port
> > >
> > > a) I understand, rx_port is feeding events to qid0
> > > b) But, Do you see any issue with following model? IMO, It scales well
> > > linearly based on number of cores available to work(Since it is ATOMIC to
> > > ATOMIC). Nothing wrong with
> > > qid1 just connects to tx_port, I am just trying understand the rational
> > > behind it?
> > >
> > >   rx_port   w1_port         w1_port
> > > 	 \     /         \     /
> > > 	  qid0 - w2_port - qid1- w2_port
> > > 	       \         /     \
> > > 		   w3_port         w3_port
> >
> >
> > This is also a valid model from the SW eventdev.
> 
> OK. If understand it correctly, On the above topology,  Even though you
> make qid2 as ATOMIC. SW PMD will not maintain ingress order when comes out of
> qid1 on different workers.


If qid0 is ORDERED, and qid1 is Atomic, then the following happens;
- after qid 0, the packets are sprayed across cores,
- they are returned out of order by worker cores
- *at the start* of qid1, packets are re-ordered back into ingress order (maintain 100% of ordering)
- on dequeue from qid1, the atomic flow distribution will keep order per flow


> A SINGLE_LINK queue with one port attached
> scheme is required at end of the pipeline or where ever ordering has to be
> maintained. Is my understanding correct?


Not quite, the SINGLE_LINK is not required at the end - we just see it as useful for common use cases.
If not useful, there is no reason (due to SW PMD) for an application to create this SINGLE_LINK to finish the pipeline.
If you have three cores that wish to TX, the above pipeline is 100% valid in the SW PMD case.


> > The value of using a SINGLE_LINK at the end of a pipeline is
> > A) can TX all traffic on a single core (using a single queue)
> > B) re-ordering of traffic from the previous stage is possible
> >
> > To illustrate (B), a very simple pipeline here
> >
> >  RX port -> QID #1 (Ordered) -> workers(eg 4 ports) -> QID # 2 (SINGLE_LINK to tx) -> TX port
> >
> > Here, QID #1 is allowed to send the packets out of order to the 4 worker ports - because they
> are later passed back to the eventdev for re-ordering before they get to the SINGLE_LINK stage,
> and then TX in the correct order.
> >
> >
> > > 3)
> > > > Does anybody have a need for a queue to be both Atomic *and* Single-link?  I understand
> the
> > > current API doesn't prohibit it, but I don't see the actual use-case in which that may be
> > > useful. Atomic implies load-balancing is occurring, single link implies there is only one
> > > consuming core. Those seem like opposites to me?
> > >
> > > I can think about the following use case:
> > >
> > > topology:
> > >
> > >   rx_port    w1_port
> > > 	 \     /         \
> > > 	  qid0 - w2_port - qid1
> > > 	       \         /     \
> > > 		    w3_port        tx_port
> > >
> > > Use case:
> > >
> > > Queue based event pipeling:
> > > ORERDED(Stage1) to ATOMIC(Stage2) pipeline:
> > > - For ingress order maintenance
> > > - For executing Stage 1 in parallel for better scaling
> > > i.e A fat flow can spray over N cores while maintaining the ingress
> > > order when it sends out on the wire(after consuming from tx_port)
> > >
> > > I am not sure how SW PMD work in the use case of ingress order maintenance.
> >
> > I think my illustration of (B) above is the same use-case as you have here. Instead of using
> an ATOMIC stage2, the SW PMD benefits from using the SINGLE_LINK port/queue, and the
> SINGLE_LINK queue ensures ingress order is also egress order to the TX port.
> >
> >
> > > But the HW and header file expects this form:
> > > Snippet from header file:
> > > --
> > >  * The source flow ordering from an event queue is maintained when events are
> > >  * enqueued to their destination queue within the same ordered flow context.
> > >  *
> > >  * Events from the source queue appear in their original order when dequeued
> > >  * from a destination queue.
> > > --
> > > Here qid0 is source queue with ORDERED sched_type and qid1 is destination
> > > queue with ATOMIC sched_type. qid1 can be linked to only port(tx_port).
> > >
> > > Are we on same page? If not, let me know the differences? We will try to
> > > accommodate the same in header file.
> >
> > Yes I think we are saying the same thing, using slightly different words.
> >
> > To summarize;
> > - SW PMD sees SINGLE_LINK as its own queue type, and does not support load-balanced (Atomic
> Ordered, Parallel) queue functionality.
> > - SW PMD would use a SINGLE_LINK queue/port for the final stage of a pipeline
> >    A) to allow re-ordering to happen if required
> >    B) to merge traffic from multiple ports into a single stream for TX
> >
> > A possible solution;
> > 1) The application creates a SINGLE_LINK for the purpose of ensuring re-ordering is taking
> place as expected, and linking only one port for TX.
> 
> The only issue is in Low-end cores case it wont scale. TX core will become as
> bottleneck and we need to have different pipelines based on the amount of traffic(40G or 10G)
> a core can handle.


See above - the SINGLE_LINK isn't required to maintain ordering. Using multiple TX cores is also valid in SW PMD.


> > 2) SW PMDs can create a SINGLE_LINK queue type, and benefit from the optimization
> 
> Yes.
> 
> > 3) HW PMDs can ignore the "SINGLE_LINK" aspect and uses an ATOMIC instead (as per your
> example in 3) above)
> 
> But topology will be fixed for both HW and SW. An extra port and
> extra core needs to wasted for ordering business in case HW. Right?


Nope, no wasting cores, see above :) The SINGLE_LINK is just an easy way to "fan in" traffic from lots of cores to one core (in a performant way in SW) to allow a single core do TX. A typical use-case might be putting RX and TX on the same core - TX is just a dequeue from a port with a SINGLE_LINK queue, and an enqueue to NIC.


Summary from the SW PMD point-of-view; 
- SINGLE_LINK is its own queue type
- SINGLE_LINK queue can NOT schedule according to (Atomic, Ordered or Parallel) rules

Is that acceptable from an API and HW point of view? 

If so, I will send a new patch for the API to specify more clearly what SINGLE_LINK is.
If not, I'm open to using a capability flag to solve the problem but my understanding right now is that there is no need.



> I think, we can roll out something based on capability.

Yes, if required that would be a good solution.


> > The application doesn't have to change anything, and just configures its pipeline. The PMD is
> able to optimize if it makes sense (SW) or just use another queue type to provide the same
> functionality to the application (HW).
> >
> > Thoughts? -Harry


More information about the dev mailing list