Eventdev dequeue-enqueue event correlation

[Bruce Richardson]

On Wed, Oct 25, 2023 at 09:40:54AM +0200, Mattias Rönnblom wrote:
<snip for brevity> 
> Another option would be to have the current RTE_EVENT_DEV_CAP_BURST_MODE
> capable PMDs start using the "impl_opaque" field for the purpose of matching
> in and out events. It would require applications to actually start adhering
> to the "don't touch impl_opaque" requirement of the Eventdev API.
> 
> Those "fixes" are not mutually exclusive.
> 
> A side note: it's unfortunate there are no bits in the rte_event struct that
> can be used for "event id"/"event SN"/"event dequeue idx" type information,
> if an app would like to work around this issue with current PMDs.
> 

Restarting this old thread.

Having looked at the eventdev API, I think that we need to tighten up the
specification for how enqueue to dequeue correlation is to be managed.
Right now, the spec seems to imply that the impl_opaque field should be
used to correlate enqueue and dequeue events, but I'm not sure if any
drivers use that, we have quite a number of drivers which require
re-enqueue in the same order as dequeue, others which just don't support
burst enq/deq, which avoids the issue, and others which may use other
methods to achieve this. There is no documentation that I have found,
- written from the application-writers viewpoint - describing how enqueued
events for reordering, or for releasing atomic locks, are to be
correlated with the equivalent dequeued event. This looks a major
documentation gap.

I think the best approach overall is to mandate that impl_opaque field
should be used for this, as Mattias suggest above. (Using
implicit-release-only will work for atomic flows where only locks need to
be released, but I don't believe it works for reordered flows where we need
to correlate the new event with a specific order slot, not just a flow
lock) Using impl_opaque field and allowing reordering of events does open
an issue for how we deal with things like fragmentation, where one dequeued
event leads to multiple enqueued ones. For this, I suspect we may need a
new event type, called PARTIAL or FRAGMENT, to indicate that an event is to
be treated for ordering purposes the same as another event, without
actually releasing any atomic locks etc. for that event.

To try and move this along, and make the discussion more focused and
concrete, I'll do up a patchset to try and improve the eventdev
documentation, and as part of that, try and document exactly what behaviour
an app should expect when forwarding events between enqueue and dequeue.
Even if the enq-deq problem is still controversial, I think there is
probably more clarification we can do anyway.

Any further thoughts or comments?

Regards,
/Bruce