[PATCH] ethdev: add template table resize API

[Etelson, Gregory]

Hello Ferruh,

> So, by design, driver will keep the old table when it is resized.
> - Can this have a performance impact, like when rules
> updated/removed/inserted driver will need to look more tables?
> - Or can this cause additional capacity complexity, like total number of
> rules will be sum of rules in all tables, but new rules only can be
> added to latest table, so number of rules can be more than size of
> latest table.
> - Or user can add more flows after resize() and this may not leave
> enough room to update old rules to new table, what is expected behavior
> for this case?
> - Or if user did not updated rules at all after resize(), after each
> rule deletion driver won't need to check if old table emptied and needs
> to be freed?
> - Or can user call resize() API multiple times, causing driver to
> maintain multiple tables? How much memory overhead this may bring?

After "resize, update, complete" sequence table performance must be
the same as before resize.
If application skiped updates or resize completion, performance is undefined.
Driver must verify that total flows number does not exceed capacity set in 
table resize.

> 'rte_flow_async_update_resized()' API is called per flow, won't this
> force application to trace which flows are created in new table and
> which are in old table, so pushing additional work to application.

Application must trace what flows require update after table resize.
As the general rule, all flows that were created before table resize call has 
returned must be updated:

"old" flows |<-----------------resize------------>| "new" flows
   update                                              keep
                unknown flow location: update

----------------------------TIME-------------------------------------->

In MLX5 PMD, if update was called with a "new" flow, the call returns will 
success, without changing the flow.

>
> Or what will happen if update() fails in the middle of update, should
> user retry, should PMD restore back the moved rules?
>

If flow update call failed, it treated as failure during flow create, update or 
destroy.

>
> I understood the logic behind the dividing responsibility to multiple
> APIs, and it makes sense, but it brings above complexities, and more
> work to application.
> Can it be possible to have monolithic API but only resize() part of it
> is blocking and update() part and later remove table part done
> asynchronously?
>

Table resize and single flow update operations consume approximately the same 
time duration.
An update of a table with 1_000_000 flows will consume driver for too much time.
During that time application will not be able to create, destroy or update 
existing "old" flows.
Such operation must be coordinated with application.

A driver could provide a batch flows update, but I don’t see how it helps.
It's ether update one or update all and the latter does not scale.

>
> I will also put more comment on the patch based on latest understanding.
>
>