[PATCH] ethdev: add template table resize API

[Etelson, Gregory]

Hello Ferruh,

>> Template table creation API sets table flows capacity.
>> If application needs more flows then the table was designed for,
>> the following procedures must be completed:
>> 1. Create a new template table with larger flows capacity.
>> 2. Re-create existing flows in the new table and delete flows from
>>    the original table.
>> 3. Destroy original table.
>>
>> Application cannot always execute that procedure:
>> * Port may not have sufficient resources to allocate a new table
>>   while maintaining original table.
>> * Application may not have existing flows "recipes" to re-create
>>   flows in a new table.
>>
>> The patch defines a new API that allows application to resize
>> existing template table:
>>
>> * Resizable template table must be created with the
>> RTE_FLOW_TABLE_SPECIALIZE_RESIZABLE_TABLE bit set.
>>
>> * Application resizes existing table with the
>>   `rte_flow_template_table_resize()` function call.
>>   The table resize procedure updates the table maximal flow number
>>   only. Other table attributes are not affected by the table resize.
>>   ** The table resize procedure must not interrupt
>>      existing table flows operations in hardware.
>>   ** The table resize procedure must not alter flow handlers held by
>>      application.
>>
>> * After `rte_flow_template_table_resize()` returned, application must
>>   update all existing table flow rules by calling
>>   `rte_flow_async_update_resized()`.
>>   The table resize procedure does not change application flow handler.
>>   However, flow object can reference internal PMD resources that are
>>   obsolete after table resize.
>>   `rte_flow_async_update_resized()` moves internal flow references
>>   to the updated table resources.
>>   The flow update must not interrupt hardware flow operations.
>>
>> * When all table flow were updated, application must call
>>   `rte_flow_template_table_resize_complete()`.
>>   The function releases PMD resources related to the original
>>   table.
>>   Application can start new table resize after
>>   `rte_flow_template_table_resize_complete()` returned.
>>
>> Signed-off-by: Gregory Etelson <getelson at nvidia.com>
>> Acked-by: Ori Kam <orika at nvidia.com>
>>
>> ---
>>  app/test-pmd/cmdline_flow.c            |  86 +++++++++++++++++++--
>>  app/test-pmd/config.c                  | 102 +++++++++++++++++++++++++
>>  app/test-pmd/testpmd.h                 |   6 ++
>>
>
> It helps to document added/updated command in the commit log, and in the
> documentation 'testpmd_funcs.rst'.
> Including new attribute to create table etc...
>
> Please check following commit as sample:
> Commit 98ab16778daa ("app/testpmd: add random item support")
>

I'll post v2 with an update.

>
>>  doc/guides/rel_notes/release_24_03.rst |   2 +
>>  lib/ethdev/ethdev_trace.h              |  33 ++++++++
>>  lib/ethdev/ethdev_trace_points.c       |   9 +++
>>  lib/ethdev/rte_flow.c                  |  69 +++++++++++++++++
>>  lib/ethdev/rte_flow.h                  |  97 +++++++++++++++++++++++
>>  lib/ethdev/rte_flow_driver.h           |  15 ++++
>>  lib/ethdev/version.map                 |   3 +
>>  10 files changed, 417 insertions(+), 5 deletions(-)
>>
>
> Just to double check if there will be driver implementation of the new
> APIs in this release?
>

I'll post driver patches after DPDK API approval.

> <...>
>
>>
>> +static __rte_always_inline bool
>> +rte_flow_table_resizable(const struct rte_flow_template_table_attr *tbl_attr)
>> +{
>> +     return (tbl_attr->specialize &
>> +             RTE_FLOW_TABLE_SPECIALIZE_RESIZABLE_TABLE) != 0;
>> +}
>> +
>
> Ahh, this is the 4th API added in this patch, missed before.
>
> Why it is exposed to the application?

That API is for application and a driver.

> What is the expected usage for the function, it doesn't get 'port_id' as
> parameter and directly works on the table_attribute, feels like API

There is no usage for `port_id` in that query.

> should be for drivers?
> Why it is inlined?

I prefer inline functions to macros.

>
> <...>
>
>> +/**
>> + * @warning
>> + * @b EXPERIMENTAL: this API may change without prior notice.
>> + *
>> + * Change template table flow rules capacity.
>> + *
>> + * @param port_id
>> + *   Port identifier of Ethernet device.
>> + * @param table
>> + *   Template table to modify.
>> + * @param nb_rules
>> + *   New flow rules capacity.
>> + * @param error
>> + *   Perform verbose error reporting if not NULL.
>> + *   PMDs initialize this structure in case of error only.
>> + *
>> + * @return
>> + *   - (0) if success.
>> + *   - (-ENODEV) if *port_id* invalid.
>> + *   - (-ENOTSUP) if underlying device does not support this functionality.
>> + *   - (-EINVAL) if *table* cannot be resized or invalid *nb_rules*
>>
>
> What is invalid 'nb_rules', does it make sense to document it in above
> @param for it. Like if 0 valid, or is there a max, or is it allowed to
> shrink the size of table?

I'll post v2 with an update.

>
>
>> + */
>> +__rte_experimental
>> +int
>> +rte_flow_template_table_resize(uint16_t port_id,
>> +                            struct rte_flow_template_table *table,
>> +                            uint32_t nb_rules,
>> +                            struct rte_flow_error *error);
>> +/**
>> + * @warning
>> + * @b EXPERIMENTAL: this API may change without prior notice.
>> + *
>> + * Following table resize, update flow resources in port.
>> + *
>> + * @param port_id
>> + *   Port identifier of Ethernet device.
>> + * @param queue
>> + *   Flow queue for async operation.
>> + * @param attr
>> + *   Async operation attributes.
>> + * @param rule
>> + *   Flow rule to update.
>> + * @param user_data
>> + *   The user data that will be returned on async completion event.
>> + * @param error
>> + *   Perform verbose error reporting if not NULL.
>> + *   PMDs initialize this structure in case of error only.
>> + *
>> + * @return
>> + *   - (0) if success.
>> + *   - (-ENODEV) if *port_id* invalid.
>> + *   - (-ENOTSUP) if underlying device does not support this functionality.
>> + *   - (-EINVAL) if *rule* cannot be updated.
>> + */
>> +__rte_experimental
>> +int
>> +rte_flow_async_update_resized(uint16_t port_id, uint32_t queue,
>> +                           const struct rte_flow_op_attr *attr,
>> +                           struct rte_flow *rule, void *user_data,
>> +                           struct rte_flow_error *error);
>> +
>
> Should API have 'table' or 'template' in name, if it is out of this
> context, it is not intuitive that update_resized refers to template
> table resize. It may be confused as if this is something related to the
> rule itself resize.

That function call should have been called
`rte_flow_async_update_flow_after_table_resize`
The current version was selected after a long debate.
Please suggest an alternative.

>
>
>> +/**
>> + * @warning
>> + * @b EXPERIMENTAL: this API may change without prior notice.
>> + *
>> + * Following table resize, notify port that all table flows were updated.
>> + *
>> + * @param port_id
>> + *   Port identifier of Ethernet device.
>> + * @param table
>> + *   Template table that undergoing resize operation.
>> + * @param error
>> + *   Perform verbose error reporting if not NULL.
>> + *   PMDs initialize this structure in case of error only.
>> + *
>> + * @return
>> + *   - (0) if success.
>> + *   - (-ENODEV) if *port_id* invalid.
>> + *   - (-ENOTSUP) if underlying device does not support this functionality.
>> + *   - (-EINVAL) PMD cannot complete table resize.
>> + */
>> +__rte_experimental
>> +int
>> +rte_flow_template_table_resize_complete(uint16_t port_id,
>> +                                     struct rte_flow_template_table *table,
>> +                                     struct rte_flow_error *error);
>>
>
> Does it make sense to add a new error type to differentiate
> unrecoverable error (I don't know if there is any) and an error that is
> caused by not all rules updated?

That API should not return unrecoverable error.
When it called, all hardware related updates already completed.
No new resources required.

>
> <...>
>
>>
>>  /**
>> diff --git a/lib/ethdev/version.map b/lib/ethdev/version.map
>> index 5c4917c020..e37bab9b81 100644
>> --- a/lib/ethdev/version.map
>> +++ b/lib/ethdev/version.map
>> @@ -316,6 +316,9 @@ EXPERIMENTAL {
>>       rte_eth_recycle_rx_queue_info_get;
>>       rte_flow_group_set_miss_actions;
>>       rte_flow_calc_table_hash;
>> +     rte_flow_template_table_resize;
>> +     rte_flow_async_update_resized;
>> +     rte_flow_template_table_resize_complete;
>>
>
> New APIs should go below "# added in 24.03" comment
>
I'll post an update in v2.