[dpdk-dev] [PATCH v5 3/8] ethdev: reserve capability flags for PMD-specific API

[Ananyev, Konstantin]

Hi Adrien,

> 
> Hi Konstantin,
> 
> On Thu, Jan 05, 2017 at 11:32:38AM +0000, Ananyev, Konstantin wrote:
> > Hi Adrien,
> >
> > >
> > > On Thu, Jan 05, 2017 at 07:56:08AM +0800, Tiwei Bie wrote:
> > > > On Thu, Jan 05, 2017 at 01:44:18AM +0800, Ananyev, Konstantin wrote:
> > > > [...]
> > > > > > >
> > > > > > > I understand that.
> > > > > > > My question was: suppose user would like to create a bonded device over 2 NICs.
> > > > > > > One of them is ixgbe, while other would be some other type.
> > > > > > > In future get_dev_info() for each of them might return DEV_RX_OFFLOAD_RESERVED_0  bit as set.
> > > > > > > But it would mean completely different thing.
> > > > > > > How bonded device would know that to deal properly?
> > > > > > >
> > > > > > > Another example - user has 2 NICs of different type and would like to send the same packet on both of them simultaneously.
> > > > > > > As PKT_TX_RESERVED might mean different things for these devices, and user would like to use let say
> > > > > > > PKT_TX_IXGBE_MACSEC on one of them, he would need to do a copy of them, instead just increment a refcnt.
> > > > > > >
> > > > > > > Similar issues might arise at RX handling: user got a packet with PKT_RX_RESERVED_0 set.
> > > > > > > What does it really mean if there are different NIC types in the system?
> > > > > > > The only way to answer that question, as I can see,  is to keep track from what NIC that packet was received.
> > > > > > > Which I suppose, is not always convenient.
> > > > > > >
> > > > > >
> > > > > > The main purpose is to put the PMD-specific APIs in a separate
> > > > > > namespace instead of mixing the PMD-specific APIs and global APIs
> > > > > > up, and also save the bits in mbuf.ol_flags.
> > > > > >
> > > > > > There are other ways to achieve this goal, such as introducing
> > > > > > the PMD specific ol_flags in mbuf second cache line as you said.
> > > > > > I just thought defining some reserved bits seems to be the most
> > > > > > simple way which won't introduce many changes.
> > > > > >
> > > > > > What's your suggestions? Should I just revert the changes and
> > > > > > define the generic flags directly?
> > > > >
> > > > > Yes, that would be my preference.
> > > > > As I said above - spending extra bit in ol_flags  doesn't look like a big problem to me.
> > > > > In return there would be no need to consider how to handle all that confusing scenarios in future.
> > > >
> > > > Okay. I'll update my patches. Thanks a lot for your comments.
> > >
> > > Well, I do not agree with Konstantin (no one saw this coming eh?)
> >
> > :)
> >
> >  >and do not think you need to update your series again.
> > >
> > > PMD-specific symbols have nothing to do in the global namespace in my
> > > opinion, they are not versioned and may evolve without notice. Neither
> > > applications nor the bonding PMD can rely on them. That's the trade-off.
> >
> > Not sure I do understand your reasoning.
> > For me MACSEC offload is just one of many HW offloads that we support
> > and should be treated in exactly the same way.
> > Applications should be able to use it in a transparent and reliable way,
> > not only under some limited conditions.
> > Otherwise what is the point to introduce it at all?
> 
> Well my first reply to this thread was asking why isn't the whole API global
> from the start then?

That's good question, and my preference would always be to have the
API to configure this feature as generic one.
I guess the main reason why it is not right now we don't reach an agreement
how this API should look like: 
http://dpdk.org/ml/archives/dev/2016-September/047810.html
But I'll leave it to the author to provide the real reason here. 

> 
> Given there are valid reasons for it not to and no plan to make it so in the
> near future, applications must be aware that they are including
> rte_pmd_ixgbe.h to use it. That in itself is a limiting condition, right?

Yes, it is definitely a limiting factor.
Though even if API to configure device to use macsec would be PMD specific right now,
The API to query that capability and the API to use it at datapath (mbuf.ol_flags) still
can be (and I think should be) device independent and transparent to use.  

> 
> > Yes, right now it is supported only by ixgbe PMD, but why that should be the
> > reason to treat is as second-class citizen?
> > Let say PKT_TX_TUNNEL_* offloads also are supported only by one PMD right now.
> 
> You are right about PKT_TX_TUNNEL_*, however these flags exist on their own
> and are not tied to any API function calls, unlike in this series where
> PKT_TX_MACSEC can only be used if the DEV_TX_OFFLOAD_MACSEC_INSERT
> capability is present 

I don't think PKT_TX_TUNNEL_* 'exists on its own'.
To use it well behaving app have to:
1) Query that device does provide that capability: DEV_TX_OFFLOAD_*_TNL_TSO
2) configure PMD( & device) to use that capability
3) use that offload at run-time TX code (mb->ol_flags |= ...; mb->tx_offload = ...)

For PKT_TX_TUNNEL_*  2) is pretty simple - user just need to make sure
that full-featured TX function will be selected:
txconf.txq_flags = 0; ...;  rte_eth_tx_queue_setup(..., &txconf);
 
For TX_MACSEC, as I understand 2) will be more complicated and
right now is PMD specific, but anyway the main pattern remains the same.
So at least 1) and 3) could be kept device neutral.

>and the whole thing was configured through
> rte_pmd_ixgbe_macsec_*() calls after including rte_pmd_ixgbe.h.
> 
> To be clear it is not about MACsec per se (as a standardized protocol, I
> think related definitions for offloads have their place), but it has to do
> with the fact that the rest of the API is PMD-specific and there is a
> dependency between them.
> 
> > > Therefore until APIs are made global, the safe compromise is to define
> > > neutral, reserved symbols that any PMD can use to implement their own
> > > temporary APIs for testing purposes. These can be renamed later without
> > > changing their value as long as a single PMD uses them.
> >
> > Ok, so what we'll gain by introducing PKT_TX_RESERVED instead of PKT_TX_MACSEC?
> > As I said in my previous mail the redefinition for the same ol_flag bit (and dev capabilities)
> > by different PMD might create a lot of confusion in future.
> > Does the potential saving of 1 bit really worth it?
> 
> That is one benefit, but my point is mainly to keep applications aware that
> they are using an API defined by a single PMD, which may be temporary and
> whose symbols are not versioned.

As applications have to use PMD specific functions to configure it they definitely are aware.

> 
> Consider this:
> 
> rte_mbuf.h:
> 
>  #define PKT_TX_RESERVED_0 (1 << 42)
> 
> rte_pmd_ixgbe.h:
> 
>  #define PKT_TX_MACSEC PKT_TX_RESERVED_0
> 
> That way, applications have to get the PKT_TX_MACSEC definition where the
> rest of the API is also defined.
> 
> Other PMDs may reuse PKT_TX_RESERVED_0 and other reserved flags to implement
> their own experimental APIs.

That's the main thing I am opposed to.
I think that by allowing PMD to redefine meaning of
mbuf.ol_flags and dev_info.(rx| tx)_offload_capa 
we just asking for trouble.

Let say tomorrow, i40e will redefine DEV_TX_OFFLOAD_RESERVED_0 and PKT_TX_RESERVED_0
to implement new specific TX offload (PKT_TX_FEATURE_X).
Now let say we have an application that works over both ixgbe and i40e
and would like to use both TX_MACSEC and TC_FEATURE_X offloads whenever they are available.
As I can see, with the approach you proposed the only way for the application to make it
is to support 2 different TX code paths (or at least some parts of it).
To me that way looks inconvenient to the users and source of future troubles.

Same for RX:  somewhere at upper layer user got a packet with PKT_RX_RESERVED_0 set.
What does it really mean if there are different NIC types in the system?

> 
> Applications and the bonding PMD can easily be made aware that such reserved
> flags cannot be shared between ports unless they know what the underlying
> PMD is, which is already a requirement to use this API in the first place
> (for instance, calling rte_pmd_ixgbe_macsec_*() functions with another
> vendor's port_id may crash the application).

I am talking about that code:
rte_eth_bond_create(const char *name, uint8_t mode, uint8_t socket_id)
{
...
/* Take the first dev's offload capabilities */
internals->rx_offload_capa = dev_info.rx_offload_capa;
internals->tx_offload_capa = dev_info.tx_offload_capa;
...
internals->rx_offload_capa &= dev_info.rx_offload_capa;
internals->tx_offload_capa &= dev_info.tx_offload_capa;

Obviously with what you are suggesting it is not valid any more.
Bonded device need to support a MASK of all device reserved offloads to exclude
them from common subset. 
Any user app(/lib) that does similar thing would also have to be changed.

> 
> So the idea if/when the API is made global is to rename PKT_TX_RESERVED_0 to
> PKT_TX_MACSEC and keep its original value.
> 
> If other PMDs also implemented PKT_TX_RESERVED_0 in the meantime, it is
> redefined using a different value. If there is no room left to do so, these
> PMDs are out of luck I guess, and their specific API is disabled/removed
> until something gets re-designed.
> 
> How about this?

I still think that we shouldn't allow PMDs to redefine mbuf.olflags and
dev_info.(rx|tx)_offload_capa. 
See above for my reasons.
Konstantin