[dpdk-dev] [PATCH v3 7/7] app/testpmd: adjust ethdev port ownership

Ananyev, Konstantin konstantin.ananyev at intel.com
Mon Jan 22 21:48:37 CET 2018


Hi Matan,

> -----Original Message-----
> From: Matan Azrad [mailto:matan at mellanox.com]
> Sent: Monday, January 22, 2018 1:23 PM
> To: Ananyev, Konstantin <konstantin.ananyev at intel.com>; Gaëtan Rivet <gaetan.rivet at 6wind.com>
> Cc: Thomas Monjalon <thomas at monjalon.net>; Wu, Jingjing <jingjing.wu at intel.com>; dev at dpdk.org; Neil Horman
> <nhorman at tuxdriver.com>; Richardson, Bruce <bruce.richardson at intel.com>
> Subject: RE: [PATCH v3 7/7] app/testpmd: adjust ethdev port ownership
> 
> 
> Hi
> From: Ananyev, Konstantin [mailto:konstantin.ananyev at intel.com]
> > Hi lads,
> >
> > >
> > > Hi Matan,
> > >
> > > On Fri, Jan 19, 2018 at 01:35:10PM +0000, Matan Azrad wrote:
> > > > Hi Konstantin
> > > >
> > > > From: Ananyev, Konstantin, Friday, January 19, 2018 3:09 PM
> > > > > > -----Original Message-----
> > > > > > From: Matan Azrad [mailto:matan at mellanox.com]
> > > > > > Sent: Friday, January 19, 2018 12:52 PM
> > > > > > To: Ananyev, Konstantin <konstantin.ananyev at intel.com>; Thomas
> > > > > > Monjalon <thomas at monjalon.net>; Gaetan Rivet
> > > > > <gaetan.rivet at 6wind.com>;
> > > > > > Wu, Jingjing <jingjing.wu at intel.com>
> > > > > > Cc: dev at dpdk.org; Neil Horman <nhorman at tuxdriver.com>;
> > > > > > Richardson, Bruce <bruce.richardson at intel.com>
> > > > > > Subject: RE: [PATCH v3 7/7] app/testpmd: adjust ethdev port
> > > > > > ownership
> > > > > >
> > > > > > Hi Konstantin
> > > > > >
> > > > > > From: Ananyev, Konstantin, Friday, January 19, 2018 2:38 PM
> > > > > > > To: Matan Azrad <matan at mellanox.com>; Thomas Monjalon
> > > > > > > <thomas at monjalon.net>; Gaetan Rivet
> > <gaetan.rivet at 6wind.com>;
> > > > > Wu,
> > > > > > > Jingjing <jingjing.wu at intel.com>
> > > > > > > Cc: dev at dpdk.org; Neil Horman <nhorman at tuxdriver.com>;
> > > > > > > Richardson, Bruce <bruce.richardson at intel.com>
> > > > > > > Subject: RE: [PATCH v3 7/7] app/testpmd: adjust ethdev port
> > > > > > > ownership
> > > > > > >
> > > > > > > Hi Matan,
> > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Matan Azrad [mailto:matan at mellanox.com]
> > > > > > > > Sent: Thursday, January 18, 2018 4:35 PM
> > > > > > > > To: Thomas Monjalon <thomas at monjalon.net>; Gaetan Rivet
> > > > > > > > <gaetan.rivet at 6wind.com>; Wu, Jingjing
> > > > > > > > <jingjing.wu at intel.com>
> > > > > > > > Cc: dev at dpdk.org; Neil Horman <nhorman at tuxdriver.com>;
> > > > > Richardson,
> > > > > > > > Bruce <bruce.richardson at intel.com>; Ananyev, Konstantin
> > > > > > > > <konstantin.ananyev at intel.com>
> > > > > > > > Subject: [PATCH v3 7/7] app/testpmd: adjust ethdev port
> > > > > > > > ownership
> > > > > > > >
> > > > > > > > Testpmd should not use ethdev ports which are managed by
> > > > > > > > other DPDK entities.
> > > > > > > >
> > > > > > > > Set Testpmd ownership to each port which is not used by
> > > > > > > > other entity and prevent any usage of ethdev ports which are
> > > > > > > > not owned by
> > > > > Testpmd.
> > > > > > > >
> > > > > > > > Signed-off-by: Matan Azrad <matan at mellanox.com>
> > > > > > > > ---
> > > > > > > >  app/test-pmd/cmdline.c      | 89 +++++++++++++++++++----------
> > -------
> > > > > ----
> > > > > > > -----
> > > > > > > >  app/test-pmd/cmdline_flow.c |  2 +-
> > > > > > > >  app/test-pmd/config.c       | 37 ++++++++++---------
> > > > > > > >  app/test-pmd/parameters.c   |  4 +-
> > > > > > > >  app/test-pmd/testpmd.c      | 63 ++++++++++++++++++++--------
> > ----
> > > > > > > >  app/test-pmd/testpmd.h      |  3 ++
> > > > > > > >  6 files changed, 103 insertions(+), 95 deletions(-)
> > > > > > > >
> > > > > > > > diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
> > > > > > > > index
> > > > > > > > 31919ba..6199c64 100644
> > > > > > > > --- a/app/test-pmd/cmdline.c
> > > > > > > > +++ b/app/test-pmd/cmdline.c
> > > > > > > > @@ -1394,7 +1394,7 @@ struct cmd_config_speed_all {
> > > > > > > >  			&link_speed) < 0)
> > > > > > > >  		return;
> > > > > > > >
> > > > > > > > -	RTE_ETH_FOREACH_DEV(pid) {
> > > > > > > > +	RTE_ETH_FOREACH_DEV_OWNED_BY(pid, my_owner.id) {
> > > > > > >
> > > > > > > Why do we need all these changes?
> > > > > > > As I understand you changed definition of
> > > > > > > RTE_ETH_FOREACH_DEV(), so no testpmd should work ok default
> > (no_owner case).
> > > > > > > Am I missing something here?
> > > > > >
> > > > > > Now, After Gaetan suggestion RTE_ETH_FOREACH_DEV(pid) will
> > > > > > iterate
> > > > > over all valid and ownerless ports.
> > > > >
> > > > > Yes.
> > > > >
> > > > > > Here Testpmd wants to iterate over its owned ports.
> > > > >
> > > > > Why? Why it can't just iterate over all valid and ownerless ports?
> > > > > As I understand it would be enough to fix current problems and
> > > > > would allow us to avoid any changes in testmpd (which I think is a good
> > thing).
> > > >
> > > > Yes, I understand that this big change is very daunted, But I think
> > > > the current a lot of bugs in testpmd(regarding port ownership) even
> > > > more
> > > daunted.
> > > >
> > > > Look,
> > > > Testpmd initiates some of its internal databases depends on specific
> > > > port iteration, In some time someone may take ownership of Testpmd
> > ports and testpmd will continue to touch them.
> >
> > But if someone will take the ownership (assign new owner_id) that port will
> > not appear in RTE_ETH_FOREACH_DEV() any more.
> >
> 
> Yes, but testpmd sometimes depends on previous iteration using internal database.
> So it uses internal database that was updated by old iteration.

That sounds like just a bug in testpmd that need to be fixed, no?
Any particular places where outdated device info is used? 

> 
> > > >
> > >
> > > If I look back on the fail-safe, its sole purpose is to have seamless
> > > hotplug with existing applications.
> > >
> > > Port ownership is a genericization of some functions introduced by the
> > > fail-safe, that could structure DPDK further. It should allow
> > > applications to have a seamless integration with subsystems using port
> > > ownership. Without this, port ownership cannot be used.
> > >
> > > Testpmd should be fixed, but follow the most common design patterns of
> > > DPDK applications. Going with port ownership seems like a paradigm
> > > shift.
> > >
> > > > In addition
> > > > Using the old iterator in some places in testpmd will cause a race for run-
> > time new ports(can be created by failsafe or any hotplug code):
> > > > - testpmd finds an ownerless port(just now created) by the old
> > > > iterator and start traffic there,
> > > > - failsafe takes ownership of this new port and start traffic there.
> > > > Problem!
> >
> > Could you shed a bit more light here - it would be race condition between
> > whom and whom?
> 
> Sure.
> 
> > As I remember in testpmd all control ops are done within one thread (main
> > lcore).
> 
> But other dpdk entity can use another thread, for example:
> Failsafe uses the host thread(using alarm callback) to create a new port and to take ownership of a port.

Hm, and you create new ports inside failsafe PMD, right and then set new owner_id for it?
And all this in alarm in interrupt thread?
If so I wonder how you can guarantee that no-one else will set different owner_id between
rte_eth_dev_allocate() and rte_eth_dev_owner_set()?
Could you point me to that place (I am not really familiar with familiar with failsafe code)?

> 
> The race:
> Testpmd iterates over all ports by the master thread.
> Failsafe takes ownership of a port by the host thread and start using it.
> => The two dpdk entities may use the device at same time!

Ok, if failsafe really assigns its owner_id(s) to ports that are already in use by the app,
then how such scheme supposed to work at all?
I.E. application has a port - it assigns some owner_id != 0 to it, then PMD tries to
set its owner_id tot the same port.
Obviously failsafe's set_owner() will always fail in such case.

>From what I hear we need to introduce a concept of 'default owner id'.
I.E. when failsafe PMD is created - user assigns some owner_id to it (default).
Then failsafe PMD generates it's own owner_id and assigns it only to the ports
whose current owner_id is equal either 0 or 'default' owner_id. 

> 
> Obeying the new ownership rules can prevent all these races.
> 

When we discussed RFC of owner_id patch, I thought we all agreed that
owner_id  API shouldn't be mandatory - i.e. existing apps not required to change
to work normally with that.
Though right now it seems that application changes seems necessary,
at least to work ok with failsafe PMD.
Which makes we wonder was it some sort of misunderstanding or
we did we do something wrong here?
Konstantin

> > The only way to attach/detach port with it - invoke testpmd CLI
> > "attach/detach" port.
> >
> > Konstantin
> >
> > >
> > > Testpmd does not handle detection of new port. If it did, testing
> > > fail-safe with it would be wrong.
> > >
> > > At startup, RTE_ETH_FOREACH_DEV already fixed the issue of registering
> > > DEFERRED ports. There are still remaining issues regarding this, but I
> > > think they should be fixed. The architecture does not need to be
> > > completely moved to port ownership.
> > >
> > > If anything, this should serve as a test for your API with common
> > > applications. I think you'd prefer to know and debug with testpmd
> > > instead of firing up VPP or something like that to determine what went
> > > wrong with using the fail-safe.
> > >
> > > >
> > > > In addition
> > > > As a good example for well-done application (free from ownership
> > > > bugs) I tried here to adjust Tespmd to the new rules and BTW to fix
> > > > a
> > > lot of bugs.
> > >
> > > Testpmd has too much cruft, it won't ever be a good example of a
> > > well-done application.
> > >
> > > If you want to demonstrate ownership, I think you should start an
> > > example application demonstrating race conditions and their mitigation.
> > >
> > > I think that would be interesting for many DPDK users.
> > >
> > > >
> > > >
> > > > So actually applications which are not aware to the port ownership
> > > > still are exposed to races, but if there use the old iterator(with
> > > > the new
> > > change) the amount of races decreases.
> > > >
> > > > Thanks, Matan.
> > > > > Konstantin
> > > > >
> > > > > >
> > > > > > I added to Testpmd ability to take an ownership of ports as the
> > > > > > new ownership and synchronization rules suggested, Since Tespmd
> > > > > > is a DPDK entity which wants that no one will touch its owned
> > > > > > ports, It must allocate
> > > > > an unique ID, set owner for its ports (see in main function) and
> > > > > recognizes them by its owner ID.
> > > > > >
> > > > > > > Konstantin
> > > > > > >
> > >
> > > Regards,
> > > --
> > > Gaëtan Rivet
> > > 6WIND



More information about the dev mailing list