[dpdk-dev] [PATCH v3 7/7] app/testpmd: adjust ethdev port ownership

Matan Azrad matan at mellanox.com
Tue Jan 23 15:43:49 CET 2018


Hi Konstantin

Please move the second thread, I'm feeling you and Gaetan have the same questions.

From: Ananyev, Konstantin, Tuesday, January 23, 2018 3:35 PM
> Hi Matan,
> 
> >
> >
> > Hi Konstantin
> > From: Ananyev, Konstantin, Monday, January 22, 2018 10:49 PM
> > > Hi Matan,
> > >
> > > > -----Original Message-----
> > > > From: Matan Azrad [mailto:matan at mellanox.com]
> > > > Sent: Monday, January 22, 2018 1:23 PM
> > > > To: Ananyev, Konstantin <konstantin.ananyev at intel.com>; Gaëtan
> > > > Rivet <gaetan.rivet at 6wind.com>
> > > > Cc: Thomas Monjalon <thomas at monjalon.net>; Wu, Jingjing
> > > > <jingjing.wu at intel.com>; dev at dpdk.org; Neil Horman
> > > > <nhorman at tuxdriver.com>; Richardson, Bruce
> > > > <bruce.richardson at intel.com>
> > > > Subject: RE: [PATCH v3 7/7] app/testpmd: adjust ethdev port
> > > > ownership
> > > >
> > > >
> > > > Hi
> > > > From: Ananyev, Konstantin [mailto:konstantin.ananyev at intel.com]
> > > > > Hi lads,
> > > > >
> > > > > >
> > > > > > Hi Matan,
> > > > > >
> > > > > > On Fri, Jan 19, 2018 at 01:35:10PM +0000, Matan Azrad wrote:
> > > > > > > Hi Konstantin
> > > > > > >
> > > > > > > From: Ananyev, Konstantin, Friday, January 19, 2018 3:09 PM
> > > > > > > > > -----Original Message-----
> > > > > > > > > From: Matan Azrad [mailto:matan at mellanox.com]
> > > > > > > > > Sent: Friday, January 19, 2018 12:52 PM
> > > > > > > > > To: Ananyev, Konstantin <konstantin.ananyev at intel.com>;
> > > > > > > > > Thomas Monjalon <thomas at monjalon.net>; Gaetan Rivet
> > > > > > > > <gaetan.rivet at 6wind.com>;
> > > > > > > > > Wu, Jingjing <jingjing.wu at intel.com>
> > > > > > > > > Cc: dev at dpdk.org; Neil Horman <nhorman at tuxdriver.com>;
> > > > > > > > > Richardson, Bruce <bruce.richardson at intel.com>
> > > > > > > > > Subject: RE: [PATCH v3 7/7] app/testpmd: adjust ethdev
> > > > > > > > > port ownership
> > > > > > > > >
> > > > > > > > > Hi Konstantin
> > > > > > > > >
> > > > > > > > > From: Ananyev, Konstantin, Friday, January 19, 2018 2:38
> > > > > > > > > PM
> > > > > > > > > > To: Matan Azrad <matan at mellanox.com>; Thomas
> Monjalon
> > > > > > > > > > <thomas at monjalon.net>; Gaetan Rivet
> > > > > <gaetan.rivet at 6wind.com>;
> > > > > > > > Wu,
> > > > > > > > > > Jingjing <jingjing.wu at intel.com>
> > > > > > > > > > Cc: dev at dpdk.org; Neil Horman
> <nhorman at tuxdriver.com>;
> > > > > > > > > > Richardson, Bruce <bruce.richardson at intel.com>
> > > > > > > > > > Subject: RE: [PATCH v3 7/7] app/testpmd: adjust ethdev
> > > > > > > > > > port ownership
> > > > > > > > > >
> > > > > > > > > > Hi Matan,
> > > > > > > > > >
> > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > From: Matan Azrad [mailto:matan at mellanox.com]
> > > > > > > > > > > Sent: Thursday, January 18, 2018 4:35 PM
> > > > > > > > > > > To: Thomas Monjalon <thomas at monjalon.net>; Gaetan
> > > > > > > > > > > Rivet <gaetan.rivet at 6wind.com>; Wu, Jingjing
> > > > > > > > > > > <jingjing.wu at intel.com>
> > > > > > > > > > > Cc: dev at dpdk.org; Neil Horman
> > > > > > > > > > > <nhorman at tuxdriver.com>;
> > > > > > > > Richardson,
> > > > > > > > > > > Bruce <bruce.richardson at intel.com>; Ananyev,
> > > > > > > > > > > Konstantin <konstantin.ananyev at intel.com>
> > > > > > > > > > > Subject: [PATCH v3 7/7] app/testpmd: adjust ethdev
> > > > > > > > > > > port ownership
> > > > > > > > > > >
> > > > > > > > > > > Testpmd should not use ethdev ports which are
> > > > > > > > > > > managed by other DPDK entities.
> > > > > > > > > > >
> > > > > > > > > > > Set Testpmd ownership to each port which is not used
> > > > > > > > > > > by other entity and prevent any usage of ethdev
> > > > > > > > > > > ports which are not owned by
> > > > > > > > Testpmd.
> > > > > > > > > > >
> > > > > > > > > > > Signed-off-by: Matan Azrad <matan at mellanox.com>
> > > > > > > > > > > ---
> > > > > > > > > > >  app/test-pmd/cmdline.c      | 89 +++++++++++++++++++--
> ----
> > > ----
> > > > > -------
> > > > > > > > ----
> > > > > > > > > > -----
> > > > > > > > > > >  app/test-pmd/cmdline_flow.c |  2 +-
> > > > > > > > > > >  app/test-pmd/config.c       | 37 ++++++++++---------
> > > > > > > > > > >  app/test-pmd/parameters.c   |  4 +-
> > > > > > > > > > >  app/test-pmd/testpmd.c      | 63
> ++++++++++++++++++++----
> > > ----
> > > > > ----
> > > > > > > > > > >  app/test-pmd/testpmd.h      |  3 ++
> > > > > > > > > > >  6 files changed, 103 insertions(+), 95 deletions(-)
> > > > > > > > > > >
> > > > > > > > > > > diff --git a/app/test-pmd/cmdline.c
> > > > > > > > > > > b/app/test-pmd/cmdline.c index
> > > > > > > > > > > 31919ba..6199c64 100644
> > > > > > > > > > > --- a/app/test-pmd/cmdline.c
> > > > > > > > > > > +++ b/app/test-pmd/cmdline.c
> > > > > > > > > > > @@ -1394,7 +1394,7 @@ struct cmd_config_speed_all {
> > > > > > > > > > >  			&link_speed) < 0)
> > > > > > > > > > >  		return;
> > > > > > > > > > >
> > > > > > > > > > > -	RTE_ETH_FOREACH_DEV(pid) {
> > > > > > > > > > > +	RTE_ETH_FOREACH_DEV_OWNED_BY(pid,
> > > my_owner.id) {
> > > > > > > > > >
> > > > > > > > > > Why do we need all these changes?
> > > > > > > > > > As I understand you changed definition of
> > > > > > > > > > RTE_ETH_FOREACH_DEV(), so no testpmd should work ok
> > > > > > > > > > default
> > > > > (no_owner case).
> > > > > > > > > > Am I missing something here?
> > > > > > > > >
> > > > > > > > > Now, After Gaetan suggestion RTE_ETH_FOREACH_DEV(pid)
> > > > > > > > > will iterate
> > > > > > > > over all valid and ownerless ports.
> > > > > > > >
> > > > > > > > Yes.
> > > > > > > >
> > > > > > > > > Here Testpmd wants to iterate over its owned ports.
> > > > > > > >
> > > > > > > > Why? Why it can't just iterate over all valid and ownerless ports?
> > > > > > > > As I understand it would be enough to fix current problems
> > > > > > > > and would allow us to avoid any changes in testmpd (which
> > > > > > > > I think is a good
> > > > > thing).
> > > > > > >
> > > > > > > Yes, I understand that this big change is very daunted, But
> > > > > > > I think the current a lot of bugs in testpmd(regarding port
> > > > > > > ownership) even more
> > > > > > daunted.
> > > > > > >
> > > > > > > Look,
> > > > > > > Testpmd initiates some of its internal databases depends on
> > > > > > > specific port iteration, In some time someone may take
> > > > > > > ownership of Testpmd
> > > > > ports and testpmd will continue to touch them.
> > > > >
> > > > > But if someone will take the ownership (assign new owner_id)
> > > > > that port will not appear in RTE_ETH_FOREACH_DEV() any more.
> > > > >
> > > >
> > > > Yes, but testpmd sometimes depends on previous iteration using
> > > > internal
> > > database.
> > > > So it uses internal database that was updated by old iteration.
> > >
> > > That sounds like just a bug in testpmd that need to be fixed, no?
> >
> > If Testpmd already took ownership for these ports(like I did), it is ok.
> >
> 
> Hmm, why not just to fix testpmd, if there is a bug?
> As I said all control ops here are done by one thread, so it should be pretty
> easy.
> Or are you talking about race conditions?
> 
> > > Any particular places where outdated device info is used?
> >
> > For example, look for the stream management in testpmd(I think I saw it
> there).
> 
> Anything particular?
> 
> >
> > > > > > If I look back on the fail-safe, its sole purpose is to have
> > > > > > seamless hotplug with existing applications.
> > > > > >
> > > > > > Port ownership is a genericization of some functions
> > > > > > introduced by the fail-safe, that could structure DPDK
> > > > > > further. It should allow applications to have a seamless
> > > > > > integration with subsystems using port ownership. Without this,
> port ownership cannot be used.
> > > > > >
> > > > > > Testpmd should be fixed, but follow the most common design
> > > > > > patterns of DPDK applications. Going with port ownership seems
> > > > > > like a paradigm shift.
> > > > > >
> > > > > > > In addition
> > > > > > > Using the old iterator in some places in testpmd will cause
> > > > > > > a race for run-
> > > > > time new ports(can be created by failsafe or any hotplug code):
> > > > > > > - testpmd finds an ownerless port(just now created) by the
> > > > > > > old iterator and start traffic there,
> > > > > > > - failsafe takes ownership of this new port and start traffic there.
> > > > > > > Problem!
> > > > >
> > > > > Could you shed a bit more light here - it would be race
> > > > > condition between whom and whom?
> > > >
> > > > Sure.
> > > >
> > > > > As I remember in testpmd all control ops are done within one
> > > > > thread (main lcore).
> > > >
> > > > But other dpdk entity can use another thread, for example:
> > > > Failsafe uses the host thread(using alarm callback) to create a
> > > > new port and
> > > to take ownership of a port.
> > >
> > > Hm, and you create new ports inside failsafe PMD, right and then set
> > > new owner_id for it?
> >
> > Yes.
> >
> > > And all this in alarm in interrupt thread?
> >
> > Yes.
> >
> > > If so I wonder how you can guarantee that no-one else will set
> > > different owner_id between
> > > rte_eth_dev_allocate() and rte_eth_dev_owner_set()?
> >
> > I check it (see failsafe patch to this series - V5).
> > Function: fs_bus_init.
> 
> You are talking about that peace of code:
> +		ret = rte_eth_dev_owner_set(pid, &PRIV(dev)-
> >my_owner);
> +		if (ret) {
> +			INFO("sub_device %d owner set failed (%s),"
> +			     " will try again later", i, strerror(ret));
> +			continue;
> 
> right?
> So you just wouldn't include that device into your failsafe device.
> But that probably not what user wanted, especially if he bothered to create a
> special new low-level device for you.
> If that' s the use case, then I think you need to set device ownership at
> creation time - inside dev_allocate().
> Again that would avoid such racing conditions inside testpmd.
> 
> >
> > > Could you point me to that place (I am not really familiar with
> > > familiar with failsafe code)?
> > >
> > > >
> > > > The race:
> > > > Testpmd iterates over all ports by the master thread.
> > > > Failsafe takes ownership of a port by the host thread and start using it.
> > > > => The two dpdk entities may use the device at same time!
> > >
> > > Ok, if failsafe really assigns its owner_id(s) to ports that are
> > > already in use by the app, then how such scheme supposed to work at
> all?
> >
> > If the app works well (with the new rules) it already took ownership and
> failsafe will see it and will wait until the application release it.
> 
> Ok, and why application would need to release it?
> How it would know that failsafe device wants to use it now?
> Again where is a guarantee that after app released it some other entity
> wouldn't grab it for itself?
> 
> > Every dpdk entity should know which port it wants to manage, If 2
> > entities want to manage the same device -  it can be ok and port ownership
> can synchronize the usage.
> >
> > Probably, application which will run fail-safe wants to manage only the fail-
> safe port and therefor to take ownership only for it.
> >
> > > I.E. application has a port - it assigns some owner_id != 0 to it,
> > > then PMD tries to set its owner_id tot the same port.
> > > Obviously failsafe's set_owner() will always fail in such case.
> > >
> > Yes, and will try again after some time.
> 
> Same question again - how app will know that it has to release the port
> ownership?
> 
> >
> > > From what I hear we need to introduce a concept of 'default owner id'.
> > > I.E. when failsafe PMD is created - user assigns some owner_id to it
> (default).
> > > Then failsafe PMD generates it's own owner_id and assigns it only to
> > > the ports whose current owner_id is equal either 0 or 'default' owner_id.
> > >
> >
> > It is a suggestion and we need to think about it more (I'm talking about it
> with Gaetan in another thread).
> > Actually I think, if we want a generic solution to the generic problem the
> current solution is ok.
> 
> From what I heard - every app that wants to use failsafe PMD would require
> quite a lot of changes.
> It doesn't look ok to me.
> 
> >
> > > >
> > > > Obeying the new ownership rules can prevent all these races.
> > > >
> > >
> > > When we discussed RFC of owner_id patch, I thought we all agreed
> > > that owner_id  API shouldn't be mandatory - i.e. existing apps not
> > > required to change to work normally with that.
> >
> > Yes, it is not mandatory if app doesn't use hotplug.
> >
> > I think with hotplug it is mandatory in the most cases.
> 
> Yes in failsafe you always install this alarm handler, so even if the app would
> have its own  way to handle hotplug  devices - it would suddenly need to use
> this new owner API - even if it doesn't need to.
> Why it has to be?
> 
> >
> > And it can ease the secondary process model too.
> >
> > Again, in the generic ownership problem as discussed in RFC:
> > Every entity, include app, should know which ports it wants to manage and
> to take ownership only for them.
> >
> > > Though right now it seems that application changes seems necessary,
> > > at least to work ok with failsafe PMD.
> >
> > And for solving the generic problem of ownership.(will defend from future
> issues by sure).
> >
> > > Which makes we wonder was it some sort of misunderstanding or we did
> > > we do something wrong here?
> >
> > Mistakes can be done all the time, but I think we are all understand the big
> issue of ownership and how the current solution solves it.
> > fail-safe it is just a current example for the problems which are possible
> because of the generic ownership issue.
> 
> Honestly that seems too much changes for the app just to make failsafe PMD
> work correctly.
> IMO - It should be some way to support it without causing changes in each
> DPDK application  - otherwise something is wrong with the PMD itself.
> If let say that ownership model is required to make failsafe PMD to operate -
> it should be done in a transparent way to the user.
> Probably something like Gaetan suggested in another mail or so.
> Konstantin
> 
> >
> > Thanks,
> > Matan
> > > Konstantin
> > >
> > > > > The only way to attach/detach port with it - invoke testpmd CLI
> > > > > "attach/detach" port.
> > > > >
> > > > > Konstantin
> > > > >
> > > > > >
> > > > > > Testpmd does not handle detection of new port. If it did,
> > > > > > testing fail-safe with it would be wrong.
> > > > > >
> > > > > > At startup, RTE_ETH_FOREACH_DEV already fixed the issue of
> > > > > > registering DEFERRED ports. There are still remaining issues
> > > > > > regarding this, but I think they should be fixed. The
> > > > > > architecture does not need to be completely moved to port
> ownership.
> > > > > >
> > > > > > If anything, this should serve as a test for your API with
> > > > > > common applications. I think you'd prefer to know and debug
> > > > > > with testpmd instead of firing up VPP or something like that
> > > > > > to determine what went wrong with using the fail-safe.
> > > > > >
> > > > > > >
> > > > > > > In addition
> > > > > > > As a good example for well-done application (free from
> > > > > > > ownership
> > > > > > > bugs) I tried here to adjust Tespmd to the new rules and BTW
> > > > > > > to fix a
> > > > > > lot of bugs.
> > > > > >
> > > > > > Testpmd has too much cruft, it won't ever be a good example of
> > > > > > a well-done application.
> > > > > >
> > > > > > If you want to demonstrate ownership, I think you should start
> > > > > > an example application demonstrating race conditions and their
> mitigation.
> > > > > >
> > > > > > I think that would be interesting for many DPDK users.
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > > So actually applications which are not aware to the port
> > > > > > > ownership still are exposed to races, but if there use the
> > > > > > > old iterator(with the new
> > > > > > change) the amount of races decreases.
> > > > > > >
> > > > > > > Thanks, Matan.
> > > > > > > > Konstantin
> > > > > > > >
> > > > > > > > >
> > > > > > > > > I added to Testpmd ability to take an ownership of ports
> > > > > > > > > as the new ownership and synchronization rules
> > > > > > > > > suggested, Since Tespmd is a DPDK entity which wants
> > > > > > > > > that no one will touch its owned ports, It must allocate
> > > > > > > > an unique ID, set owner for its ports (see in main
> > > > > > > > function) and recognizes them by its owner ID.
> > > > > > > > >
> > > > > > > > > > Konstantin
> > > > > > > > > >
> > > > > >
> > > > > > Regards,
> > > > > > --
> > > > > > Gaëtan Rivet
> > > > > > 6WIND



More information about the dev mailing list