[dpdk-dev] [PATCH v3 7/7] app/testpmd: adjust ethdev port ownership

[Matan Azrad]

Hi Gaetan

From: Gaëtan Rivet, Friday, January 19, 2018 5:00 PM
> Hi Matan,
> 
> On Fri, Jan 19, 2018 at 01:35:10PM +0000, Matan Azrad wrote:
> > Hi Konstantin
> >
> > From: Ananyev, Konstantin, Friday, January 19, 2018 3:09 PM
> > > > -----Original Message-----
> > > > From: Matan Azrad [mailto:matan at mellanox.com]
> > > > Sent: Friday, January 19, 2018 12:52 PM
> > > > To: Ananyev, Konstantin <konstantin.ananyev at intel.com>; Thomas
> > > > Monjalon <thomas at monjalon.net>; Gaetan Rivet
> > > <gaetan.rivet at 6wind.com>;
> > > > Wu, Jingjing <jingjing.wu at intel.com>
> > > > Cc: dev at dpdk.org; Neil Horman <nhorman at tuxdriver.com>;
> Richardson,
> > > > Bruce <bruce.richardson at intel.com>
> > > > Subject: RE: [PATCH v3 7/7] app/testpmd: adjust ethdev port
> > > > ownership
> > > >
> > > > Hi Konstantin
> > > >
> > > > From: Ananyev, Konstantin, Friday, January 19, 2018 2:38 PM
> > > > > To: Matan Azrad <matan at mellanox.com>; Thomas Monjalon
> > > > > <thomas at monjalon.net>; Gaetan Rivet <gaetan.rivet at 6wind.com>;
> > > Wu,
> > > > > Jingjing <jingjing.wu at intel.com>
> > > > > Cc: dev at dpdk.org; Neil Horman <nhorman at tuxdriver.com>;
> > > > > Richardson, Bruce <bruce.richardson at intel.com>
> > > > > Subject: RE: [PATCH v3 7/7] app/testpmd: adjust ethdev port
> > > > > ownership
> > > > >
> > > > > Hi Matan,
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Matan Azrad [mailto:matan at mellanox.com]
> > > > > > Sent: Thursday, January 18, 2018 4:35 PM
> > > > > > To: Thomas Monjalon <thomas at monjalon.net>; Gaetan Rivet
> > > > > > <gaetan.rivet at 6wind.com>; Wu, Jingjing <jingjing.wu at intel.com>
> > > > > > Cc: dev at dpdk.org; Neil Horman <nhorman at tuxdriver.com>;
> > > Richardson,
> > > > > > Bruce <bruce.richardson at intel.com>; Ananyev, Konstantin
> > > > > > <konstantin.ananyev at intel.com>
> > > > > > Subject: [PATCH v3 7/7] app/testpmd: adjust ethdev port
> > > > > > ownership
> > > > > >
> > > > > > Testpmd should not use ethdev ports which are managed by other
> > > > > > DPDK entities.
> > > > > >
> > > > > > Set Testpmd ownership to each port which is not used by other
> > > > > > entity and prevent any usage of ethdev ports which are not
> > > > > > owned by
> > > Testpmd.
> > > > > >
> > > > > > Signed-off-by: Matan Azrad <matan at mellanox.com>
> > > > > > ---
> > > > > >  app/test-pmd/cmdline.c      | 89 +++++++++++++++++++-------------
> ----
> > > ----
> > > > > -----
> > > > > >  app/test-pmd/cmdline_flow.c |  2 +-
> > > > > >  app/test-pmd/config.c       | 37 ++++++++++---------
> > > > > >  app/test-pmd/parameters.c   |  4 +-
> > > > > >  app/test-pmd/testpmd.c      | 63 ++++++++++++++++++++----------
> --
> > > > > >  app/test-pmd/testpmd.h      |  3 ++
> > > > > >  6 files changed, 103 insertions(+), 95 deletions(-)
> > > > > >
> > > > > > diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
> > > > > > index
> > > > > > 31919ba..6199c64 100644
> > > > > > --- a/app/test-pmd/cmdline.c
> > > > > > +++ b/app/test-pmd/cmdline.c
> > > > > > @@ -1394,7 +1394,7 @@ struct cmd_config_speed_all {
> > > > > >  			&link_speed) < 0)
> > > > > >  		return;
> > > > > >
> > > > > > -	RTE_ETH_FOREACH_DEV(pid) {
> > > > > > +	RTE_ETH_FOREACH_DEV_OWNED_BY(pid, my_owner.id) {
> > > > >
> > > > > Why do we need all these changes?
> > > > > As I understand you changed definition of RTE_ETH_FOREACH_DEV(),
> > > > > so no testpmd should work ok default (no_owner case).
> > > > > Am I missing something here?
> > > >
> > > > Now, After Gaetan suggestion RTE_ETH_FOREACH_DEV(pid) will iterate
> > > over all valid and ownerless ports.
> > >
> > > Yes.
> > >
> > > > Here Testpmd wants to iterate over its owned ports.
> > >
> > > Why? Why it can't just iterate over all valid and ownerless ports?
> > > As I understand it would be enough to fix current problems and would
> > > allow us to avoid any changes in testmpd (which I think is a good thing).
> >
> > Yes, I understand that this big change is very daunted, But I think the
> current a lot of bugs in testpmd(regarding port ownership) even more
> daunted.
> >
> > Look,
> > Testpmd initiates some of its internal databases depends on specific
> > port iteration, In some time someone may take ownership of Testpmd
> ports and testpmd will continue to touch them.
> >
> 
> If I look back on the fail-safe, its sole purpose is to have seamless hotplug
> with existing applications.
> 

Yes.

> Port ownership is a genericization of some functions introduced by the fail-
> safe, that could structure DPDK further.

Not only.
Port ownership is a new concept saying that not all the ports are only for the application
and defines well the new port usage synchronization rules.

It can be a solution for failsafe scenario, but it solves a big generic problem regardless fail-safe.  

> It should allow applications to have a seamless integration with subsystems using port ownership. Without this, port ownership cannot be used.

I do not think it is accurate.
We can use different solution to solve the fail-safe case (seamless) by using the DEFFERED state as you did.
Port ownership is not only for failsafe case - it is a generic new concept which BTW can fix the fail-safe case(full fix).
So, application should use port ownership regardless the failsafe using, just to be sure no one touch its ports.

> Testpmd should be fixed, but follow the most common design patterns of
> DPDK applications. Going with port ownership seems like a paradigm shift.

I think this patch is a classic fix for it and for the full generic issue.
Do you have simpler fix to the races?

> > In addition
> > Using the old iterator in some places in testpmd will cause a race for run-
> time new ports(can be created by failsafe or any hotplug code):
> > - testpmd finds an ownerless port(just now created) by the old
> > iterator and start traffic there,
> > - failsafe takes ownership of this new port and start traffic there.
> > Problem!
> 
> Testpmd does not handle detection of new port. If it did, testing fail-safe
> with it would be wrong.

It used the old iterator everywhere.
So it see(and uses) all the valid ports all the time.
As the new concept - it should be changed to use its owned ports,
It is a simple classic solution just to use the new iterator.

> At startup, RTE_ETH_FOREACH_DEV already fixed the issue of registering
> DEFERRED ports. There are still remaining issues regarding this, but I think
> they should be fixed. The architecture does not need to be completely
> moved to port ownership.

I think port ownership fixes the issues nicely, don't you think so?

> If anything, this should serve as a test for your API with common applications.
> I think you'd prefer to know and debug with testpmd instead of firing up VPP
> or something like that to determine what went wrong with using the fail-
> safe.
> >

Yes as your examples in docs.

> > In addition
> > As a good example for well-done application (free from ownership bugs) I
> tried here to adjust Tespmd to the new rules and BTW to fix a lot of bugs.
> 
> Testpmd has too much cruft, it won't ever be a good example of a well-done
> application.
> 
> If you want to demonstrate ownership, I think you should start an example
> application demonstrating race conditions and their mitigation.
> 
> I think that would be interesting for many DPDK users.

I will think about that regardless the testpmd adjustment (need to find time :))

> >
> > So actually applications which are not aware to the port ownership still are
> exposed to races, but if there use the old iterator(with the new change) the
> amount of races decreases.
> >
> > Thanks, Matan.
> > > Konstantin
> > >
> > > >
> > > > I added to Testpmd ability to take an ownership of ports as the
> > > > new ownership and synchronization rules suggested, Since Tespmd is
> > > > a DPDK entity which wants that no one will touch its owned ports,
> > > > It must allocate
> > > an unique ID, set owner for its ports (see in main function) and
> > > recognizes them by its owner ID.
> > > >
> > > > > Konstantin
> > > > >
> 
> Regards,
> --
> Gaëtan Rivet
> 6WIND