[dpdk-ci] script to determine target repo (was DPDK Lab)

[Richardson, Bruce]

> -----Original Message-----
> From: Xu, Qian Q
> Sent: Friday, June 23, 2017 9:44 AM
> To: Thomas Monjalon <thomas at monjalon.net>; Wei, FangfangX
> <fangfangx.wei at intel.com>
> Cc: ci at dpdk.org; Richardson, Bruce <bruce.richardson at intel.com>;
> O'Driscoll, Tim <tim.odriscoll at intel.com>; Eugene Voronov
> <eugene at mellanox.com>
> Subject: RE: [dpdk-ci] script to determine target repo (was DPDK Lab)
> 
> Thomas/Bruce
> 1. For determining the repo tree to target, I don't believe that we can
> ever
> > come up with a 100% accurate rule, as the tree to which a set is to be
> > applied can be difficult to determine, so it may be done on the basis of
> on-list discussion.
> > A 90% accurate rule it what we may have to accept.
> 
> -- Then if we find the performance issue, then maybe it's a false alarm
> due to apply to the wrong repo. So, we may face many false alarms
> according with the time.
> Then people may not treat the performance issue as a problem, so I still
> think we need to try 100% accurate to have a more trustable result when we
> send out the alarm.

I find that rather improbable, and not worth considering. For that to per a problem multiple unlikely events have to occur:
1) we mis-identify the tree on which the set is to be applied (we should be able to get to 90% accuracy here)
2) the patchset must apply cleanly to the "wrong" tree (this is reasonably likely, but it's still another condition that has to be met for us to have a problem)
3) the patchset has to cause a performance regression in the "wrong" tree
4) but NOT cause a regression when in the right tree.

If we assume 90% accuracy of tree identification, optimistically that 90% of patches will apply to the wrong tree, that 5% of patches cause a performance regression (an overestimate IMHO), and that even 1/3 of those won't cause a performance regression in the right tree (a very overestimate IMHO, I would expect just about none of them to even have this), it still means that only about 1 patch in 1000 will show as a false positive performance regression.

0.1 (mis-identify) * 0.9 (applies ok) * 0.05 (regression) * 0.33 (no regression) = 0.0015, or 0.15%

So worst case, I still don't think we have a problem for the scenario you describe.

/Bruce

> 
> 
> > -----Original Message-----
> > From: Thomas Monjalon [mailto:thomas at monjalon.net]
> > Sent: Wednesday, June 21, 2017 4:33 PM
> > To: Wei, FangfangX <fangfangx.wei at intel.com>
> > Cc: ci at dpdk.org; Richardson, Bruce <bruce.richardson at intel.com>; Xu,
> > Qian Q <qian.q.xu at intel.com>; O'Driscoll, Tim
> > <tim.odriscoll at intel.com>; Eugene Voronov <eugene at mellanox.com>
> > Subject: Re: [dpdk-ci] script to determine target repo (was DPDK Lab)
> >
> > I agree with Bruce.
> >
> > Sorry for not having written the scripts yet.
> > Someone else in Mellanox should do it in July.
> > In the meantime, do not hesitate to share your code if it speed up
> things.
> >
> > Thanks a lot
> >
> >
> > 21/06/2017 10:20, Richardson, Bruce:
> > > Hi Fangfang,
> > >
> > > My comments on the questions you asked:
> > >
> > > 1. For determining the repo tree to target, I don't believe that we
> > > can ever
> > come up with a 100% accurate rule, as the tree to which a set is to be
> > applied can be difficult to determine, so it may be done on the basis of
> on-list discussion.
> > A 90% accurate rule it what we may have to accept. However, since
> > applying a patchset to a tree should not be a time-consuming
> > operation, I suggest any script produce a list of possible trees in
> > priority order to try. If not net, then try main, etc. etc.
> > >
> > > 2. Using the order from patch titles is correct.
> > >
> > > /Bruce