[dpdk-ci] script to determine target repo (was DPDK Lab)

[Xu, Qian Q]

> -----Original Message-----
> From: Richardson, Bruce
> Sent: Friday, June 23, 2017 5:30 PM
> To: Xu, Qian Q <qian.q.xu at intel.com>; Thomas Monjalon
> <thomas at monjalon.net>; Wei, FangfangX <fangfangx.wei at intel.com>
> Cc: ci at dpdk.org; O'Driscoll, Tim <tim.odriscoll at intel.com>; Eugene Voronov
> <eugene at mellanox.com>
> Subject: RE: [dpdk-ci] script to determine target repo (was DPDK Lab)
> 
> 
> 
> > -----Original Message-----
> > From: Xu, Qian Q
> > Sent: Friday, June 23, 2017 9:44 AM
> > To: Thomas Monjalon <thomas at monjalon.net>; Wei, FangfangX
> > <fangfangx.wei at intel.com>
> > Cc: ci at dpdk.org; Richardson, Bruce <bruce.richardson at intel.com>;
> > O'Driscoll, Tim <tim.odriscoll at intel.com>; Eugene Voronov
> > <eugene at mellanox.com>
> > Subject: RE: [dpdk-ci] script to determine target repo (was DPDK Lab)
> >
> > Thomas/Bruce
> > 1. For determining the repo tree to target, I don't believe that we
> > can ever
> > > come up with a 100% accurate rule, as the tree to which a set is to
> > > be applied can be difficult to determine, so it may be done on the
> > > basis of
> > on-list discussion.
> > > A 90% accurate rule it what we may have to accept.
> >
> > -- Then if we find the performance issue, then maybe it's a false
> > alarm due to apply to the wrong repo. So, we may face many false
> > alarms according with the time.
> > Then people may not treat the performance issue as a problem, so I
> > still think we need to try 100% accurate to have a more trustable
> > result when we send out the alarm.
> 
> I find that rather improbable, and not worth considering. For that to per a
> problem multiple unlikely events have to occur:
> 1) we mis-identify the tree on which the set is to be applied (we should be able
> to get to 90% accuracy here)
> 2) the patchset must apply cleanly to the "wrong" tree (this is reasonably likely,
> but it's still another condition that has to be met for us to have a problem)
> 3) the patchset has to cause a performance regression in the "wrong" tree
> 4) but NOT cause a regression when in the right tree.
> 
> If we assume 90% accuracy of tree identification, optimistically that 90% of
> patches will apply to the wrong tree, that 5% of patches cause a performance
> regression (an overestimate IMHO), and that even 1/3 of those won't cause a
> performance regression in the right tree (a very overestimate IMHO, I would
> expect just about none of them to even have this), it still means that only about
> 1 patch in 1000 will show as a false positive performance regression.
> 
> 0.1 (mis-identify) * 0.9 (applies ok) * 0.05 (regression) * 0.33 (no regression) =
> 0.0015, or 0.15%
> 
> So worst case, I still don't think we have a problem for the scenario you describe.

OK, Bruce, so the question is that how can we ensure 90% accuracy? How to check if it's 90% or 80%?