[dpdk-ci] Intel PerPatch Build

[Thomas Monjalon]

2016-11-29 03:56, Xu, Qian Q:
> I feel we need to split the report.
> What do you think of having a report per OS? or a report per build?
> It would show easily how big is the failure by looking at the counters and descriptions in patchwork.
> The email volume would be bigger but is it a problem?
> ----current report is for per patch build report, and for each patch, we now have 18 builds. We will send out 1 build report for 1 patch.  If we send 18 reports for 1 patch, then it may be too many reports.

Why is it too many reports?

> If you want to check how many failures for the build, for example, 18 builds, then 1 failures, there is 2 ways to check. 
> 1. you can get how many failures information from the Build Summary. 
> > Patch17274-17274 --> compile pass
> > Build Summary: 18 Builds Done, 18 Successful, 0 Failures
> 
> 2. We can add the failure numbers in the title, such as [dpdk-test-report] |2 FAILURE|xxxxx
> 
> Does it make sense? 

In one build test, there can be several errors (even if we stop at the first error).
And one error can be seen in several build tests.
So we cannot really count the number of real errors, but we can count the number
of tests which are failing.
Yes we could add the number of failed tests in a test report.
I just thought that it would be simpler to understand if sending 1 report per test.

An example of the benefit of splitting:
An error with recent GCC is a failure.
An error with ICC or an old GCC may be just a warning.
Finer is the grain of the reports, better is the patchwork overview.
Today, we have few tests only. But later we could have more and more
test instances sending the same kind of reports.

Another example:
If a test is failing for some reasons, it will fail for every patches.
We must have a way to ignore it when reading the results.
If it is a separate report, it is easier to ignore.

> If I understand well you test every patches of a series at once and send the report for the last patch of the series?
> ----No. We send the report for each patch of the series, not the last one. The process of per patch build check is as below. 
> 1. We pull the patchset, for example, we have 10 patches in 1 patchset. We will apply all the patches to the git tree. 
> If apply failed for one tree, we will try other trees(such as next-virtio, next-crypto, next-net), if any tree can be applied, then 
> We think the patch apply OK. If all trees can't be applied, we take it as Failure. Then we will send out 10 patch reports, each one is Failure. 
> 
> 2. If apply patch is OK, then we do per patch build check one by one. After 10 patch's build done, we will send report one by one. 
> For example, 10 patches in 1 patchset, we will do 18 builds for each patch, totally 180 builds for the patchset. Then send report for each patch after the last patch. 
> Test1: patch1: 18 build
> Test2: patch1 + patch2: 18 build
> ....
> Test10: patch1+xxx+patch10: 18build
> 
> Is it clear? 

OK perfect, thanks


PS: please Qian, could you configure your email client to use reply quoting?