[dpdk-stable] [dpdk-dev] [dpdklab] RE: [PATCH v4] mbuf: fix reset on mbuf free

[Morten Brørup]

> From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of Lincoln Lavoie
> Sent: Thursday, January 21, 2021 5:35 PM
> To: Morten Brørup
> 
> Hi All,
> 
> Trying to follow the specific conversation.  It is correct, the lab
> does
> not list the specific throughput values achieved by the hardware, as
> that
> data can be sensitive to the hardware vendors, etc. The purpose of the
> lab
> is to check for degradations caused by patches, so the difference is
> really
> the important factor.  The comparison is against a prior run on the
> same
> hardware, via the DPDK main branch, so any delta should be caused by
> the
> specific patch changes (excluding statistical "wiggle").
> 

Thank you for clarifying, Lincoln.

This sounds like a perfect solution to the meet the purpose.

I request that you change the output to show the relative difference (i.e. percentage above/below baseline), instead of the absolute difference (i.e. number of packets per second).

By showing a percentage, anyone reading the test report can understand if the difference is insignificant, or big enough to require further discussion before accepting the patch. Showing the difference in packets per second requires the reader of the test report to have prior knowledge about the expected performance.

> If the group would prefer, we could calculate additional references if
> desired (i.e. difference from the last official release, or a monthly
> run
> of the current, etc.).  We just need the community to define their
> needs,
> and we can add this to the development queue.
> 

I retract my suggestion about adding additional references. You clearly explained how your baselining works, and I think it fully serves the needs of a regression test.

So please just put this small change in your development queue: Output the difference in percent with a few decimals after the comma, instead of the difference in packets per second.

For readability, performance drops should be shown as negative values, and increases as positive, e.g.:

Difference = (NewPPS - BaselinePPS) / BaselinePPS * 100.00 %.


If we want to compare performance against official releases, current or older, it should go elsewhere, not in the continuous testing environment. E.g. the release notes could include a report showing differences from the last few official releases. But that is a task for another day.


> Cheers,
> Lincoln
> 
> 
> On Thu, Jan 21, 2021 at 4:29 AM Morten Brørup
> <mb at smartsharesystems.com>
> wrote:
> 
> > > From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of Ferruh Yigit
> > > Sent: Thursday, January 21, 2021 10:19 AM
> > >
> > > On 1/15/2021 6:39 PM, Ali Alnubani wrote:
> > > > Hi,
> > > > Adding Ferruh and Zhaoyan,
> > > >
> > > >> Ali,
> > > >>
> > > >> You reported some performance regression, did you confirm it?
> > > >> If I get no reply by monday, I'll proceed with this patch.
> > > >
> > > > Sure I'll confirm by Monday.
> > > >
> > > > Doesn't the regression also reproduce on the Lab's Intel servers?
> > > > Even though the check iol-intel-Performance isn't failing, I can
> see
> > > that the throughput differences from expected for this patch are
> less
> > > than those of another patch that was tested only 20 minutes
> earlier.
> > > Both patches were applied to the same tree:
> > > >
> > > > https://mails.dpdk.org/archives/test-report/2021-
> January/173927.html
> > > >> | 64         | 512     | 1.571                               |
> > > >
> > > > https://mails.dpdk.org/archives/test-report/2021-
> January/173919.html
> > > >> | 64         | 512     | 2.698                               |
> > > >
> > > > Assuming that pw86457 doesn't have an effect on this test, it
> looks
> > > to me that this patch caused a regression in Intel hardware as
> well.
> > > >
> > > > Can someone update the baseline's expected values for the Intel
> NICs
> > > and rerun the test on this patch?
> > > >
> > >
> > > Zhaoyan said that the baseline is calculated dynamically,
> > > what I understand is baseline set based on previous days
> performance
> > > result, so
> > > it shouldn't require updating.
> >
> > That sounds smart!
> >
> > Perhaps another reference baseline could be added, for informational
> > purposes only:
> > Deviation from the performance of the last official release.
> >
> > >
> > > But cc'ed the lab for more details.
> >
> >
> 
> --
> *Lincoln Lavoie*
> Senior Engineer, Broadband Technologies
> 21 Madbury Rd., Ste. 100, Durham, NH 03824
> lylavoie at iol.unh.edu
> https://www.iol.unh.edu
> +1-603-674-2755 (m)
> <https://www.iol.unh.edu>