[dpdk-dev] [PATCH v2] app/test: enhance sanity script

[Thomas Monjalon]

20/05/2018 19:41, Parthasarathy, JananeeX M:
> From: Thomas Monjalon [mailto:thomas at monjalon.net]
> > 18/05/2018 15:32, Parthasarathy, JananeeX M:
> > > From: Thomas Monjalon [mailto:thomas at monjalon.net]
> > > > 15/05/2018 18:14, Parthasarathy, JananeeX M:
> > > > > From: Thomas Monjalon [mailto:thomas at monjalon.net]
> > > > 
> > > > I think we need to keep a fast subset of tests.
> > > > 
> > > > Please check in mk/rte.sdktest.mk:
> > > >               PERFLIST=ring_perf,mempool_perf,memcpy_perf,hash_perf,ti
> > > >               mer_perf
> > > >         
> > > >         test-fast: BLACKLIST=-$(PERFLIST)
> > > 
> > > We have added the below test cases to blacklist in mk/rte.sdktest.mk
> > > reciprocal_division,reciprocal_division_perf,lpm_perf,red_all
> > > 
> > > Now time taken is around 4mins.
> > > Please let us know is there any minimum time limit for test execution.
> > > Is this time duration fine?
> > 
> > 4 minutes, it's a bit long for developer oriented tests.
> > 
> > We need to distinguish which tests are enough for quick tests,
> > and which ones can be in another category for full validation purpose.
> > 
> > Please can you make a summary of which tests you are adding,
> > and propose a priority classification.
> > 
> > Thanks
> 
> Please find below the test cases newly added to sanity script (autotest_data.py) with priority classification.
> 
> P1 - Main test cases

We can move some tests from P1. Examples:
	- bonding can be P2 with other drivers.
	- PMD perf and ring perf can P3 with other perf tests.

> P2 - Cryptodev test cases which requires dependent libraries   (Can be added if the environment is set accordingly)
> P3 - Perf test cases and others which takes more time (> 10s)  (these will be in the blacklist test cases)
> P4 - Logging/Dump test cases (to be checked whether these are really necessary)

I think this classification is good.
We can adjust the categories in the makefile to reflect it.
The goal is to have a really quick and core subset (equal P1 category),
and other categories available with a different make command
(drivers for P2, perf for P3 and debug for P4).

> The blacklist is updated in makefile "rte.sdktest.mk" with below list
> of test cases to reduce the total test execution time.
> 
> Blacklist test cases are ring_perf,mempool_perf,memcpy_perf,hash_perf,timer_perf,
> reciprocal_division,reciprocal_division_perf,lpm_perf,red_all,barrier,
> hash_multiwriter,timer_racecond,efd,hash_functions,eventdev_selftest_sw,
> member_perf,efd_perf,lpm6_perf,red_perf,distributor_perf
> 
> Currently after the above blacklist update, the total execution time of "make test-fast" is 54s.

Less than a minute is a good limit!
Maybe we can do even better with above suggestions.

Thanks for working on it.
Feel free to send a new patch when you think it is ready.
We can try to have the new autotests when starting 18.08.
Then we should communicate to enforce a better coverage of the autotests.