Opensource ethdev tests

[Honnappa Nagarahalli]

Hi Andrew,
	Few questions inline.

> -----Original Message-----
> From: Andrew Rybchenko <andrew.rybchenko at oktetlabs.ru>
> Sent: Monday, October 3, 2022 5:00 AM
> To: thomas at monjalon.net
> Cc: dev at dpdk.org; Ferruh Yigit <ferruh.yigit at xilinx.com>; Ajit Khaparde
> (ajit.khaparde at broadcom.com) <ajit.khaparde at broadcom.com>;
> jerinj at marvell.com; Dongdong Liu <liudongdong3 at huawei.com>; Qiming
> Yang <qiming.yang at intel.com>; Yuying Zhang <Yuying.Zhang at intel.com>;
> Beilei Xing <beilei.xing at intel.com>; Qi Zhang <qi.z.zhang at intel.com>;
> hemant.agrawal at nxp.com; Maxime Coquelin
> <maxime.coquelin at redhat.com>; Viacheslav Ovsiienko
> <viacheslavo at nvidia.com>; Stephen Hemminger
> <sthemmin at microsoft.com>
> Subject: Opensource ethdev tests
> 
> Hi Thomas and community,
> 
> May I ask to add https://ts-factory.io/ to the DPDK ecosystem.
> I'm not 100% that it is suitable for the ecosystem since it is not consuming
> DPDK, but rather testing DPDK ethdev.
Few questions:
1) Are you asking that we add these to the UNH infrastructure?
2) We have DTS already and the community is working on integrating DTS into DPDK. Does it make sense to understand any gaps in the test cases and incorporate them in DTS instead?
3) Are there any additional benefits this brings compared to DTS?

> Anyway it could be useful for ethdev PMD developers and maintainers.
> 
> I'll not repeat what is written on the site [1] and documentation (including
> the framework [2] and test scenarios documentation [3]) to keep the mail
> small enough.
> 
> [1] https://ts-factory.io/
> [2] https://ts-factory.io/doc/test-environment/
> [3] https://ts-factory.io/doc/dpdk-ethdev-ts/
> 
> First of all I'd like to thank Xilinx/AMD for making these tests opensource.
> Testing framework (Test Environment) used by these tests is an opensource as
> well.
> 
> The database has examples of testing log for QEMU virtio [4], Solarflare
> SFN8522 [5], Intel X710 [6] and Mellanox ConnectX-5 [7] NICs.
> 
> [4]
> https://ts-factory.io/bublik/v2/runs?runData=pci-
> 1af4%3BTS_NAME%3Ddpdk-ethdev-ts
> [5]
> https://ts-factory.io/bublik/v2/runs?runData=pci-
> 1924%3BTS_NAME%3Ddpdk-ethdev-ts
> [6]
> https://ts-factory.io/bublik/v2/runs?runData=pci-8086-
> 1572%3BTS_NAME%3Ddpdk-ethdev-ts
> [7]
> https://ts-factory.io/bublik/v2/runs?runData=pci-15b3-
> 1017%3BTS_NAME%3Ddpdk-ethdev-ts
> 
> Full list of sample DPDK ethdev logs [8].
> 
> [8] https://ts-factory.io/bublik/v2/runs?runData=TS_NAME%3Ddpdk-ethdev-
> ts
> 
> Testing results are classified into 6 categories. There are 3 results:
> passed, failed and skipped (when test fails to do its job because tested
> functionality itself or some required per-conditions are not supported).
> Each result could be either expected in accordance with filled in expectations
> or unexpected if obtained result does not match expectations. These
> expectations could differ for different NICs, tested DPDK version etc.
> 
> High rate of expected results for SFN8522 and virtio is explained by origin of
> the tests. Expectations for these NICs are mostly filled in.
> High number of unexpected results for i40e and mlx5 drivers does not mean
> these drivers or NICs are bad. First of all it is tests which could be wrong, too
> strict or just have bugs. Second, expectations (because of missing
> functionality or known aspects of the behaviour) for these NICs are not filled
> in in many-many cases.
> 
> Let's get down to few examples of unexpected results.
> 
> 1. QEMU virtio. VLAN tagged packet is not delivered. it is a virtio testing when
> two VMs talk to each other via Linux bridge.
>     Sent packet is observed on Peer (line 38), but DPDK fails to receive it (line
> 65). Most likely it is some kind of misconfiguration.
> https://ts-
> factory.io/bublik/v2/log/93205?focusId=93288&mode=treeAndinfoAndlog
> 
> 2. QEMU virtio. Inconsistent number of xstats on get number (38) and actual
> get (just 13 returned).
>      The trick here is that xstats API is called just after rte_eth_dev_configiure().
> I.e. queues are not configured and device is not started yet.
> https://ts-
> factory.io/bublik/v2/log/93205?focusId=93983&mode=treeAndinfoAndlog
> 
> 3. Intel X710. CWR TCP flag loss in dummy TSO case (i.e. when TSO payload is
> less than TCP MSS).
>      Sent packet has CWR bit set in TCP flags, but the packet received on Peer
> does not have it.
>      Since it is a dummy TSO case it is hardly critical, but still interesting aspect
> of the behaviour.
> https://ts-
> factory.io/bublik/v2/log/70553?focusId=72767&mode=treeAndinfoAndlog
> 
> 4. ConnectX-5. Prepared but stuck on Tx burst single segment TSO packet.
>      Tx prepare on line 51 accepts the packet, but attempt to transmit fails on
> line 62.
>      Of course the packet layout is specific since typically TSO header goes in its
> own segment,
>      but the behavbiour is still unfriendly since application never knows if Tx
> burst returns 0 since Tx ring is full or something else is wrong.
> https://ts-
> factory.io/bublik/v2/log/85618?focusId=87546&mode=treeAndinfoAndlog
> 
> 5. ConnectX-5: Sometimes huge number of tests bring fruits like this when all
> further tests fail because of NIC probe failure.
>      Most likely it was a bug in a particular driver version since the problem is
> not always repeatable.
> https://ts-
> factory.io/bublik/v2/log/48348?focusId=51740&mode=treeAndinfoAndlog
> 
> Performance testing results using testpmd are not representative since used
> hosts are too weak and not really tuned for performance testing. If testing
> hosts are good, these tests can do its job as well.
> 
> For example a number of bugs in net/virtion were found using these tests and
> corresponding patches sent upstream in the past.
> 
> These tests are fully automatic and suitable for release testing as well as
> everyday regressions tracking as soon as expectations are filled in.
> 
> Also having expectations for different NICs filled in allows to generate
> comparison reports to understand the difference in behaviour and supported
> features.
> 
> The testing framework supports collection of gcov-based coverage and
> generates reports if requested, for example [9] for i40e.
> 
> [9] https://ts-factory.io/logs/2022/09/30/fror-x710-p0-7/tce_log_dpdk.html
> 
> Besides DPDK ethdev tests there are testing logs for Linux net drivers for same
> NICs [10].
> 
> [10] https://ts-factory.io/bublik/v2/runs?runData=TS_NAME%3Dnet-drv-ts
> 
> Andrew.