[PATCH v2] test: add cryptodev crosscheck suite

[Power, Ciara]

> -----Original Message-----
> From: Power, Ciara
> Sent: Wednesday 8 March 2023 17:31
> To: Volodymyr Fialko <vfialko at marvell.com>; dev at dpdk.org; Akhil Goyal
> <gakhil at marvell.com>; Fan Zhang <fanzhang.oss at gmail.com>
> Cc: jerinj at marvell.com; anoobj at marvell.com; hemant.agrawal at nxp.com; Ji,
> Kai <kai.ji at intel.com>
> Subject: RE: [PATCH v2] test: add cryptodev crosscheck suite
> 
> Hi Volodymyr,
> 
> 
> > -----Original Message-----
> > From: Volodymyr Fialko <vfialko at marvell.com>
> > Sent: Thursday 9 February 2023 12:34
> > To: dev at dpdk.org; Akhil Goyal <gakhil at marvell.com>; Fan Zhang
> > <fanzhang.oss at gmail.com>
> > Cc: jerinj at marvell.com; anoobj at marvell.com; hemant.agrawal at nxp.com;
> > Ji, Kai <kai.ji at intel.com>; Power, Ciara <ciara.power at intel.com>;
> > Volodymyr Fialko <vfialko at marvell.com>
> > Subject: [PATCH v2] test: add cryptodev crosscheck suite
> >
> > Add a validation test suite that helps in verifying that the output
> > generated by two different cryptodevs match for a wide range of input
> > parameter combinations.
> >
> > Crypto autotest performs a comprehensive testing of the cryptodev but
> > since it performs verification by comparing against known vectors, the
> > extend to which various parameters (like packet size) can be tested is
> > limited. This test suite attempts to simulate various cases by running
> > same test case on different cryptodevs and compares the output
> > generated. The test suite relies on capabilities to determine the
> > combinations of tests to be attempted.
> >
> > A typical use case would be to compare outputs generated from a
> > standard driver such as openSSL PMD and a new cryptodev PMD. This test
> > suite is to compliment the testing coverage that crypto autotest provides.
> >
> > Currently supported symmetric xforms(cipher, auth, aead) without
> chaining.
> >
> > Example command:
> > DPDK_TEST=cryptodev_crosscheck ./dpdk-test \
> > 	-a <cryptodev> --vdev "crypto_openssl"
> >
> > Signed-off-by: Volodymyr Fialko <vfialko at marvell.com>
> > ---
> > V2:
> > - Updated commit message.
> >
> <snip>
> > +
> > +static void
> > +capabilities_inspect(void)
> > +{
> > +	struct rte_cryptodev_sym_capability_idx
> > cap_indexes[CRYPTO_ALGOS_LEN], *cap_idx;
> > +	struct crypto_testsuite_params *ts_params = &testsuite_params;
> > +	const struct rte_cryptodev_symmetric_capability *sym_capa;
> > +	struct rte_cryptodev_symmetric_capability *common_capa;
> > +	uint32_t algo, i, dev_id, caps_idx;
> > +
> > +	caps_idx = 0;
> > +	/* Create capability idx for known algorithms*/
> > +	for (algo = 1; algo <= CRYPTO_AUTH_MAX_IDX; algo++) {
> > +		cap_idx = &cap_indexes[caps_idx++];
> > +		cap_idx->type = RTE_CRYPTO_SYM_XFORM_AUTH;
> > +		cap_idx->algo.auth = algo;
> > +	}
> > +	for (algo = 1; algo <= CRYPTO_CIPHER_MAX_IDX; algo++) {
> > +		cap_idx = &cap_indexes[caps_idx++];
> > +		cap_idx->type = RTE_CRYPTO_SYM_XFORM_CIPHER;
> > +		cap_idx->algo.cipher = algo;
> > +	}
> > +	for (algo = 1; algo <= CRYPTO_AEAD_MAX_IDX; algo++) {
> > +		cap_idx = &cap_indexes[caps_idx++];
> > +		cap_idx->type = RTE_CRYPTO_SYM_XFORM_AEAD;
> > +		cap_idx->algo.aead = algo;
> > +	}
> > +
> > +	for (caps_idx = 0; caps_idx < CRYPTO_ALGOS_LEN; caps_idx++) {
> > +		/* Gather common capabilities */
> > +		common_capa = &common_symm_capas[caps_idx];
> > +		common_capa->xform_type =
> > RTE_CRYPTO_SYM_XFORM_NOT_SPECIFIED;
> > +		for (i = 0; i < ts_params->valid_dev_count; i++) {
> > +			dev_id = ts_params->valid_devs[i];
> > +			sym_capa =
> > rte_cryptodev_sym_capability_get(dev_id,
> > +					&cap_indexes[caps_idx]);
> > +			if (sym_capa == NULL) {
> > +				/* Capability not supported by one of devs,
> > mark and skip */
> > +				goto next_algo;
> > +			}
> > +
> > +			if (common_capa->xform_type ==
> > RTE_CRYPTO_SYM_XFORM_NOT_SPECIFIED) {
> > +				/* First time initialization, copy data, go to
> > next device  */
> > +				*common_capa = *sym_capa;
> > +				continue;
> > +			}
> [CP]
> 
> This function - from what I understand after review,  is looping through all
> algorithms in DPDK crypto, and then checking if each is supported by each
> device?
> Could we instead just take the capabilities list from one device as the starting
> point - That will be the max list of capabilities, they will only get knocked out
> if not supported on the 2nd/3rd device etc.
> 
> 
> <snip>
> 
> Some trials I did with this patch showed some issues.
> 
> 1. For QAT + AESNI_MB I get all skipped tests. I would have expected some
> common algs here.
> + ------------------------------------------------------- +
>  + Test Suite Summary : Crosscheck Unit Test Suite  + -----------------------------
> -------------------------- +  + Algo AUTH  'null' : 0/0 passed, 0/0 skipped, 0/0
> failed, 0/0 unsupported  + Algo AUTH  'aes-cbc-mac' : 0/0 passed, 0/0
> skipped, 0/0 failed, 0/0 unsupported  + Algo AUTH  'aes-cmac' : 0/0 passed,
> 0/0 skipped, 0/0 failed, 0/0 unsupported  + Algo AUTH  'aes-gmac' : 0/0
> passed, 0/0 skipped, 0/0 failed, 0/0 unsupported  + Algo AUTH  'aes-xcbc-
> mac' : 0/0 passed, 0/0 skipped, 0/0 failed, 0/0 unsupported  + Algo AUTH
> 'kasumi-f9' : 0/0 passed, 0/0 skipped, 0/0 failed, 0/0 unsupported  + Algo
> AUTH  'md5' : 0/0 passed, 0/0 skipped, 0/0 failed, 0/0 unsupported  + Algo
> AUTH  'md5-hmac' : 0/0 passed, 0/0 skipped, 0/0 failed, 0/0 unsupported  +
> Algo AUTH  'sha1' : 0/0 passed, 0/0 skipped, 0/0 failed, 0/0 unsupported  +
> Algo AUTH  'sha1-hmac' : 0/0 passed, 0/0 skipped, 0/0 failed, 0/0
> unsupported  + Algo AUTH  'sha2-224' : 0/0 passed, 0/0 skipped, 0/0 failed,
> 0/0 unsupported  + Algo AUTH  'sha2-224-hmac' : 0/0 passed, 0/0 skipped, 0/0
> failed, 0/0 unsupported  + Algo AUTH  'sha2-256' : 0/0 passed, 0/0 skipped,
> 0/0 failed, 0/0 unsupported  + Algo AUTH  'sha2-256-hmac' : 0/0 passed, 0/0
> skipped, 0/0 failed, 0/0 unsupported  + Algo AUTH  'sha2-384' : 0/0 passed,
> 0/0 skipped, 0/0 failed, 0/0 unsupported  + Algo AUTH  'sha2-384-hmac' : 0/0
> passed, 0/0 skipped, 0/0 failed, 0/0 unsupported  + Algo AUTH  'sha2-512' :
> 0/0 passed, 0/0 skipped, 0/0 failed, 0/0 unsupported  + Algo AUTH  'sha2-512-
> hmac' : 0/0 passed, 0/0 skipped, 0/0 failed, 0/0 unsupported  + Algo AUTH
> 'snow3g-uia2' : 0/0 passed, 0/0 skipped, 0/0 failed, 0/0 unsupported  + Algo
> AUTH  'zuc-eia3' : 0/0 passed, 0/0 skipped, 0/0 failed, 0/0 unsupported  + Algo
> AUTH  'sha3-224' : 0/0 passed, 0/0 skipped, 0/0 failed, 0/0 unsupported  +
> Algo AUTH  'sha3-224-hmac' : 0/0 passed, 0/0 skipped, 0/0 failed, 0/0
> unsupported  + Algo AUTH  'sha3-256' : 0/0 passed, 0/0 skipped, 0/0 failed,
> 0/0 unsupported  + Algo AUTH  'sha3-256-hmac' : 0/0 passed, 0/0 skipped, 0/0
> failed, 0/0 unsupported  + Algo AUTH  'sha3-384' : 0/0 passed, 0/0 skipped,
> 0/0 failed, 0/0 unsupported  + Algo AUTH  'sha3-384-hmac' : 0/0 passed, 0/0
> skipped, 0/0 failed, 0/0 unsupported  + Algo AUTH  'sha3-512' : 0/0 passed,
> 0/0 skipped, 0/0 failed, 0/0 unsupported  + Algo AUTH  'sha3-512-hmac' : 0/0
> passed, 0/0 skipped, 0/0 failed, 0/0 unsupported  + Algo CIPHER 'null' : 0/0
> passed, 0/0 skipped, 0/0 failed, 0/0 unsupported  + Algo CIPHER '3des-cbc' :
> 0/0 passed, 0/0 skipped, 0/0 failed, 0/0 unsupported  + Algo CIPHER '3des-ctr'
> : 0/0 passed, 0/0 skipped, 0/0 failed, 0/0 unsupported  + Algo CIPHER '3des-
> ecb' : 0/0 passed, 0/0 skipped, 0/0 failed, 0/0 unsupported  + Algo CIPHER
> 'aes-cbc' : 0/0 passed, 0/0 skipped, 0/0 failed, 0/0 unsupported  + Algo
> CIPHER 'aes-ctr' : 0/0 passed, 0/0 skipped, 0/0 failed, 0/0 unsupported  + Algo
> CIPHER 'aes-ecb' : 0/0 passed, 0/0 skipped, 0/0 failed, 0/0 unsupported  +
> Algo CIPHER 'aes-f8' : 0/0 passed, 0/0 skipped, 0/0 failed, 0/0 unsupported  +
> Algo CIPHER 'aes-xts' : 0/0 passed, 0/0 skipped, 0/0 failed, 0/0 unsupported  +
> Algo CIPHER 'arc4' : 0/0 passed, 0/0 skipped, 0/0 failed, 0/0 unsupported  +
> Algo CIPHER 'kasumi-f8' : 0/0 passed, 0/0 skipped, 0/0 failed, 0/0
> unsupported  + Algo CIPHER 'snow3g-uea2' : 0/0 passed, 0/0 skipped, 0/0
> failed, 0/0 unsupported  + Algo CIPHER 'zuc-eea3' : 0/0 passed, 0/0 skipped,
> 0/0 failed, 0/0 unsupported  + Algo CIPHER 'des-cbc' : 0/0 passed, 0/0
> skipped, 0/0 failed, 0/0 unsupported  + Algo CIPHER 'aes-docsisbpi' : 0/0
> passed, 0/0 skipped, 0/0 failed, 0/0 unsupported  + Algo CIPHER 'des-
> docsisbpi' : 0/0 passed, 0/0 skipped, 0/0 failed, 0/0 unsupported  + Algo AEAD
> 'aes-gcm' : 0/0 passed, 0/0 skipped, 0/0 failed, 0/0 unsupported  + Algo AEAD
> 'chacha20-poly1305' : 0/0 passed, 0/0 skipped, 0/0 failed, 0/0 unsupported  +
> ------------------------------------------------------- +
>  + Sub Testsuites Total :     46
>  + Sub Testsuites Skipped :   46
>  + Sub Testsuites Passed :     0
>  + Sub Testsuites Failed :     0
>  + ------------------------------------------------------- +
>  + Tests Total :        0
>  + Tests Skipped :      0
>  + Tests Executed :     0
>  + Tests Unsupported:   0
>  + Tests Passed :       0
>  + Tests Failed :       0
>  + ------------------------------------------------------- + Test Skipped
> 
> 
> 2. For 2 openssl devices I get 3 failures, which is curious as they are the same.
> Seems to be a space issue:
> USER1: Operation status 2
> USER1: No space for aad in single mbuf
> EAL: Test assert crosscheck_all_devices line 776 failed: Error occurred during
> processing
> EAL: Test assert crosscheck_with_profile_run line 841 failed: Error occurred
> during encryption
> 
> 
> Thanks,
> Ciara
[CP] 

Comments above are based on V3 version of patch ... replied to the wrong email.
-Ciara