[dpdk-users] Issue with ipsec-secgw sample application on VM using Intel QAT device (pass-through mode)

[Chinmaya Dwibedy]

Hi Sergio,


Thank your for clarification. I am using dpdk-2.2.0. Here goes our
configuration on VM. Can you please have a look into the same and let me
know if anything is wrong. Will try to run the autotest, ipsec-segw on host
as per your suggestion.


[root at vpn-s ~]# uname -r

3.12.9-301.fc20.x86_64

[root at vpn-s ~]#


[root at vpn-s ~]# cat /etc/redhat-release

Fedora release 20 (Heisenbug)

[root at vpn-s ~]#




[root at vpn-s ~]# cat /proc/cmdline

BOOT_IMAGE=/boot/vmlinuz-3.12.9-301.fc20.x86_64
root=UUID=ab47cbc9-68ee-403c-96a5-184e68238e65 ro console=ttyS0,115200n8
iommu=pt intel_iommu=on

[root at vpn-s ~]#


[root at vpn-s QAT]# grep -i "iommu.*enabled" /var/log/messages

Jun 22 07:31:17 vpn-s kernel: [    0.000000] Intel-IOMMU: enabled


[root at vpn-s QAT]#

[root at vpn-s dpdk-2.2.0]# lspci -nn | grep 0443

00:06.0 Co-processor [0b40]: Intel Corporation Device [8086:0443]

[root at vpn-s dpdk-2.2.0]#


Installed Intel qat driver installation on VM (downloaded from
https://01.org/packet-processing/intel%C2%AE-quickassist-technology-drivers-and-patches)
and followed the

https://01.org/sites/default/files/page/330750-005_qat_gsg.pdf guide.
Note, used the following installation options i.e., ./installer.sh install
QAT1.6 guest. After this , it detects QAT VF.


[root at vpn-s QAT]# service qat_service status

There is 1 acceleration device(s) in the system:

 icp_dev0 - type=dh895xcc, inst_id=0, node_id=0,  bdf=00:06:0, #accel=1,
#engines=1, state=up

[root at vpn-s QAT]#


[root at vpn-s QAT]# lsmod | grep qa

icp_qa_al_vf          813442  1

[root at vpn-s QAT]#



The below are DPDK configuration options in
config/defconfig_x86_64-native-linuxapp-gcc.


include "common_linuxapp"



CONFIG_RTE_MACHINE="native"

CONFIG_RTE_ARCH="x86_64"

CONFIG_RTE_ARCH_X86_64=y

CONFIG_RTE_ARCH_64=y

CONFIG_RTE_TOOLCHAIN="gcc"

CONFIG_RTE_TOOLCHAIN_GCC=y

CONFIG_RTE_LIBRTE_PMD_AESNI_MB=y

CONFIG_RTE_LIBRTE_CRYPTODEV_DEBUG=y

CONFIG_RTE_LIBRTE_VIRTIO_PMD=y

CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_INIT=y

CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_DRIVER=y

CONFIG_RTE_LIBRTE_PMD_QAT=y

CONFIG_RTE_QAT_PMD_MAX_NB_SESSIONS=128

CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_INIT=y

CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_TX=y

CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_RX=y

CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_DRIVER=y



The below is my setup.sh


export RTE_SDK=/opt/dpdk-2.2.0

export RTE_TARGET=x86_64-native-linuxapp-gcc

export RTE_ARCH=x86_64

export AESNI_MULTI_BUFFER_LIB_PATH=/opt/code

make config T=x86_64-native-linuxapp-gcc

sed -ri 's,(PMD_PCAP=).*,\1y,' build/.config

make install T=x86_64-native-linuxapp-gcc

mkdir -p /mnt/huge

mount -t hugetlbfs nodev /mnt/huge

echo 2048 >
/sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages

sudo modprobe uio_pci_generic

sudo modprobe uio

rmmod igb_uio

insmod
./x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/igb_uio/igb_uio.ko

echo -n "0000:00:06.0" > /sys/bus/pci/devices/0000\:00\:06.0/driver/unbind

echo "8086 0443" > /sys/bus/pci/drivers/igb_uio/new_id

lspci -vvd:443

./tools/dpdk_nic_bind.py --bind=uio_pci_generic eth1

./tools/dpdk_nic_bind.py --bind=uio_pci_generic 00:03.0

./tools/dpdk_nic_bind.py -s


Also modified the cryptodevs_init() ( examples/ipsec-secgw/ipsec-secgw.c)
to rte_cryptodev_start(cdev_id); after rte_cryptodev_queue_pair_setup().
The purpose of calling this to start the transmit and the receive units of
the QAT device. It returns success. But still getting the same issue i.e.
rte_cryptodev_dequeue_burst () does not return crypto packet. Note that,
when I call rte_cryptodev_queue_pair_setup () with 3rd argument as NULL (in
which case default configuration will be used) , it crashes. Is it a known
issue?





Regards,

Chinmaya

On Tue, Jun 21, 2016 at 9:10 PM, Sergio Gonzalez Monroy <
sergio.gonzalez.monroy at intel.com> wrote:

> On 21/06/2016 08:26, Chinmaya Dwibedy wrote:
>
>>
>> Hi Sergio,
>>
>>
>> As suggested by you, run ‘app/test' application and then run
>> 'cryptodev_qat_autotest' . Found the same issue i.e., It halts in
>> rte_cryptodev_dequeue_burst () (while loop) in process_crypto_request ()
>> function (app/test/test_cryptodev.c).
>>
>> Debugged the qat_crypto_pkt_rx_burst() (in
>> drivers/crypto/qat/qat_crypto.c) and found that , the value of resp_msg is
>> 0x7F7F7F7F (i.e., ADF_RING_EMPTY_SIG) . Thus it is not getting crypto
>> packet (i.e., rte_mbuf) from the RX queue.  But I find that, in
>> qat_crypto_pkt_tx_burst(), the  qat_alg_write_mbuf_entry() is being called.
>> The qat_alg_write_mbuf_entry () prints qat_req  using rte_hexdump () and
>> returns zero.
>>
>>
>> One question, the base_addr + tail of tx_q (in qat_crypto_pkt_tx_burst())
>> should be same as base_addr + head (in qat_crypto_pkt_rx_burst()). Right ?
>> Please feel free to correct me if I am wrong.
>>
>>
>> Also can you please provide some pointers for further debugging? Would it
>> be configuration issue? Note that, I am using VF in VM.
>>
>>
> In theory  you should be able to run ipsec-secgw sample app with QAT VF as
> long as the setup is right.
> That the autotest fails would likely point to some possible issues with
> the setup.
>
> Have you tried to run the autotest in the host?
>
> Sergio
>
>
>> Thank you once again for your prompt response and great support.
>>
>>
>> Regards,
>>
>> Chinmaya
>>
>>
>> On Mon, Jun 20, 2016 at 3:04 PM, Sergio Gonzalez Monroy <
>> sergio.gonzalez.monroy at intel.com <mailto:sergio.gonzalez.monroy at intel.com>>
>> wrote:
>>
>>     On 20/06/2016 10:08, Chinmaya Dwibedy wrote:
>>
>>
>>         Hi Sergio,
>>
>>
>>         Agreed.  We might not dequeue the same amount of crypto ops we
>>         just previously enqueued, it's asynchronous. But in this case,
>>         I have sent just one UDP packet. So there will be one crypto
>>         ops. Right?  Also I put a sleep (50) after the
>>         rte_crypto_enqueue_burst() function in ipsec_processing()
>>         (ipsec.c) , so as to allow more time  ( for QAT device) for
>>         processing. Still getting the same result i.e., the
>>         rte_crypto_dequeue_burst () functionreturns zero.
>>
>>
>>         In case of S/W crypto device (i.e., AESNI),   the VM gets
>>         inbound UDP packets on Port 1/eth1, encapsulates (after
>>         consulting its SPD) in an IPsec ESP packet and sends to its
>>         peer through Port 0/eth0 interface.
>>
>>
>>         Yes, the security policy, security association and Routing
>>         entries/configurations are exactly same. Please feel free to
>>         let me know if you need additional information.
>>
>>
>>     Could you try to run 'app/test' application then run
>>     'cryptodev_qat_autotest' ? That is a functional test for cryptodev
>>     QAT PMD.
>>
>>     Sergio
>>
>>
>>
>