[dpdk-users] Issue with ipsec-secgw sample application on VM using Intel QAT device (pass-through mode)

[Chinmaya Dwibedy]

Hi Sergio/All,


Upon further investigation found that, on Guest VM if I remove
“intel_iommu=on” boot line parameters and run app/test and then
crypto_qat_autotest tool, it shows all the test cases are passed.
Thereafter when I run ipsec-segw application, I got the below error in
dmesg on host. Is it an issue with crypto device configuration and queue
pair setup? Please suggest.



[root at stag48 ~(keystone_admin)]# dmesg -c

[ 1264.266965] dmar: DRHD: handling fault status reg 2

[ 1264.266972] dmar: DMAR:[DMA Read] Request device [88:04.7] fault addr
ff53f000f000

DMAR:[fault reason 06] PTE Read access is not set

[ 1264.266978] icp_qa_al err: adf_dh895xcc_adf_isr_LogAE_Int: AE #5 Error

Uncorrectable Error: Yes

Control Store Error: No

Control Store information: Address 0x2A4C - syndrome 0x0

Register ECC Error No - Parity Error: Yes

Register information: address 0x2 - type Transfer

[ 1264.266980] dmar: DMAR:[DMA Read] Request device [88:04.7] fault addr
ff53f000e000

DMAR:[fault reason 06] PTE Read access is not set

[ 1264.266986] dmar: DMAR:[DMA Read] Request device [88:04.7] fault addr
ff53f000e000

DMAR:[fault reason 06] PTE Read access is not set

[ 1264.266990] dmar: DMAR:[DMA Read] Request device [88:04.7] fault addr
ff53f000f000

DMAR:[fault reason 06] PTE Read access is not set

[ 1264.266993] dmar: DMAR:[DMA Read] Request device [88:04.7] fault addr
ff53f000e000

DMAR:[fault reason 06] PTE Read access is not set

[ 1264.266997] icp_qa_al err: adf_dh895xcc_adf_isr_LogAE_Int: AE #7 Error

Uncorrectable Error: Yes

Control Store Error: No

Control Store information: Address 0x3FFD - syndrome 0x0

Register ECC Error No - Parity Error: Yes

Register information: address 0x22 - type Transfer

[ 1264.266999] dmar: DMAR:[DMA Read] Request device [88:04.7] fault addr
ff53f000e000

DMAR:[fault reason 06] PTE Read access is not set

[ 1264.267003] dmar: DMAR:[DMA Read] Request device [88:04.7] fault addr
ff53f000e000

DMAR:[fault reason 06] PTE Read access is not set

[ 1264.267007] dmar: DMAR:[DMA Read] Request device [88:04.7] fault addr
ff53f000f000

DMAR:[fault reason 06] PTE Read access is not set

[ 1264.267014] icp_qa_al err: adf_dh895xcc_adf_isr_LogAE_Int: AE #8 Error

Uncorrectable Error: Yes

Control Store Error: No

Control Store information: Address 0x2102 - syndrome 0x0

Register ECC Error No - Parity Error: Yes

Register information: address 0x2 - type Transfer

[ 1264.267020] icp_qa_al err: adf_dh895xcc_adf_isr_LogAE_Int: AE #9 Error

Uncorrectable Error: Yes

Control Store Error: No

Control Store information: Address 0x0 - syndrome 0x0

Register ECC Error No - Parity Error: Yes

Register information: address 0x22 - type Transfer

[ 1264.267026] icp_qa_al err: adf_dh895xcc_adf_isr_LogAE_Int: AE #b Error

Uncorrectable Error: Yes

Control Store Error: No

Control Store information: Address 0x3FFF - syndrome 0x0

Register ECC Error No - Parity Error: Yes

Register information: address 0x2 - type Transfer

[ 1264.267032] icp_qa_al err: adf_dh895xcc_adf_isr_handleUncoInterrupt:
Uncorrectable Push/Pull Misc Error

memory status: No errors occurred - Transaction Id 0x0 - Error type reserved

Bus Operation Type Push - Id 0x3859C020

[ 1264.267038] Reset needed for device: icp_dev1

[ 1264.267039] Auto Reset disabled. Please reset device1

[root at stag48 ~(keystone_admin)]#







Regards,

Chinmaya

On Thu, Jun 23, 2016 at 5:18 PM, Chinmaya Dwibedy <ckdwibedy at gmail.com>
wrote:

> Hi All,
>
>  On host, I installed dpdk-2.2.0 and then run 'cryptodev_qat_autotest’. I found this test case to be failed in dequeue burst API ().Upon looking into the below system logs (#dmesg), it seems, when this function is called, the QAT
>
> hardware fails to write the crypto packet to the DMA buffer.  When I passed intel_iommu=off command in boot line, it works. I think, when we have no IOMMU, "physical" address space is accessed directly by hardware, so code works.
>
> The issue has been observed if the system runs with the kernel parameter “intel_iommu=on”.
>
>
>
> [root at stag48 ~]# dmesg
>
> [  794.746134] dmar: DRHD: handling fault status reg 2
>
> [  794.746141] dmar: DMAR:[DMA Read] Request device [83:01.0] fault addr
> 3398c000
>
> DMAR:[fault reason 02] Present bit in context entry is clear
>
> [  794.746158] icp_qa_al err: adf_dh895xcc_adf_isr_handleUncoInterrupt:
> Uncorrectable Push/Pull Misc Error
>
> memory status: No errors occurred - Transaction Id 0x0 - Error type
> reserved
>
> Bus Operation Type Push - Id 0x80000
>
> [  794.746167] Reset needed for device: icp_dev0
>
> [  794.746167] Auto Reset disabled. Please reset device0
>
> [root at stag48 ~]#
>
>
>
> *Note: *The QAT VF (*83:01.0*) is bound to the DPDK UIO driver.
>
>
>
> 83:01.0 Co-processor: Intel Corporation DH895XCC Series QAT Virtual
> Function
>
>         Subsystem: Intel Corporation Device 0000
>
>         Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx-
>
>         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>
>         Latency: 0
>
>         Region 0: [virtual] Memory at c8280000 (64-bit, non-prefetchable)
> [size=4K]
>
>         Region 2: [virtual] Memory at c82a0000 (64-bit, non-prefetchable)
> [size=4K]
>
>         Capabilities: [90] MSI: Enable- Count=1/1 Maskable+ 64bit+
>
>                 Address: 0000000000000000  Data: 0000
>
>                 Masking: 00000000  Pending: 00000000
>
>         Kernel driver in use: igb_uio
>
>
>
>
>
> Now testing the same on Guest and finding the same issue (i.e., failing on dequeue burst API ()). Upon looking into error message (in dmesg) on host/hypervisor it says i.e., [28475.040313]
>
> vfio-pci 0000:88:04.6: buffer not found in pci_save_pcie_state. The PCI device 88:04.6 is the QAT VF, being assigned to Guest. I am trying to figure out what causes “buffer not found” issue .
>
> Please suggest if anyone has an idea on it.
>
>
>
> Here goes the output of dmesg
>
>
>
> [27086.464014] vfio-pci 0000:88:04.6: enabling device (0000 -> 0002)
>
> [27086.464040] vfio-pci 0000:88:04.6: buffer not found in pci_save_pcie_state
>
> [27086.564610] vfio-pci 0000:88:04.6: buffer not found in pci_save_pcie_state
>
> [27086.574052] vfio-pci 0000:88:04.6: buffer not found in pci_save_pcie_state
>
> [27089.786623] kvm: zapping shadow pages for mmio generation wraparound
>
> [27106.320042] vfio-pci 0000:88:04.6: irq 136 for MSI/MSI-X
>
> [28474.784373] vfio-pci 0000:88:04.6: buffer not found in pci_save_pcie_state
>
> [28475.040313] vfio-pci 0000:88:04.6: buffer not found in pci_save_pcie_state
>
> [28493.271950] vfio-pci 0000:88:04.6: irq 136 for MSI/MSI-X
>
>
>
>
>
> Regards,
>
> Chinmaya
>
>
>
>
>
>
>
> On Wed, Jun 22, 2016 at 1:39 PM, Chinmaya Dwibedy <ckdwibedy at gmail.com>
> wrote:
>
>> Hi Sergio,
>>
>>
>> Thank your for clarification. I am using dpdk-2.2.0. Here goes our
>> configuration on VM. Can you please have a look into the same and let me
>> know if anything is wrong. Will try to run the autotest, ipsec-segw on host
>> as per your suggestion.
>>
>>
>> [root at vpn-s ~]# uname -r
>>
>> 3.12.9-301.fc20.x86_64
>>
>> [root at vpn-s ~]#
>>
>>
>> [root at vpn-s ~]# cat /etc/redhat-release
>>
>> Fedora release 20 (Heisenbug)
>>
>> [root at vpn-s ~]#
>>
>>
>>
>>
>> [root at vpn-s ~]# cat /proc/cmdline
>>
>> BOOT_IMAGE=/boot/vmlinuz-3.12.9-301.fc20.x86_64
>> root=UUID=ab47cbc9-68ee-403c-96a5-184e68238e65 ro console=ttyS0,115200n8
>> iommu=pt intel_iommu=on
>>
>> [root at vpn-s ~]#
>>
>>
>> [root at vpn-s QAT]# grep -i "iommu.*enabled" /var/log/messages
>>
>> Jun 22 07:31:17 vpn-s kernel: [    0.000000] Intel-IOMMU: enabled
>>
>>
>> [root at vpn-s QAT]#
>>
>> [root at vpn-s dpdk-2.2.0]# lspci -nn | grep 0443
>>
>> 00:06.0 Co-processor [0b40]: Intel Corporation Device [8086:0443]
>>
>> [root at vpn-s dpdk-2.2.0]#
>>
>>
>> Installed Intel qat driver installation on VM (downloaded from
>> https://01.org/packet-processing/intel%C2%AE-quickassist-technology-drivers-and-patches)
>> and followed the
>>
>> https://01.org/sites/default/files/page/330750-005_qat_gsg.pdf guide.
>> Note, used the following installation options i.e., ./installer.sh install
>> QAT1.6 guest. After this , it detects QAT VF.
>>
>>
>> [root at vpn-s QAT]# service qat_service status
>>
>> There is 1 acceleration device(s) in the system:
>>
>>  icp_dev0 - type=dh895xcc, inst_id=0, node_id=0,  bdf=00:06:0, #accel=1,
>> #engines=1, state=up
>>
>> [root at vpn-s QAT]#
>>
>>
>> [root at vpn-s QAT]# lsmod | grep qa
>>
>> icp_qa_al_vf          813442  1
>>
>> [root at vpn-s QAT]#
>>
>>
>>
>> The below are DPDK configuration options in
>> config/defconfig_x86_64-native-linuxapp-gcc.
>>
>>
>> include "common_linuxapp"
>>
>>
>>
>> CONFIG_RTE_MACHINE="native"
>>
>> CONFIG_RTE_ARCH="x86_64"
>>
>> CONFIG_RTE_ARCH_X86_64=y
>>
>> CONFIG_RTE_ARCH_64=y
>>
>> CONFIG_RTE_TOOLCHAIN="gcc"
>>
>> CONFIG_RTE_TOOLCHAIN_GCC=y
>>
>> CONFIG_RTE_LIBRTE_PMD_AESNI_MB=y
>>
>> CONFIG_RTE_LIBRTE_CRYPTODEV_DEBUG=y
>>
>> CONFIG_RTE_LIBRTE_VIRTIO_PMD=y
>>
>> CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_INIT=y
>>
>> CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_DRIVER=y
>>
>> CONFIG_RTE_LIBRTE_PMD_QAT=y
>>
>> CONFIG_RTE_QAT_PMD_MAX_NB_SESSIONS=128
>>
>> CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_INIT=y
>>
>> CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_TX=y
>>
>> CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_RX=y
>>
>> CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_DRIVER=y
>>
>>
>>
>> The below is my setup.sh
>>
>>
>> export RTE_SDK=/opt/dpdk-2.2.0
>>
>> export RTE_TARGET=x86_64-native-linuxapp-gcc
>>
>> export RTE_ARCH=x86_64
>>
>> export AESNI_MULTI_BUFFER_LIB_PATH=/opt/code
>>
>> make config T=x86_64-native-linuxapp-gcc
>>
>> sed -ri 's,(PMD_PCAP=).*,\1y,' build/.config
>>
>> make install T=x86_64-native-linuxapp-gcc
>>
>> mkdir -p /mnt/huge
>>
>> mount -t hugetlbfs nodev /mnt/huge
>>
>> echo 2048 >
>> /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
>>
>> sudo modprobe uio_pci_generic
>>
>> sudo modprobe uio
>>
>> rmmod igb_uio
>>
>> insmod
>> ./x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/igb_uio/igb_uio.ko
>>
>> echo -n "0000:00:06.0" > /sys/bus/pci/devices/0000\:00\:06.0/driver/unbind
>>
>> echo "8086 0443" > /sys/bus/pci/drivers/igb_uio/new_id
>>
>> lspci -vvd:443
>>
>> ./tools/dpdk_nic_bind.py --bind=uio_pci_generic eth1
>>
>> ./tools/dpdk_nic_bind.py --bind=uio_pci_generic 00:03.0
>>
>> ./tools/dpdk_nic_bind.py -s
>>
>>
>> Also modified the cryptodevs_init() ( examples/ipsec-secgw/ipsec-secgw.c)
>> to rte_cryptodev_start(cdev_id); after rte_cryptodev_queue_pair_setup().
>> The purpose of calling this to start the transmit and the receive units
>> of the QAT device. It returns success. But still getting the same issue
>> i.e. rte_cryptodev_dequeue_burst () does not return crypto packet. Note
>> that, when I call rte_cryptodev_queue_pair_setup () with 3rd argument as
>> NULL (in which case default configuration will be used) , it crashes. Is
>> it a known issue?
>>
>>
>>
>>
>>
>> Regards,
>>
>> Chinmaya
>>
>> On Tue, Jun 21, 2016 at 9:10 PM, Sergio Gonzalez Monroy <
>> sergio.gonzalez.monroy at intel.com> wrote:
>>
>>> On 21/06/2016 08:26, Chinmaya Dwibedy wrote:
>>>
>>>>
>>>> Hi Sergio,
>>>>
>>>>
>>>> As suggested by you, run ‘app/test' application and then run
>>>> 'cryptodev_qat_autotest' . Found the same issue i.e., It halts in
>>>> rte_cryptodev_dequeue_burst () (while loop) in process_crypto_request ()
>>>> function (app/test/test_cryptodev.c).
>>>>
>>>> Debugged the qat_crypto_pkt_rx_burst() (in
>>>> drivers/crypto/qat/qat_crypto.c) and found that , the value of resp_msg is
>>>> 0x7F7F7F7F (i.e., ADF_RING_EMPTY_SIG) . Thus it is not getting crypto
>>>> packet (i.e., rte_mbuf) from the RX queue.  But I find that, in
>>>> qat_crypto_pkt_tx_burst(), the  qat_alg_write_mbuf_entry() is being called.
>>>> The qat_alg_write_mbuf_entry () prints qat_req  using rte_hexdump () and
>>>> returns zero.
>>>>
>>>>
>>>> One question, the base_addr + tail of tx_q (in
>>>> qat_crypto_pkt_tx_burst()) should be same as base_addr + head (in
>>>> qat_crypto_pkt_rx_burst()). Right ? Please feel free to correct me if I am
>>>> wrong.
>>>>
>>>>
>>>> Also can you please provide some pointers for further debugging? Would
>>>> it be configuration issue? Note that, I am using VF in VM.
>>>>
>>>>
>>> In theory  you should be able to run ipsec-secgw sample app with QAT VF
>>> as long as the setup is right.
>>> That the autotest fails would likely point to some possible issues with
>>> the setup.
>>>
>>> Have you tried to run the autotest in the host?
>>>
>>> Sergio
>>>
>>>
>>>> Thank you once again for your prompt response and great support.
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Chinmaya
>>>>
>>>>
>>>> On Mon, Jun 20, 2016 at 3:04 PM, Sergio Gonzalez Monroy <
>>>> sergio.gonzalez.monroy at intel.com <mailto:
>>>> sergio.gonzalez.monroy at intel.com>> wrote:
>>>>
>>>>     On 20/06/2016 10:08, Chinmaya Dwibedy wrote:
>>>>
>>>>
>>>>         Hi Sergio,
>>>>
>>>>
>>>>         Agreed.  We might not dequeue the same amount of crypto ops we
>>>>         just previously enqueued, it's asynchronous. But in this case,
>>>>         I have sent just one UDP packet. So there will be one crypto
>>>>         ops. Right?  Also I put a sleep (50) after the
>>>>         rte_crypto_enqueue_burst() function in ipsec_processing()
>>>>         (ipsec.c) , so as to allow more time  ( for QAT device) for
>>>>         processing. Still getting the same result i.e., the
>>>>         rte_crypto_dequeue_burst () functionreturns zero.
>>>>
>>>>
>>>>         In case of S/W crypto device (i.e., AESNI),   the VM gets
>>>>         inbound UDP packets on Port 1/eth1, encapsulates (after
>>>>         consulting its SPD) in an IPsec ESP packet and sends to its
>>>>         peer through Port 0/eth0 interface.
>>>>
>>>>
>>>>         Yes, the security policy, security association and Routing
>>>>         entries/configurations are exactly same. Please feel free to
>>>>         let me know if you need additional information.
>>>>
>>>>
>>>>     Could you try to run 'app/test' application then run
>>>>     'cryptodev_qat_autotest' ? That is a functional test for cryptodev
>>>>     QAT PMD.
>>>>
>>>>     Sergio
>>>>
>>>>
>>>>
>>>
>>
>