[dpdk-users] Issue with Physical Function passthrough on VM Guest (Fedora 20) using qatmux.l.2.6.0-60

Chinmaya Dwibedy ckdwibedy at gmail.com
Fri Jun 10 09:16:37 CEST 2016
Previous message: [dpdk-users] pktgen send a burst, then wait
Next message: [dpdk-users] Issue with Physical Function passthrough on VM Guest (Fedora 20) using qatmux.l.2.6.0-60
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Hi,


I have assigned Intel QAT device to VMs as a full device (PF - physical
function, PCI passthrough). Below is the output of lspci in host (Cent OS
7.2).


[root at localhost nova(keystone_admin)]#  lspci -nn | grep 0435

83:00.0 Co-processor [0b40]: Intel Corporation DH895XCC Series QAT [
8086:0435]

88:00.0 Co-processor [0b40]: Intel Corporation DH895XCC Series QAT [
8086:0435]

[root at localhost nova(keystone_admin)]#

[root at localhost ~]# lspci -vnd 8086:0435

83:00.0 0b40: 8086:0435

        Subsystem: 8086:35c5

        Physical Slot: 0-1

        Flags: bus master, fast devsel, latency 0, IRQ 35

        Memory at 387fffd00000 (64-bit, prefetchable) [size=512K]

        Memory at c8200000 (64-bit, non-prefetchable) [size=256K]

        Memory at c8240000 (64-bit, non-prefetchable) [size=256K]

        Capabilities: [b0] MSI: Enable- Count=1/1 Maskable+ 64bit+

        Capabilities: [60] MSI-X: Enable- Count=33 Masked-

        Capabilities: [6c] Power Management version 3

        Capabilities: [74] Express Endpoint, MSI 00

        Capabilities: [100] Advanced Error Reporting

        Capabilities: [138] Alternative Routing-ID Interpretation (ARI)

        Capabilities: [140] Single Root I/O Virtualization (SR-IOV)

        Kernel driver in use: vfio-pci



88:00.0 0b40: 8086:0435

        Subsystem: 8086:35c5

        Physical Slot: 0-2

        Flags: fast devsel, IRQ 39

        Memory at 387fffe00000 (64-bit, prefetchable) [size=512K]

        Memory at c8000000 (64-bit, non-prefetchable) [size=256K]

        Memory at c8040000 (64-bit, non-prefetchable) [size=256K]

        Capabilities: [b0] MSI: Enable- Count=1/1 Maskable+ 64bit+

        Capabilities: [60] MSI-X: Enable- Count=33 Masked-

        Capabilities: [6c] Power Management version 3

        Capabilities: [74] Express Endpoint, MSI 00

        Capabilities: [100] Advanced Error Reporting

        Capabilities: [138] Alternative Routing-ID Interpretation (ARI)

        Capabilities: [140] Single Root I/O Virtualization (SR-IOV)

        Kernel driver in use: vfio-pci



[root at localhost ~]#


In VM,  the Intel QAT device shows up in lspci as "Intel Corporation Coleto
Creek PCIe Endpoint".


[root at test1 QAT]# lspci -nn | grep 0435

00:05.0 Co-processor [0b40]: Intel Corporation Coleto Creek PCIe Endpoint
[8086:0435]

[root at test1 QAT]#


Afterward I installed Intel® QuickAssist Technology Software i.e.,
qatmux.l.2.6.0-60.tgz ( downloaded at
https://01.org/packet-processing/intel%C2%AE-quickassist-technology-drivers-and-patches)
on the Guest VM . Here are the details of VM.

1)      Fedora release 20 (Heisenbug).

2)      Linux Kernel version: 3.12.9-301.fc20.x86_64.

3)      BOOT_IMAGE=/boot/vmlinuz-3.12.9-301.fc20.x86_64
root=UUID=ab47cbc9-68ee-403c-96a5-184e68238e65 ro console=ttyS0,115200n8
intel_iommu=off



*Notes: *

1)      We are not using SR-IOV and are instead passing through a PF for
acceleration services on  guest only. Hence we have not installed the
Intel® QuickAssist Technology Software package on the host.

2)      Found an Intel QAT driver with this kernel . Upon doing #lsmod |
grep qa, showed icp_qa_al_vf. Thus did rmmod icp_qa_al_vf and service
qat_service shutdown. This was done before building Intel QuickAssist
Technology Software package.



For passthrough of PF, opted the below Install Acceleration option

a)      export ICP_SRIOV=0

b)      export ICP_WITHOUT_IOMMU=1

c)      ./installer.sh build_accel QAT1.6

d)      ./installer.sh install QAT1.6 guest

e)      insmod /root/QAT /QAT1.6/build/icp_qa_al.ko



When I issue the below command to start the acceleration software, it fails
to start the driver.


[root at test1 ~]# service qat_service start

Processing file: /etc/dh895xcc_qa_dev0.conf

ADF_CONFIG_CTL err: adf_set_bank_core_affinity: Invalid core affinity
settings for accelerator 0 bank 8. Max cpu number is 3. Defaulting to 0

ADF_CONFIG_CTL err: adf_set_bank_core_affinity: Invalid core affinity
settings for accelerator 0 bank 9. Max cpu number is 3. Defaulting to 0

Parity err reporting is disabled.

Error: failed to start device 0

There is 1 acceleration device(s) in the system:

 icp_dev0 - type=dh895xcc, inst_id=0, node_id=0,  bdf=00:05:0, #accel=6,
#engines=12, state=down

[root at test1 ~]#



Here are the kernel logs



[root at test1 ~]# dmesg -c

[ 3104.979666] Reading config file.

[ 3104.980250] Starting acceleration device icp_dev0.

[ 3104.980731] Resetting device icp_dev0

[ 3105.126920] qat_1_6_adf 0000:00:05.0: irq 45 for MSI/MSI-X

[ 3105.126939] qat_1_6_adf 0000:00:05.0: irq 46 for MSI/MSI-X

[ 3105.126956] qat_1_6_adf 0000:00:05.0: irq 47 for MSI/MSI-X

[ 3105.126973] qat_1_6_adf 0000:00:05.0: irq 48 for MSI/MSI-X

[ 3105.126989] qat_1_6_adf 0000:00:05.0: irq 49 for MSI/MSI-X

[ 3105.127029] qat_1_6_adf 0000:00:05.0: irq 50 for MSI/MSI-X

[ 3105.127048] qat_1_6_adf 0000:00:05.0: irq 51 for MSI/MSI-X

[ 3105.127065] qat_1_6_adf 0000:00:05.0: irq 52 for MSI/MSI-X

[ 3105.127082] qat_1_6_adf 0000:00:05.0: irq 53 for MSI/MSI-X

[ 3105.127098] qat_1_6_adf 0000:00:05.0: irq 54 for MSI/MSI-X

[ 3105.127115] qat_1_6_adf 0000:00:05.0: irq 55 for MSI/MSI-X

[ 3105.127131] qat_1_6_adf 0000:00:05.0: irq 56 for MSI/MSI-X

[ 3105.127148] qat_1_6_adf 0000:00:05.0: irq 57 for MSI/MSI-X

[ 3105.127164] qat_1_6_adf 0000:00:05.0: irq 58 for MSI/MSI-X

[ 3105.127181] qat_1_6_adf 0000:00:05.0: irq 59 for MSI/MSI-X

[ 3105.127198] qat_1_6_adf 0000:00:05.0: irq 60 for MSI/MSI-X

[ 3105.127215] qat_1_6_adf 0000:00:05.0: irq 61 for MSI/MSI-X

[ 3105.127231] qat_1_6_adf 0000:00:05.0: irq 62 for MSI/MSI-X

[ 3105.127248] qat_1_6_adf 0000:00:05.0: irq 63 for MSI/MSI-X

[ 3105.127265] qat_1_6_adf 0000:00:05.0: irq 64 for MSI/MSI-X

[ 3105.127282] qat_1_6_adf 0000:00:05.0: irq 65 for MSI/MSI-X

[ 3105.127298] qat_1_6_adf 0000:00:05.0: irq 66 for MSI/MSI-X

[ 3105.127315] qat_1_6_adf 0000:00:05.0: irq 67 for MSI/MSI-X

[ 3105.127331] qat_1_6_adf 0000:00:05.0: irq 68 for MSI/MSI-X

[ 3105.127348] qat_1_6_adf 0000:00:05.0: irq 69 for MSI/MSI-X

[ 3105.127364] qat_1_6_adf 0000:00:05.0: irq 70 for MSI/MSI-X

[ 3105.127380] qat_1_6_adf 0000:00:05.0: irq 71 for MSI/MSI-X

[ 3105.127397] qat_1_6_adf 0000:00:05.0: irq 72 for MSI/MSI-X

[ 3105.127413] qat_1_6_adf 0000:00:05.0: irq 73 for MSI/MSI-X

[ 3105.127429] qat_1_6_adf 0000:00:05.0: irq 74 for MSI/MSI-X

[ 3105.127446] qat_1_6_adf 0000:00:05.0: irq 75 for MSI/MSI-X

[ 3105.127462] qat_1_6_adf 0000:00:05.0: irq 76 for MSI/MSI-X

[ 3105.127487] qat_1_6_adf 0000:00:05.0: irq 77 for MSI/MSI-X

[ 3105.157973] icp_qa_al err: adf_subsystemInit: Invalid SRIOV_Enabled
value, driver is built with SRIOV.

[ 3105.158891] icp_qa_al err: adf_do_init: adf_subsystemInit error,
stopping and shutting down

[ 3106.162962] icp_qa_al err: adf_init_all: failed to initialize accelerator

[ 3106.163670] icp_qa_al err: adf_init_devices: failed init accel 0

[ 3106.164256] icp_qa_al err: adf_ctl_ioctl_dev_start: failed call to
adf_init_devices

[root at test1 ~]#



Can anyone please let me know what might be the issue and its remedy ?
please feel free to correct me if I am wrong.  If it is not the right forum
, please let me know the correct one. Thank you in advance for time and
support.



Regards,

Chinmaya
Previous message: [dpdk-users] pktgen send a burst, then wait
Next message: [dpdk-users] Issue with Physical Function passthrough on VM Guest (Fedora 20) using qatmux.l.2.6.0-60
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
More information about the users mailing list