[dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue

Ravi Kerur rkerur at gmail.com
Mon Jan 29 23:35:16 CET 2018


Hi Burakov,

When using vfio-pci on host both VF and PF interfaces works fine with dpdk
i.e. I don't see DMAR fault messages anymore. However, when I attach a VF
interface to a VM and start DPDK with vfio-pci inside VM I still see DMAR
fault messages on host. Both host and VM are booted with 'intel-iommu=on'
on GRUB. Ping from VM with DPDK/vfio-pci doesn't work (I think it's
expected because of DMAR faults), however, when VF interface uses ixgbevf
driver ping works.

Following are some details

/*****************On VM***************/
dpdk-devbind -s

Network devices using DPDK-compatible driver
============================================
0000:00:07.0 '82599 Ethernet Controller Virtual Function' drv=vfio-pci
unused=ixgbevf

Network devices using kernel driver
===================================
0000:03:00.0 'Device 1041' if=eth0 drv=virtio-pci unused=vfio-pci *Active*
0000:04:00.0 'Device 1041' if=eth1 drv=virtio-pci unused=vfio-pci
0000:05:00.0 'Device 1041' if=eth2 drv=virtio-pci unused=vfio-pci

Other network devices
=====================
<none>

Crypto devices using DPDK-compatible driver
===========================================
<none>

Crypto devices using kernel driver
==================================
<none>

Other crypto devices
====================
<none>


00:07.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller
Virtual Function (rev 01)
        Subsystem: Intel Corporation 82599 Ethernet Controller Virtual
Function
        Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
        Region 0: Memory at fda00000 (64-bit, prefetchable) [size=16K]
        Region 3: Memory at fda04000 (64-bit, prefetchable) [size=16K]
        Capabilities: [70] MSI-X: Enable+ Count=3 Masked-
                Vector table: BAR=3 offset=00000000
                PBA: BAR=3 offset=00002000
        Capabilities: [a0] Express (v1) Root Complex Integrated Endpoint,
MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0
                        ExtTag- RBE-
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
TransPend-
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
                AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap-
ChkEn-
        Kernel driver in use: vfio-pci
        Kernel modules: ixgbevf

/***************on Host*************/
dmesg | grep DMAR
...
[  978.268143] DMAR: DRHD: handling fault status reg 2
[  978.268147] DMAR: [DMA Read] *Request device [04:10.0]* fault addr
33a128000 [fault reason 06] PTE Read access is not set
[ 1286.677726] DMAR: DRHD: handling fault status reg 102
[ 1286.677730] DMAR: [DMA Read] Request device [04:10.0] fault addr
fb663000 [fault reason 06] PTE Read access is not set
[ 1676.436145] DMAR: DRHD: handling fault status reg 202
[ 1676.436149] DMAR: [DMA Read] Request device [04:10.0] fault addr
33a128000 [fault reason 06] PTE Read access is not set
[ 1734.433649] DMAR: DRHD: handling fault status reg 302
[ 1734.433652] DMAR: [DMA Read] Request device [04:10.0] fault addr
33a128000 [fault reason 06] PTE Read access is not set
[ 2324.428938] DMAR: DRHD: handling fault status reg 402
[ 2324.428942] DMAR: [DMA Read] Request device [04:10.0] fault addr
7770c000 [fault reason 06] PTE Read access is not set
[ 2388.553640] DMAR: DRHD: handling fault status reg 502
[ 2388.553643] DMAR: [DMA Read] *Request device [04:10.0]* fault addr
33a128000 [fault reason 06] PTE Read access is not set


VM is started with

qemu-system-x86_64 -enable-kvm -M q35,accel=kvm,kernel-irqchip=split
-object iothread,id=iothread0 -device
intel-iommu,intremap=on,device-iotlb=on,caching-mode=on -cpu host
-daemonize -m 16G -smp 14 -uuid 0fc91c66-f0b1-11e7-acf4-525400123456 -name
212748-sriov-ravi-smac-alpha-SMAC10 -device ioh3420,id=root.1,chassis=1
-device ioh3420,id=root.2,chassis=2 -netdev
tap,vhost=on,queues=2,ifname=vn-vn2_1_,downscript=no,id=vn-vn2_1_,script=no
-device ioh3420,id=root.3,chassis=3 -device
virtio-net-pci,netdev=vn-vn2_1_,bus=root.3,ats=on,mq=on,vectors=6,mac=DE:AD:02:88:10:37,id=vn-vn2_1__dev
-netdev
tap,vhost=on,queues=2,ifname=vn-vn92_1_,downscript=no,id=vn-vn92_1_,script=no
-device ioh3420,id=root.4,chassis=4 -device
virtio-net-pci,mac=DE:AD:02:88:10:38,netdev=vn-vn92_1_,bus=root.4,ats=on,mq=on,vectors=6,id=vn-vn92_1__dev
-netdev
tap,vhost=on,queues=2,ifname=vn-vn93_1_,downscript=no,id=vn-vn93_1_,script=no
-device ioh3420,id=root.5,chassis=5 -device
virtio-net-pci,mac=DE:AD:02:88:10:39,netdev=vn-vn93_1_,bus=root.5,ats=on,mq=on,vectors=6,id=vn-vn93_1__dev
-vnc :16,websocket=15916 -qmp tcp:127.0.0.1:12001,server,nowait -chardev
socket,id=charmonitor,path=/tmp/mon.12001,server,nowait -mon
chardev=charmonitor,id=monitor -cdrom
/var/venom/cloud_init/0fc91c66-f0b1-11e7-acf4-525400123456.iso -*device
vfio-pci,host=04:10.0* -drive
file=/var/venom/instance_repo/test.img,if=none,id=drive-virtio-disk0,format=raw,aio=native,cache=none
-balloon none -device
virtio-blk-pci,scsi=off,iothread=iothread0,drive=drive-virtio-disk0,id=virtio-disk0,bus=root.1,ats=on,bootindex=1

Thanks.


On Thu, Jan 25, 2018 at 2:49 AM, Burakov, Anatoly <anatoly.burakov at intel.com
> wrote:

> On 24-Jan-18 7:13 PM, Ravi Kerur wrote:
>
>> Hi Burakov, Thank you. I will try with vfio-pci driver. I am assuming it
>> will work for both PF and VF interfaces since I am using both in my setup?
>>
>> Thanks.
>>
>
> Yes, it should work for both PF and VF devices.
>
>
>> On Wed, Jan 24, 2018 at 2:31 AM, Burakov, Anatoly <
>> anatoly.burakov at intel.com <mailto:anatoly.burakov at intel.com>> wrote:
>>
>>     On 23-Jan-18 5:25 PM, Ravi Kerur wrote:
>>
>>         Hi,
>>
>>         I am running into an issue when DPDK is started with iommu on
>>         via GRUB
>>         command. Problem is not seen with regular kernel driver, error
>>         messages
>>         show when DPDK is started and happens for both PF and VF
>> interfaces.
>>
>>         I am using DPDK 17.05 so the patch proposed in the following link
>> is
>>         available
>>         http://dpdk.org/ml/archives/dev/2017-February/057048.html
>>         <http://dpdk.org/ml/archives/dev/2017-February/057048.html>
>>
>>         Workaround is to use "iommu=pt" but I want iommu enabled in my
>>         setup. I
>>         checked BIOS for reserved memory(DMA RMRR for IXGBE) didn't get
>>         any details
>>         on it.
>>
>>         Kindly let me know how to resolve this issue.
>>
>>         Following are the details
>>
>>         (1) Linux kernel 4.9
>>         (2) DPDK 17.05
>>
>>         (3) IXGBE details
>>         ethtool -i enp4s0f0  (PF driver)
>>         driver: ixgbe
>>         version: 5.3.3
>>         firmware-version: 0x800007b8, 1.1018.0
>>         bus-info: 0000:04:00.0
>>         supports-statistics: yes
>>         supports-test: yes
>>         supports-eeprom-access: yes
>>         supports-register-dump: yes
>>         supports-priv-flags: yes
>>
>>         ethtool -i enp4s16f2 (VF driver)
>>         driver: ixgbevf
>>         version: 4.3.2
>>         firmware-version:
>>         bus-info: 0000:04:10.2
>>         supports-statistics: yes
>>         supports-test: yes
>>         supports-eeprom-access: no
>>         supports-register-dump: yes
>>         supports-priv-flags: no
>>
>>         Bus info          Device       Class          Description
>>         =========================================================
>>         pci at 0000:01:00.0  ens11f0      network        82599ES 10-Gigabit
>>         SFI/SFP+
>>         Network Connection
>>         pci at 0000:01:00.1  ens11f1      network        82599ES 10-Gigabit
>>         SFI/SFP+
>>         Network Connection
>>         pci at 0000:04:00.0  enp4s0f0     network        82599ES 10-Gigabit
>>         SFI/SFP+
>>         Network Connection
>>         pci at 0000:04:00.1  enp4s0f1     network        82599ES 10-Gigabit
>>         SFI/SFP+
>>         Network Connection
>>         pci at 0000:04:10.0  enp4s16      network        Illegal Vendor ID
>>         pci at 0000:04:10.2  enp4s16f2    network        Illegal Vendor ID
>>
>>         (4) DPDK bind interfaces
>>
>>         # dpdk-devbind -s
>>
>>         Network devices using DPDK-compatible driver
>>         ============================================
>>         0000:01:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
>>         drv=igb_uio unused=vfio-pci
>>         0000:04:10.2 '82599 Ethernet Controller Virtual Function 10ed'
>>         drv=igb_uio
>>         unused=vfio-pci
>>
>>         Network devices using kernel driver
>>         ===================================
>>         0000:01:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
>>         if=ens11f1 drv=ixgbe unused=igb_uio,vfio-pci
>>         0000:04:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
>>         if=enp4s0f0 drv=ixgbe unused=igb_uio,vfio-pci
>>         0000:04:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
>>         if=enp4s0f1 drv=ixgbe unused=igb_uio,vfio-pci
>>         0000:04:10.0 '82599 Ethernet Controller Virtual Function 10ed'
>>         if=enp4s16
>>         drv=ixgbevf unused=igb_uio,vfio-pci
>>         0000:06:00.0 'I210 Gigabit Network Connection 1533' if=eno1
>> drv=igb
>>         unused=igb_uio,vfio-pci *Active*
>>
>>         Other Network devices
>>         =====================
>>         <none>
>>
>>         ...
>>
>>         (5) Kernel dmesg
>>
>>         # dmesg | grep -e DMAR
>>         [    0.000000] ACPI: DMAR 0x000000007999BAD0 0000E0 (v01 ALASKA
>>         A M I
>>         00000001 INTL 20091013)
>>         [    0.000000] DMAR: IOMMU enabled
>>         [    0.518747] DMAR: Host address width 46
>>         [    0.526616] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
>>         [    0.537447] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap
>>         d2078c106f0466 ecap f020df
>>         [    0.553620] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1
>>         [    0.564445] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap
>>         d2078c106f0466 ecap f020df
>>         [    0.580611] DMAR: RMRR base: 0x0000007bbc6000 end:
>>         0x0000007bbd4fff
>>         [    0.593344] DMAR: ATSR flags: 0x0
>>         [    0.600178] DMAR: RHSA base: 0x000000c7ffc000 proximity
>>         domain: 0x0
>>         [    0.612905] DMAR: RHSA base: 0x000000fbffc000 proximity
>>         domain: 0x1
>>         [    0.625632] DMAR-IR: IOAPIC id 3 under DRHD base  0xfbffc000
>>         IOMMU 0
>>         [    0.638522] DMAR-IR: IOAPIC id 1 under DRHD base  0xc7ffc000
>>         IOMMU 1
>>         [    0.651426] DMAR-IR: IOAPIC id 2 under DRHD base  0xc7ffc000
>>         IOMMU 1
>>         [    0.664324] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000
>>         [    0.675326] DMAR-IR: Queued invalidation will be enabled to
>>         support
>>         x2apic and Intr-remapping.
>>         [    0.693805] DMAR-IR: Enabled IRQ remapping in x2apic mode
>>         [    9.395170] DMAR: dmar1: Using Queued invalidation
>>         [    9.405011] DMAR: Setting RMRR:
>>         [    9.412006] DMAR: Setting identity map for device 0000:00:1d.0
>>         [0x7bbc6000 - 0x7bbd4fff]
>>         [    9.428569] DMAR: Prepare 0-16MiB unity mapping for LPC
>>         [    9.439712] DMAR: Setting identity map for device
>>         0000:00:1f.0 [0x0 -
>>         0xffffff]
>>         [    9.454684] DMAR: Intel(R) Virtualization Technology for
>>         Directed I/O
>>         [  287.023068] DMAR: DRHD: handling fault status reg 2
>>         [  287.023073] DMAR: [DMA Read] Request device [01:00.0] fault
>> addr
>>         18a260a000 [fault reason 06] PTE Read access is not set
>>         [  287.023180] DMAR: DRHD: handling fault status reg 102
>>         [  287.023183] DMAR: [DMA Read] Request device [01:00.0] fault
>> addr
>>         18a3010000 [fault reason 06] PTE Read access is not set
>>         [  287.038250] DMAR: DRHD: handling fault status reg 202
>>         [  287.038252] DMAR: [DMA Read] Request device [01:00.0] fault
>> addr
>>         18a3010000 [fault reason 06] PTE Read access is not set
>>         [  288.170165] DMAR: DRHD: handling fault status reg 302
>>         [  288.170170] DMAR: [DMA Read] Request device [01:00.0] fault
>> addr
>>         1890754000 [fault reason 06] PTE Read access is not set
>>         [  288.694496] DMAR: DRHD: handling fault status reg 402
>>         [  288.694499] DMAR: [DMA Read] Request device [04:10.2] fault
>> addr
>>         189069c000 [fault reason 06] PTE Read access is not set
>>         [  289.927113] DMAR: DRHD: handling fault status reg 502
>>         [  289.927116] DMAR: [DMA Read] Request device [01:00.0] fault
>> addr
>>         1890754000 [fault reason 06] PTE Read access is not set
>>         [  290.174275] DMAR: DRHD: handling fault status reg 602
>>         [  290.174279] DMAR: [DMA Read] Request device [01:00.0] fault
>> addr
>>         1890754000 [fault reason 06] PTE Read access is not set
>>         [  292.174247] DMAR: DRHD: handling fault status reg 702
>>         [  292.174251] DMAR: [DMA Read] Request device [01:00.0] fault
>> addr
>>         1890754000 [fault reason 06] PTE Read access is not set
>>         [  294.174227] DMAR: DRHD: handling fault status reg 2
>>         [  294.174230] DMAR: [DMA Read] Request device [01:00.0] fault
>> addr
>>         1890754000 [fault reason 06] PTE Read access is not set
>>         [  296.174216] DMAR: DRHD: handling fault status reg 102
>>         [  296.174219] DMAR: [DMA Read] Request device [01:00.0] fault
>> addr
>>         1890754000 [fault reason 06] PTE Read access is not set
>>         [root at infradev-comp006.naw02.infradev.viasat.io
>>         <mailto:root at infradev-comp006.naw02.infradev.viasat.io> ~]
>>         #
>>
>>         Thanks.
>>
>>
>>     Hi Ravi,
>>
>>     The "iommu=pt" workaround applies only when you want to use igb_uio
>>     driver. VFIO driver is able to fully utilize IOMMU without the need
>>     for pass-through mode. From your log i can see that some devices are
>>     bound to igb_uio while others are bound to vfio-pci. Just bind all
>>     of the devices you want to use with DPDK to vfio-pci and these
>>     errors should go away.
>>
>>     --     Thanks,
>>     Anatoly
>>
>>
>>
>
> --
> Thanks,
> Anatoly
>


More information about the dev mailing list