[dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue

Burakov, Anatoly anatoly.burakov at intel.com
Thu Jan 25 11:49:27 CET 2018


On 24-Jan-18 7:13 PM, Ravi Kerur wrote:
> Hi Burakov, Thank you. I will try with vfio-pci driver. I am assuming it 
> will work for both PF and VF interfaces since I am using both in my setup?
> 
> Thanks.

Yes, it should work for both PF and VF devices.

> 
> On Wed, Jan 24, 2018 at 2:31 AM, Burakov, Anatoly 
> <anatoly.burakov at intel.com <mailto:anatoly.burakov at intel.com>> wrote:
> 
>     On 23-Jan-18 5:25 PM, Ravi Kerur wrote:
> 
>         Hi,
> 
>         I am running into an issue when DPDK is started with iommu on
>         via GRUB
>         command. Problem is not seen with regular kernel driver, error
>         messages
>         show when DPDK is started and happens for both PF and VF interfaces.
> 
>         I am using DPDK 17.05 so the patch proposed in the following link is
>         available
>         http://dpdk.org/ml/archives/dev/2017-February/057048.html
>         <http://dpdk.org/ml/archives/dev/2017-February/057048.html>
> 
>         Workaround is to use "iommu=pt" but I want iommu enabled in my
>         setup. I
>         checked BIOS for reserved memory(DMA RMRR for IXGBE) didn't get
>         any details
>         on it.
> 
>         Kindly let me know how to resolve this issue.
> 
>         Following are the details
> 
>         (1) Linux kernel 4.9
>         (2) DPDK 17.05
> 
>         (3) IXGBE details
>         ethtool -i enp4s0f0  (PF driver)
>         driver: ixgbe
>         version: 5.3.3
>         firmware-version: 0x800007b8, 1.1018.0
>         bus-info: 0000:04:00.0
>         supports-statistics: yes
>         supports-test: yes
>         supports-eeprom-access: yes
>         supports-register-dump: yes
>         supports-priv-flags: yes
> 
>         ethtool -i enp4s16f2 (VF driver)
>         driver: ixgbevf
>         version: 4.3.2
>         firmware-version:
>         bus-info: 0000:04:10.2
>         supports-statistics: yes
>         supports-test: yes
>         supports-eeprom-access: no
>         supports-register-dump: yes
>         supports-priv-flags: no
> 
>         Bus info          Device       Class          Description
>         =========================================================
>         pci at 0000:01:00.0  ens11f0      network        82599ES 10-Gigabit
>         SFI/SFP+
>         Network Connection
>         pci at 0000:01:00.1  ens11f1      network        82599ES 10-Gigabit
>         SFI/SFP+
>         Network Connection
>         pci at 0000:04:00.0  enp4s0f0     network        82599ES 10-Gigabit
>         SFI/SFP+
>         Network Connection
>         pci at 0000:04:00.1  enp4s0f1     network        82599ES 10-Gigabit
>         SFI/SFP+
>         Network Connection
>         pci at 0000:04:10.0  enp4s16      network        Illegal Vendor ID
>         pci at 0000:04:10.2  enp4s16f2    network        Illegal Vendor ID
> 
>         (4) DPDK bind interfaces
> 
>         # dpdk-devbind -s
> 
>         Network devices using DPDK-compatible driver
>         ============================================
>         0000:01:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
>         drv=igb_uio unused=vfio-pci
>         0000:04:10.2 '82599 Ethernet Controller Virtual Function 10ed'
>         drv=igb_uio
>         unused=vfio-pci
> 
>         Network devices using kernel driver
>         ===================================
>         0000:01:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
>         if=ens11f1 drv=ixgbe unused=igb_uio,vfio-pci
>         0000:04:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
>         if=enp4s0f0 drv=ixgbe unused=igb_uio,vfio-pci
>         0000:04:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
>         if=enp4s0f1 drv=ixgbe unused=igb_uio,vfio-pci
>         0000:04:10.0 '82599 Ethernet Controller Virtual Function 10ed'
>         if=enp4s16
>         drv=ixgbevf unused=igb_uio,vfio-pci
>         0000:06:00.0 'I210 Gigabit Network Connection 1533' if=eno1 drv=igb
>         unused=igb_uio,vfio-pci *Active*
> 
>         Other Network devices
>         =====================
>         <none>
> 
>         ...
> 
>         (5) Kernel dmesg
> 
>         # dmesg | grep -e DMAR
>         [    0.000000] ACPI: DMAR 0x000000007999BAD0 0000E0 (v01 ALASKA
>         A M I
>         00000001 INTL 20091013)
>         [    0.000000] DMAR: IOMMU enabled
>         [    0.518747] DMAR: Host address width 46
>         [    0.526616] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
>         [    0.537447] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap
>         d2078c106f0466 ecap f020df
>         [    0.553620] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1
>         [    0.564445] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap
>         d2078c106f0466 ecap f020df
>         [    0.580611] DMAR: RMRR base: 0x0000007bbc6000 end:
>         0x0000007bbd4fff
>         [    0.593344] DMAR: ATSR flags: 0x0
>         [    0.600178] DMAR: RHSA base: 0x000000c7ffc000 proximity
>         domain: 0x0
>         [    0.612905] DMAR: RHSA base: 0x000000fbffc000 proximity
>         domain: 0x1
>         [    0.625632] DMAR-IR: IOAPIC id 3 under DRHD base  0xfbffc000
>         IOMMU 0
>         [    0.638522] DMAR-IR: IOAPIC id 1 under DRHD base  0xc7ffc000
>         IOMMU 1
>         [    0.651426] DMAR-IR: IOAPIC id 2 under DRHD base  0xc7ffc000
>         IOMMU 1
>         [    0.664324] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000
>         [    0.675326] DMAR-IR: Queued invalidation will be enabled to
>         support
>         x2apic and Intr-remapping.
>         [    0.693805] DMAR-IR: Enabled IRQ remapping in x2apic mode
>         [    9.395170] DMAR: dmar1: Using Queued invalidation
>         [    9.405011] DMAR: Setting RMRR:
>         [    9.412006] DMAR: Setting identity map for device 0000:00:1d.0
>         [0x7bbc6000 - 0x7bbd4fff]
>         [    9.428569] DMAR: Prepare 0-16MiB unity mapping for LPC
>         [    9.439712] DMAR: Setting identity map for device
>         0000:00:1f.0 [0x0 -
>         0xffffff]
>         [    9.454684] DMAR: Intel(R) Virtualization Technology for
>         Directed I/O
>         [  287.023068] DMAR: DRHD: handling fault status reg 2
>         [  287.023073] DMAR: [DMA Read] Request device [01:00.0] fault addr
>         18a260a000 [fault reason 06] PTE Read access is not set
>         [  287.023180] DMAR: DRHD: handling fault status reg 102
>         [  287.023183] DMAR: [DMA Read] Request device [01:00.0] fault addr
>         18a3010000 [fault reason 06] PTE Read access is not set
>         [  287.038250] DMAR: DRHD: handling fault status reg 202
>         [  287.038252] DMAR: [DMA Read] Request device [01:00.0] fault addr
>         18a3010000 [fault reason 06] PTE Read access is not set
>         [  288.170165] DMAR: DRHD: handling fault status reg 302
>         [  288.170170] DMAR: [DMA Read] Request device [01:00.0] fault addr
>         1890754000 [fault reason 06] PTE Read access is not set
>         [  288.694496] DMAR: DRHD: handling fault status reg 402
>         [  288.694499] DMAR: [DMA Read] Request device [04:10.2] fault addr
>         189069c000 [fault reason 06] PTE Read access is not set
>         [  289.927113] DMAR: DRHD: handling fault status reg 502
>         [  289.927116] DMAR: [DMA Read] Request device [01:00.0] fault addr
>         1890754000 [fault reason 06] PTE Read access is not set
>         [  290.174275] DMAR: DRHD: handling fault status reg 602
>         [  290.174279] DMAR: [DMA Read] Request device [01:00.0] fault addr
>         1890754000 [fault reason 06] PTE Read access is not set
>         [  292.174247] DMAR: DRHD: handling fault status reg 702
>         [  292.174251] DMAR: [DMA Read] Request device [01:00.0] fault addr
>         1890754000 [fault reason 06] PTE Read access is not set
>         [  294.174227] DMAR: DRHD: handling fault status reg 2
>         [  294.174230] DMAR: [DMA Read] Request device [01:00.0] fault addr
>         1890754000 [fault reason 06] PTE Read access is not set
>         [  296.174216] DMAR: DRHD: handling fault status reg 102
>         [  296.174219] DMAR: [DMA Read] Request device [01:00.0] fault addr
>         1890754000 [fault reason 06] PTE Read access is not set
>         [root at infradev-comp006.naw02.infradev.viasat.io
>         <mailto:root at infradev-comp006.naw02.infradev.viasat.io> ~]
>         #
> 
>         Thanks.
> 
> 
>     Hi Ravi,
> 
>     The "iommu=pt" workaround applies only when you want to use igb_uio
>     driver. VFIO driver is able to fully utilize IOMMU without the need
>     for pass-through mode. From your log i can see that some devices are
>     bound to igb_uio while others are bound to vfio-pci. Just bind all
>     of the devices you want to use with DPDK to vfio-pci and these
>     errors should go away.
> 
>     -- 
>     Thanks,
>     Anatoly
> 
> 


-- 
Thanks,
Anatoly


More information about the dev mailing list