[dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue

[Burakov, Anatoly]

On 01-Feb-18 7:26 PM, Ravi Kerur wrote:
> 
> 
> On Thu, Feb 1, 2018 at 2:10 AM, Burakov, Anatoly 
> <anatoly.burakov at intel.com <mailto:anatoly.burakov at intel.com>> wrote:
> 
>     On 31-Jan-18 9:51 PM, Ravi Kerur wrote:
> 
> 
>         Hi Anatoly,
> 
>         Thanks. I am following wiki link below which uses vIOMMU with
>         DPDK as a use-case and instantiate VM as specified with Q35
>         chipset in Qemu.
> 
>         https://wiki.qemu.org/Features/VT-d
>         <https://wiki.qemu.org/Features/VT-d>
> 
>         Qemu-version is 2.11
>         Host kernel 4.9
>         Guest kernel 4.4
> 
>         I can only guess that guest kernel needs an upgrade in my setup
>         to work correctly, if versions on my setup rings a bell on not
>         having support kindly let me know.
> 
>         When 'modprobe vfio enable_unsafe_noiommu_node=Y' is executed on
>         guest I get following error
>         ...
>         vfio: unknown parameter 'enable_unsafe_noiommu_node' ignored
>         ...
> 
>         in guest.
> 
>         Thanks.
> 
> 
>     AFAIK kernel 4.4 should have noiommu mode - it was introduced in
>     3.1x days. However, in order for that to work, kernel also has to be
>     built with this mode enabled. My guess is, whoever is the supplier
>     of your kernel, did not do that. You should double-check the kernel
>     configuration of your distribution.
> 
>     However, if you have vIOMMU in QEMU, you shouldn't need noiommu mode
>     - "regular" vfio should work fine. noiommu mode should only be
>     needed if you know you don't have IOMMU enabled in your kernel, and
>     even if you can't enable it, you can still use igb_uio.
> 
> Hi Anatoly,
> 
> Do you suggest I take this discussion to kvm/qemu mailing list as I am 
> not sure which component has the issue? I check dmesg for BIOS physical 
> memory map and address reported as fault by DMAR is reported by BIOS as 
> usable on both host and vm.
> 
> [ 4539.597737] DMAR: [DMA Read] Request device [04:10.0] fault addr 
> *33a128000 *[fault reason 06] PTE Read access is not set
> 
> dmesg | grep BIOS
> [    0.000000] e820: BIOS-provided physical RAM map:
> [    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009afff] usable
> [    0.000000] BIOS-e820: [mem 0x000000000009b000-0x000000000009ffff] 
> reserved
> [    0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] 
> reserved
> [    0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007938afff] usable
> [    0.000000] BIOS-e820: [mem 0x000000007938b000-0x000000007994bfff] 
> reserved
> [    0.000000] BIOS-e820: [mem 0x000000007994c000-0x000000007999cfff] 
> ACPI data
> [    0.000000] BIOS-e820: [mem 0x000000007999d000-0x0000000079f7dfff] 
> ACPI NVS
> [    0.000000] BIOS-e820: [mem 0x0000000079f7e000-0x000000007bd37fff] 
> reserved
> [    0.000000] BIOS-e820: [mem 0x000000007bd38000-0x000000007bd38fff] usable
> [    0.000000] BIOS-e820: [mem 0x000000007bd39000-0x000000007bdbefff] 
> reserved
> [    0.000000] BIOS-e820: [mem 0x000000007bdbf000-0x000000007bffffff] usable
> [    0.000000] BIOS-e820: [mem 0x000000007c000000-0x000000008fffffff] 
> reserved
> [    0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed44fff] 
> reserved
> [    0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] 
> reserved
> [*    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000407fffffff] 
> usable*
> *
> *
> Kindly let me know your inputs.
> 
> Thanks.
> 
> 
>     -- 
>     Thanks,
>     Anatoly
> 
> 

The "PTE Read not set" error usually indicates that you are trying to 
use a non-IOMMU method when you have IOMMU enabled (i.e. trying to use 
igb_uio when IOMMU is on). That, to me, indicates that you do have IOMMU 
emulation enabled.

I would go about it this way.

First, i'd ensure that your VM has IOMMU emulation enabled and working. 
You have mentioned that your QEMU version should have IOMMU emulation, 
so let's assume that's the case.

I am not sure of the exact command-line needed to activate the vIOMMU 
emulation, but assuming your VM emulates an Intel processor, your kernel 
command-line should have "iommu=on intel_iommu=on" in it. Check 
/etc/default/grub for GRUB_CMDLINE_LINUX_DEFAULT value, and if the above 
values are not in there, add the above changes, do "update-grub" and 
reboot your VM.

If it already did have the necessary kernel configuration, do "dmesg | 
grep IOMMU" and look for "IOMMU Enabled". That should tell you that 
IOMMU is enabled and working in the kernel.

After that, you can modprobe vfio and vfio-pci, bind NICs to it, and it 
should be working. Please bear in mind that all of that is how i 
would've gone about it if i had similar problems on baremetal, but i'm 
hoping all of it is applicable to VM's. So, either disable IOMMU and use 
igb_uio, or enable IOMMU and use VFIO. Both will work.

-- 
Thanks,
Anatoly