[dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue

[Ravi Kerur]

On Tue, Feb 13, 2018 at 6:31 AM, Burakov, Anatoly <anatoly.burakov at intel.com
> wrote:

> On 12-Feb-18 10:00 PM, Ravi Kerur wrote:
>
>>
>>         Let me just give you what has been tested and working/nonworking
>>         scenarios. Some of your questions might get answered as well.
>>         Test bed is very simple with 2 VF's created under IXGBE PF on
>>         host with one VF interface added to ovs-bridge on host and
>>         another VF interface given to guest. Test connectivity between
>>         VF's via ping.
>>
>>         Host and guest -- Kernel 4.9
>>         Host -- Qemu 2.11.50 (tried both released 2.11 and tip of the
>>         git (2.11.50))
>>         DPDK -- 17.05.1 on host and guest
>>         Host and guest -- booted with GRUB intel_iommu=on (which enables
>>         IOMMU). Have tried with "iommu=on and intel_iommu=on" as well,
>>         but iommu=on is not needed when intel_iommu=on is set.
>>
>>         Test-scenario-1: Host -- ixgbe_vf driver, Guest ixgbe_vf driver
>>         ping works
>>         Test-scenario-2: Host -- DPDK vfio-pci driver, Guest ixgbe_vf
>>         driver ping works
>>         Test-scenario-3: Host -- DPDK vfio-pci driver, Guest DPDK
>>         vfio-pci driver, DMAR errors seen on host, ping doesn't work
>>
>>
>>     OK, that makes it clearer, thanks. Does the third scenario work in
>>     other DPDK versions?
>>
>>
>> No. Tried 16.11 same issue on guest and works fine on host.
>>
>>
>> So now we've moved from "this worked on 16.11" to "this never worked".
>
> It would be good to see output of rte_dump_physmem_layout() on both host
> and guest, and check which address triggers the DMAR error (i.e. if the
> physical address is present in mappings for either DPDK process).
>
> --
>

Earlier I was focusing only on DMAR errors and I might have said 'it
worked' when I didn't notice them on host when dpdk was started on guest.
When trying to send packets out of that interface from guest I did see DMAR
errors. I am attaching information you requested.  I have enabled
log-level=8 and files contain dpdk EAL/PMD logs as well.

Snippets below

on host, DMAR fault address from dmesg

[351576.998109] DMAR: DRHD: handling fault status reg 702
[351576.998113] DMAR: [DMA Read] Request device [04:10.0] fault addr
257617000 [fault reason 06] PTE Read access is not set

on guest (dump phys_mem_layout)

Segment 235: phys:0x257600000, len:2097152, virt:0x7fce87e00000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0
...
PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce87e0f4c0
sw_sc_ring=0x7fce87e07380 hw_ring=0x7fce87e17600 dma_addr=0x257617600
PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce89c67d40
sw_sc_ring=0x7fce89c5fc00 hw_ring=0x7fce89c6fe80 dma_addr=0x25406fe80
...

Thanks,
Ravi



> Thanks,
> Anatoly
>