[dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue

[Ravi Kerur]

On Thu, Feb 15, 2018 at 2:28 AM, Burakov, Anatoly <anatoly.burakov at intel.com
> wrote:

> On 14-Feb-18 8:00 PM, Ravi Kerur wrote:
>
>>
>> Earlier I was focusing only on DMAR errors and I might have said 'it
>> worked' when I didn't notice them on host when dpdk was started on guest.
>> When trying to send packets out of that interface from guest I did see DMAR
>> errors. I am attaching information you requested.  I have enabled
>> log-level=8 and files contain dpdk EAL/PMD logs as well.
>>
>
> Great, now we're on the same page.
>
>
>> Snippets below
>>
>> on host, DMAR fault address from dmesg
>>
>> [351576.998109] DMAR: DRHD: handling fault status reg 702
>> [351576.998113] DMAR: [DMA Read] Request device [04:10.0] fault addr
>> 257617000 [fault reason 06] PTE Read access is not set
>>
>> on guest (dump phys_mem_layout)
>>
>> Segment 235: phys:0x257600000, len:2097152, virt:0x7fce87e00000,
>> socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0
>> ...
>> PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce87e0f4c0
>> sw_sc_ring=0x7fce87e07380 hw_ring=0x7fce87e17600 dma_addr=0x257617600
>> PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce89c67d40
>> sw_sc_ring=0x7fce89c5fc00 hw_ring=0x7fce89c6fe80 dma_addr=0x25406fe80
>> ...
>>
>>
> To me this looks like host (i.e. either QEMU or the PF driver) is trying
> to do DMA using guest-physical (and not host-physical). I'm not too
> well-versed in how QEMU works, but i'm pretty sure that's not supposed to
> happen.
>
> Is PF also bound to DPDK, or are you using native Linux ixgbe driver?
>

Thanks for your help. I cannot use PF with DPDK (vfio-pci), VF interfaces
disappear after it is bound to DPDK. If there is a way to use PF and VF
with DPDK let me know I can try it out. I am not sure how to move forward
on this, Is CPU/IXGBE PF driver playing a role? Following are the versions
I have

lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                56
On-line CPU(s) list:   0-27
Off-line CPU(s) list:  28-55
Thread(s) per core:    1
Core(s) per socket:    14
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 63
Model name:            Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz
Stepping:              2
CPU MHz:               2500.610
CPU max MHz:           3000.0000
CPU min MHz:           1200.0000
BogoMIPS:              4000.74
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              35840K
NUMA node0 CPU(s):     0-13
NUMA node1 CPU(s):     14-27

# ethtool -i enp4s0f0
driver: ixgbe
version: 5.3.3
firmware-version: 0x800007b8, 1.1018.0
bus-info: 0000:04:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

Thanks,
Ravi



>
> --
> Thanks,
> Anatoly
>