[dpdk-users] Cannot mmap device resource file when running the secondary process in docker

[Kai Zhang]

Hi there,

I am using DPDK-16.11 on XL710 40GbE NIC, only binding port 1 to DPDK.
OS: CentOS 7.3.1611 with Linux kernel version 3.8.0-30.

Running an open source project: https://github.com/sdnfv/openNetVM

The primary process is running in the user space, while the secondary
process is running in docker. But having the following error:

===========================
[kai at server scripts]$ sudo ./docker.sh -h /dev/hugepages/ -o
/home/kai/openNetVM/ -n test -D /dev/uio0
root at 3f18f6cf89da:/openNetVM# ./examples/speed_tester/go.sh 5 1 1
sudo: unable to resolve host 3f18f6cf89da
EAL: Detected 24 lcore(s)
EAL: Probing VFIO support...
EAL: PCI device 0000:02:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1583 net_i40e
EAL: PCI device 0000:02:00.1 on NUMA socket 0
EAL:   probe driver: 8086:1583 net_i40e
EAL: Cannot mmap device resource file
/sys/bus/pci/devices/0000:02:00.1/resource3
to address: 0x7ffff7fee000
EAL: Error - exiting with code: 1
  Cause: Requested device 0000:02:00.1 cannot be used
===========================

I have encountered a similar issue in the user space before, and asked
questions in this mailing list. I have tried the suggestions from David
Coen for my previous issue as follows, but it does not work here.
------------------------------------------------
- Disabling ASLR by adding those two lines to "/etc/sysctl.conf":
# Disable Address Space Layout Randomization (ASLR)  (needed by DPDK)
kernel.randomize_va_space = 0

- Getting virtual address of the first (the one with the minimum address
value) memory segment returned from the function
"rte_eal_get_physmem_layout ()", called from a "dummy" primary application
used only to get this address.

- Passing the above virtual address as a parameter for the "real" primary
application using the " --base-virtaddr= " dpdk command line option. When
secondary app starts, it all goes well with the specified base address.
------------------------------------------------

I print the virtual address of memory zones with
rte_eal_get_physmem_layout()
------------------------------------------------
APP: virtual address [0x7ffff4800000, 0x7ffff4a00000], physical address
0x40800000, len 2097152
APP: virtual address [0x7fffc1e00000, 0x7ffff4600000], physical address
0x40c00000, len 847249408
APP: virtual address [0x7fffc1400000, 0x7fffc1c00000], physical address
0x7b400000, len 8388608
APP: virtual address [0x7ffdf4600000, 0x7fffc1200000], physical address
0xe4f400000, len 7730102272
APP: virtual address [0x7ffdf4200000, 0x7ffdf4400000], physical address
0x101c200000, len 2097152
------------------------------------------------

I try to print the mapped address that caused the error in docker:
*mapaddr* from pci_map_resource() 0x7ffff7fe5000
*uio_res->map[i].addr* 0x7ffff7fee000
They do not equal leads to the error
in lib/librte_eal/common/eal_common_pci_uio.c
But their address are not belong to any of the above memory regions.

I am not familiar with the virtual address mapping. Anyone knows why this
happens?

Thanks,
Kai