[dpdk-dev] Creation of mbuf pool for socket 0 failed on VM

[Olivier Matz]

Hi,

On 05/28/2016 09:39 AM, Murali Krishna wrote:
> Hi,
> 
> I have attached two 10G NICs to VM as pass-through devices and I’m trying
> to run testpmd on the VM. I have assigned 4GB RAM and 2 cores to the VM.
> 
> /DPDK22/testpmd -c 0x3 -n 2 -d /DPDK22/librte_pmd_oce.so -w $P0 -w $P1 --
> -i --nb-cores=1 --nb-ports=2
> 
> Where P0 and P1 are the PCI IDs of the two NIC ports.
> 
> Below is the complete testpmd app output:
> 
> EAL: Detected lcore 0 as core 0 on socket 0
> EAL: Detected lcore 1 as core 0 on socket 0
> EAL: Support maximum 128 logical core(s) by configuration.
> EAL: Detected 2 lcore(s)
> EAL: VFIO modules not all loaded, skip VFIO support...
> EAL: Setting up physically contiguous memory...
> EAL: Ask a virtual area of 0x200000 bytes
> EAL: Virtual area found at 0x7f1d27600000 (size = 0x200000)
> EAL: Ask a virtual area of 0x5000000 bytes
> EAL: Virtual area found at 0x7f1d22400000 (size = 0x5000000)
> EAL: Ask a virtual area of 0x200000 bytes
> EAL: Virtual area found at 0x7f1d22000000 (size = 0x200000)
> EAL: Ask a virtual area of 0x200000 bytes
> EAL: Virtual area found at 0x7f1d21c00000 (size = 0x200000)
> EAL: Ask a virtual area of 0x200000 bytes
> EAL: Virtual area found at 0x7f1d21800000 (size = 0x200000)
> EAL: Ask a virtual area of 0x800000 bytes
> EAL: Virtual area found at 0x7f1d20e00000 (size = 0x800000)
> EAL: Ask a virtual area of 0x200000 bytes
> EAL: Virtual area found at 0x7f1d20a00000 (size = 0x200000)
> EAL: Ask a virtual area of 0x400000 bytes
> EAL: Virtual area found at 0x7f1d20400000 (size = 0x400000)
> EAL: Ask a virtual area of 0x200000 bytes
> EAL: Virtual area found at 0x7f1d20000000 (size = 0x200000)
> EAL: Ask a virtual area of 0x600000 bytes
> EAL: Virtual area found at 0x7f1d1f800000 (size = 0x600000)
> EAL: Ask a virtual area of 0x200000 bytes
> EAL: Virtual area found at 0x7f1d1f400000 (size = 0x200000)
> EAL: Ask a virtual area of 0x400000 bytes
> EAL: Virtual area found at 0x7f1d1ee00000 (size = 0x400000)
> EAL: Ask a virtual area of 0x400000 bytes
> EAL: Virtual area found at 0x7f1d1e800000 (size = 0x400000)
> EAL: Ask a virtual area of 0x200000 bytes
> EAL: Virtual area found at 0x7f1d1e400000 (size = 0x200000)
> EAL: Ask a virtual area of 0x200000 bytes
> EAL: Virtual area found at 0x7f1d1e000000 (size = 0x200000)
> EAL: Ask a virtual area of 0x200000 bytes
> EAL: Virtual area found at 0x7f1d1dc00000 (size = 0x200000)
> EAL: Ask a virtual area of 0x200000 bytes
> EAL: Virtual area found at 0x7f1d1d800000 (size = 0x200000)
> EAL: Requesting 64 pages of size 2MB from socket 0

The logs above show the memory mapped in huge pages. The largest
physically contiguous zone you have is 0x5000000 = ~80MB.

> EAL: TSC frequency is ~2596992 KHz
> EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable
> clock cycles !
> EAL: open shared lib //DPDK22/librte_pmd_oce.so
> PMD: librte_pmd_oce by 6WIND registered
> EAL: Master lcore 0 is ready (tid=288c3900;cpuset=[0])
> EAL: lcore 1 is ready (tid=1cdf2700;cpuset=[1])
> EAL: PCI device 0000:00:09.0 on NUMA socket -1
> EAL:   probe driver: 10df:720 librte_pmd_oce
> EAL: PCI device 0000:00:0a.0 on NUMA socket -1
> EAL:   probe driver: 10df:720 librte_pmd_oce
> Interactive-mode selected
> EAL: Error - exiting with code: 1
>   Cause: Creation of mbuf pool for socket 0 failed
> 
> Hugepage allocation:
> 
> mkdir -p /mnt/huge
> mount -t hugetlbfs nodev /mnt/huge
> echo 64 >
> /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
> 
> Can someone tell me if there is anything wrong here.

The number of mbuf that testpmd tries to allocate depends on
several parameters (number of cores, max number of ports, ...).
>From what I remember, it's more than 100K by default, and a
mbuf is around 2.5KB, which means ~ 250MB of memory. I think
the error log in testpmd should be more user friendly however.

The solutions are:
- allocate more memory (note that in dpdk.org head, the memory
  does need to be physically contiguous for the whole mbuf pool)
- reduce the number of mbufs, ex with --total-num-mbufs=16384


Regards,
Olivier