[dpdk-users] Slow DPDK startup with many 1G hugepages

[Tan, Jianfeng]

On 6/6/2017 11:25 PM, Imre Pinter wrote:
> [...]
> I've double-checked it, and if I combine Step (1) and Step (2), then OVS start end up in the following error:
> EAL: Detected 32 lcore(s)
> EAL: 1024 hugepages of size 2097152 reserved, but no mounted hugetlbfs found for that size
> EAL: Probing VFIO support...
> EAL: Not enough memory available on socket 1! Requested: 1024MB, available: 0MB
> PANIC in rte_eal_init():
> Cannot init memory
>
> I experienced the same behavior with testpmd.
> However when they use hugepages from the 'huge_qemu_1G' mountpoint, then they start properly.
>
> Br,
> Imre

Ah yes, I tried by myself and encounter similar error as you. And try to 
deep dive into kernel hugetlb related code to see what's going on:

The hugepage allocation path is: hugetlb_fault -> hugetlb_no_page -> 
alloc_huge_page -> dequeue_huge_page_vma. Inside 
dequeue_huge_page_vma(), we can see the code logic to iterate node. But 
from huge_zonelist(), we can see "interleave" policy only applies to a 
VMA. In our case, each hugepage file is an independent VMA. As a result, 
we will go though all hugepages from node to another node one by one.

Sorry that I take "interleave" as granted. Fortunately, there is a try 
to fix this: http://dpdk.org/dev/patchwork/patch/25069/.
Besides, we can write a simple application which will allocate all 
hugepages except those used by OVS-DPDK.

Thanks,
Jianfeng