[dpdk-users] Slow DPDK startup with many 1G hugepages

[Imre Pinter]

> -----Original Message-----
> From: Tan, Jianfeng [mailto:jianfeng.tan at intel.com]
> Sent: 2017. június 7. 10:22
> 
> 
> 
> On 6/6/2017 11:25 PM, Imre Pinter wrote:
> > [...]
> > I've double-checked it, and if I combine Step (1) and Step (2), then OVS
> start end up in the following error:
> > EAL: Detected 32 lcore(s)
> > EAL: 1024 hugepages of size 2097152 reserved, but no mounted hugetlbfs
> > found for that size
> > EAL: Probing VFIO support...
> > EAL: Not enough memory available on socket 1! Requested: 1024MB,
> > available: 0MB PANIC in rte_eal_init():
> > Cannot init memory
> >
> > I experienced the same behavior with testpmd.
> > However when they use hugepages from the 'huge_qemu_1G'
> mountpoint, then they start properly.
> >
> > Br,
> > Imre
> 
> Ah yes, I tried by myself and encounter similar error as you. And try to deep
> dive into kernel hugetlb related code to see what's going on:
> 
> The hugepage allocation path is: hugetlb_fault -> hugetlb_no_page ->
> alloc_huge_page -> dequeue_huge_page_vma. Inside
> dequeue_huge_page_vma(), we can see the code logic to iterate node. But
> from huge_zonelist(), we can see "interleave" policy only applies to a VMA.
> In our case, each hugepage file is an independent VMA. As a result, we will
> go though all hugepages from node to another node one by one.
> 
> Sorry that I take "interleave" as granted. Fortunately, there is a try to fix this:
> http://dpdk.org/dev/patchwork/patch/25069/.
> Besides, we can write a simple application which will allocate all hugepages
> except those used by OVS-DPDK.
> 
> Thanks,
> Jianfeng

Thanks, this is the same patch, as the result of the mail thread suggested by Sergio. I've verified it, and the patch SOLVES the slow DPDK startup issue.

The working setup:
Hugepage mount in fstab:
    nodev    /mnt/huge_qemu_1G    hugetlbfs       pagesize=1G     0       0
    nodev    /mnt/huge_ovs_1G    hugetlbfs       pagesize=1G,size=2G     0       0
DPDK vesrion: 16.11 + patch: [dpdk-dev,v5,1/2] mem: balanced allocation of hugepages
Allocate memory for DPDK application from /mnt/huge_ovs_1G mountpoint.

Thanks,
Imre