[dpdk-dev] [PATCH v6 2/3] eal: add memory pre-allocation from existing files

[David Marchand]

Hello Dmitry, Slava,

On Mon, Oct 11, 2021 at 10:57 AM Dmitry Kozlyuk <dkozlyuk at oss.nvidia.com> wrote:
>
> From: Viacheslav Ovsiienko <viacheslavo at nvidia.com>
>
> The primary DPDK process launch might take a long time if initially
> allocated memory is large. From practice allocation of 1 TB of memory
> over 1 GB hugepages on Linux takes tens of seconds. Fast restart
> is highly desired for some applications and launch delay presents
> a problem.
>
> The primary delay happens in this call trace:
>   rte_eal_init()
>     rte_eal_memory_init()
>       rte_eal_hugepage_init()
>         eal_dynmem_hugepage_init()
>           eal_memalloc_alloc_seg_bulk()
>             alloc_seg()
>               mmap()
>
> The largest part of the time spent in mmap() is filling the memory
> with zeros. Kernel does so to prevent data leakage from a process
> that was last using the page. However, in a controlled environment
> it may not be the issue, while performance is. (Linux-specific
> MAP_UNINITIALIZED flag allows mapping without clearing, but it is
> disabled in all popular distributions for the reason above.)
>
> It is proposed to add a new EAL option: --mem-file FILE1,FILE2,...
> to map hugepages "as is" from specified FILEs in hugetlbfs.
> Compared to using external memory for the task, EAL option requires
> no change to application code, while allowing administrator
> to control hugepage sizes and their NUMA affinity.
>
> Limitations of the feature:
>
> * Linux-specific (only Linux maps hugepages from files).
> * Incompatible with --legacy-mem (partially replaces it).
> * Incompatible with --single-file-segments
>   (--mem-file FILEs can contain as many segments as needed).
> * Incompatible with --in-memory (logically).
>
> A warning about possible security implications is printed
> when --mem-file is used.
>
> Until this patch DPDK allocator always cleared memory on freeing,
> so that it did not have to do that on allocation, while new memory
> was cleared by the kernel. When --mem-file is in use, DPDK clears memory
> after allocation in rte_zmalloc() and does not clean it on freeing.
> Effectively user trades fast startup for occasional allocation slowdown
> whenever it is absolutely necessary. When memory is recycled, it is
> cleared again, which is suboptimal par se, but saves complication
> of memory management.

I have some trouble figuring the need for the list of files.
Why not use a global knob --mem-clear-on-alloc for this behavior change?


>
> Signed-off-by: Viacheslav Ovsiienko <viacheslavo at nvidia.com>
> Signed-off-by: Dmitry Kozlyuk <dkozlyuk at nvidia.com>


-- 
David Marchand