[dpdk-users] rte_segments: hugepages are not in contiguous memory

[Renata Saiakhova]

Hi Sergio,

thank you for your quick answer. I also tried to allocate 1GB hugepage, 
but seems kernel fails to allocate it: previously I've seen that 
HugePages_Total in /proc/meminfo is set to 0, now - kernel hangs at boot 
time (don't know why).
But anyway, if there is no way to control hugepage allocation in the 
sense they are in contiguous memory there is only way to accept it and 
adapt the code that it creates several pools which in total satisfy the 
requested size.

Renata

On 10/04/2016 10:27 AM, Sergio Gonzalez Monroy wrote:
> On 04/10/2016 09:00, Renata Saiakhova wrote:
>> Hi all,
>>
>> I'm using dpdk 16.04 (I tried 16.07 with the same results) and linux 
>> kernel 4.4.20 in a virtual machine (I'm using libvirt framework). I 
>> pass a parameter in kernel command line to allocate 512 hugepages of 
>> 2 MB at boot time. They are successfully allocated. When an 
>> application with dpdk starts it calls rte_pktmbuf_pool_create() which 
>> in turns requests internally 649363712 bytes.  Those bytes should be 
>> allocated from one of rte_memseg. rte_memsegs describes contiguous 
>> portions of memory (both physical and virtual) built on hugepages. 
>> This allocation fails, because there are no rte_memsegs of this size 
>> (or bigger). Further debugging shows that hugepages are allocated in 
>> non-contiguous physical memory and therefore rte_memsegs are built 
>> respecting gaps in physical memory.
>> Below are the sizes of segments built on hugepages (in bytes)
>> 2097152
>> 6291456
>> 2097152
>> 524288000
>> 2097152
>> 532676608
>> 2097152
>> 2097152
>> So there are 5 segments which includes only one hugepage!
>> This behavior is completely different to what I observe with linux 
>> kernel 3.8 (used with the same application with dpdk) - where all 
>> hugepages are allocated in contiguous memory.
>> Does anyone experience the same issue? Could it be some kernel option 
>> which can do the magic? If not, and kernel can allocated hugepages in 
>> non-contiguous memory how dpdk is going to resolve it?
>>
>
> I don't think there is anything we can do to force the kernel to 
> pre-allocate contig hugepages on boot. If there was, we wouldn't need 
> to do all this mapping sorting and grouping we do on DPDK
> as we would rely on the kernel giving us pre-allocated contig hugepages.
>
> If you have plenty of memory one possible work around would be to 
> increase the number of default hugepages so we are likely to find more 
> contiguous ones.
>
> Is using 1GB hugepages a possibility in your case?
>
> Sergio
>
>> Thanks in advance,
>> Renata
>>
>
> .
>