[dpdk-users] rte_segments: hugepages are not in contiguous memory

[Sergio Gonzalez Monroy]

Hi folks,

In theory, there shouldn't be any performance difference between having 
a mempool allocated from a single memseg (given the use the same number 
of hugepages) versus multiple memsegs
as it is all done on mempool creation/setup and each mbuf has its own 
phys address.

Tom, I cannot think of a reason why you would have higher memory access 
for having scatter hugapages vs contig hugepages.
Any details on the test you were running?

Sergio

On 04/10/2016 13:02, tom.barbette at ulg.ac.be wrote:
> There is a noticeable performance drop with more scattering of the huge pages.
>
> I did not measure any difference accurately but I ended up rebooting my DUT between each performance test because the pages get scattered with time and re-launch of the DPDK application instead of the whole machine, because the tests showed higher memory access cost each time I re-launched the application.
>
> Tom
>
> ----- Mail original -----
> De: "Andriy Berestovskyy"<aber at semihalf.com>
> À: "Renata Saiakhova"<Renata.Saiakhova at oneaccess-net.com>
> Cc: "Sergio Gonzalez Monroy"<sergio.gonzalez.monroy at intel.com>, "users"<users at dpdk.org>
> Envoyé: Mardi 4 Octobre 2016 13:27:23
> Objet: Re: [dpdk-users] rte_segments: hugepages are not in contiguous	memory
>
> Renata,
> In theory 512 contiguous 2MB huge pages might get transparently
> promoted to a single 1GB "superpage" and single TLB entry, but I am
> not even sure if it is implemented in Linux...
>
> So, I do not think there will be any noticeable performance difference
> between contiguous and non-contiguous 2MB huge pages. But you better
> measure it to make sure ;)
>
> Regards,
> Andriy
>
> On Tue, Oct 4, 2016 at 12:48 PM, Renata Saiakhova
> <Renata.Saiakhova at oneaccess-net.com>  wrote:
>> Hi Andriy,
>>
>> thanks for your reply. I guess that contiguous memory is requested because
>> of the performance reasons. Do you know if I can expect a noticeable
>> performance drop using non-contiguous memory?
>>
>> Renata
>>
>>
>> On 10/04/2016 12:13 PM, Andriy Berestovskyy wrote:
>>> Hi Renata,
>>> DPDK supports non-contiguous memory pools, but
>>> rte_pktmbuf_pool_create() uses rte_mempool_create_empty() with flags
>>> set to zero, i.e. requests contiguous memory.
>>>
>>> As a workaround, in rte_pktmbuf_pool_create() try to pass
>>> MEMPOOL_F_NO_PHYS_CONTIG flag as the last argument to
>>> rte_mempool_create_empty().
>>>
>>> Note that KNI and some PMDs in 16.07 still require contiguous memory
>>> pools, so the trick might not work for your setup. For the KNI try the
>>> DPDK's master branch which includes the commit by Ferruh Yigit:
>>>
>>> 8451269 kni: remove continuous memory restriction
>>>
>>> Regards,
>>> Andriy
>>>
>>>
>>> On Tue, Oct 4, 2016 at 11:38 AM, Renata Saiakhova
>>> <Renata.Saiakhova at oneaccess-net.com>  wrote:
>>>> Hi Sergio,
>>>>
>>>> thank you for your quick answer. I also tried to allocate 1GB hugepage,
>>>> but
>>>> seems kernel fails to allocate it: previously I've seen that
>>>> HugePages_Total
>>>> in /proc/meminfo is set to 0, now - kernel hangs at boot time (don't know
>>>> why).
>>>> But anyway, if there is no way to control hugepage allocation in the
>>>> sense
>>>> they are in contiguous memory there is only way to accept it and adapt
>>>> the
>>>> code that it creates several pools which in total satisfy the requested
>>>> size.
>>>>
>>>> Renata
>>>>
>>>>
>>>> On 10/04/2016 10:27 AM, Sergio Gonzalez Monroy wrote:
>>>>> On 04/10/2016 09:00, Renata Saiakhova wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> I'm using dpdk 16.04 (I tried 16.07 with the same results) and linux
>>>>>> kernel 4.4.20 in a virtual machine (I'm using libvirt framework). I
>>>>>> pass a
>>>>>> parameter in kernel command line to allocate 512 hugepages of 2 MB at
>>>>>> boot
>>>>>> time. They are successfully allocated. When an application with dpdk
>>>>>> starts
>>>>>> it calls rte_pktmbuf_pool_create() which in turns requests internally
>>>>>> 649363712 bytes.  Those bytes should be allocated from one of
>>>>>> rte_memseg.
>>>>>> rte_memsegs describes contiguous portions of memory (both physical and
>>>>>> virtual) built on hugepages. This allocation fails, because there are
>>>>>> no
>>>>>> rte_memsegs of this size (or bigger). Further debugging shows that
>>>>>> hugepages
>>>>>> are allocated in non-contiguous physical memory and therefore
>>>>>> rte_memsegs
>>>>>> are built respecting gaps in physical memory.
>>>>>> Below are the sizes of segments built on hugepages (in bytes)
>>>>>> 2097152
>>>>>> 6291456
>>>>>> 2097152
>>>>>> 524288000
>>>>>> 2097152
>>>>>> 532676608
>>>>>> 2097152
>>>>>> 2097152
>>>>>> So there are 5 segments which includes only one hugepage!
>>>>>> This behavior is completely different to what I observe with linux
>>>>>> kernel
>>>>>> 3.8 (used with the same application with dpdk) - where all hugepages
>>>>>> are
>>>>>> allocated in contiguous memory.
>>>>>> Does anyone experience the same issue? Could it be some kernel option
>>>>>> which can do the magic? If not, and kernel can allocated hugepages in
>>>>>> non-contiguous memory how dpdk is going to resolve it?
>>>>>>
>>>>> I don't think there is anything we can do to force the kernel to
>>>>> pre-allocate contig hugepages on boot. If there was, we wouldn't need to
>>>>> do
>>>>> all this mapping sorting and grouping we do on DPDK
>>>>> as we would rely on the kernel giving us pre-allocated contig hugepages.
>>>>>
>>>>> If you have plenty of memory one possible work around would be to
>>>>> increase
>>>>> the number of default hugepages so we are likely to find more contiguous
>>>>> ones.
>>>>>
>>>>> Is using 1GB hugepages a possibility in your case?
>>>>>
>>>>> Sergio
>>>>>
>>>>>> Thanks in advance,
>>>>>> Renata
>>>>>>
>>>>> .
>>>>>
>