[dpdk-dev] [PATCH v2 13/41] eal: replace memseg with memseg lists

santosh santosh.shukla at caviumnetworks.com
Sat Mar 24 07:01:21 CET 2018


Hi Anatoly,


On Wednesday 07 March 2018 10:26 PM, Anatoly Burakov wrote:
> Before, we were aggregating multiple pages into one memseg, so the
> number of memsegs was small. Now, each page gets its own memseg,
> so the list of memsegs is huge. To accommodate the new memseg list
> size and to keep the under-the-hood workings sane, the memseg list
> is now not just a single list, but multiple lists. To be precise,
> each hugepage size available on the system gets one or more memseg
> lists, per socket.
>
> In order to support dynamic memory allocation, we reserve all
> memory in advance. As in, we do an anonymous mmap() of the entire
> maximum size of memory per hugepage size, per socket (which is
> limited to either RTE_MAX_MEMSEG_PER_TYPE pages or
> RTE_MAX_MEM_PER_TYPE gigabytes worth of memory, whichever is the
> smaller one), split over multiple lists (which are limited to
> either RTE_MAX_MEMSEG_PER_LIST memsegs or RTE_MAX_MEM_PER_LIST
> gigabytes per list, whichever is the smaller one).
>
> So, for each hugepage size, we get (by default) up to 128G worth
> of memory, per socket, split into chunks of up to 32G in size.
> The address space is claimed at the start, in eal_common_memory.c.
> The actual page allocation code is in eal_memalloc.c (Linux-only
> for now), and largely consists of copied EAL memory init code.
>
> Pages in the list are also indexed by address. That is, for
> non-legacy mode, in order to figure out where the page belongs,
> one can simply look at base address for a memseg list. Similarly,
> figuring out IOVA address of a memzone is a matter of finding the
> right memseg list, getting offset and dividing by page size to get
> the appropriate memseg. For legacy mode, old behavior of walking
> the memseg list remains.
>
> Due to switch to fbarray and to avoid any intrusive changes,
> secondary processes are not supported in this commit. Also, one
> particular API call (dump physmem layout) no longer makes sense
> and was removed, according to deprecation notice [1].
>
> In legacy mode, nothing is preallocated, and all memsegs are in
> a list like before, but each segment still resides in an appropriate
> memseg list.
>
> The rest of the changes are really ripple effects from the memseg
> change - heap changes, compile fixes, and rewrites to support
> fbarray-backed memseg lists.
>
> [1] http://dpdk.org/dev/patchwork/patch/34002/
>
> Signed-off-by: Anatoly Burakov <anatoly.burakov at intel.com>
> ---

Thanks for good work!.
Few observations:
# Noticed performance regression for thunderx platform for l3fwd application,
drops by 3%. git bisect shows this changeset is offending commit.
I'm still investigating reason for perf-dip..
Would like to know - have you noticed any regression on x86 platform?

# In next version, pl. make sure that individual patch builds successfully.
Right now, Some patches are dependent on other, leads to build break, observed while
git-bisecting.

Few examples are:
>>fa71cdef6963ed795fdd7e7f35085170bb300e39
>>1037fcd989176c5cc83db6223534205cac469765
>> befdec10759d30275a17a829919ee45228d91d3c  
>> 495e60f4e02af8a344c0f817a60d1ee9b9322df4 
[above commits are from your github repo..]

# Nits:
Perhaps you may club all below comits into one single patch,
as changes are identical... that way you'd reduce patch count by few less.
9a1e2a7bd9f6248c680ad3e444b6f173eb92d457 net/vmxnet3: use contiguous allocation for DMA memory
46388b194cd559b5cf7079e01b04bf67a99b64d7 net/virtio: use contiguous allocation for DMA memory
a3d2eb10bd998ba3ae3a3d39adeaff38d2e53a9d net/qede: use contiguous allocation for DMA memory
6f16b23ef1f472db475edf05159dea5ae741dbf8 net/i40e: use contiguous allocation for DMA memory
f9f7576eed35cb6aa50793810cdda43bcc0f4642 net/enic: use contiguous allocation for DMA memory
2af6c33009b8008da7028a351efed2932b1a13d0 net/ena: use contiguous allocation for DMA memory
18003e22bd7087e5e2e03543cb662d554f7bec52 net/cxgbe: use contiguous allocation for DMA memory
59f79182502dcb3634dfa3e7b918195829777460 net/bnx2x: use contiguous allocation for DMA memory
f481a321e41da82ddfa00f5ddbcb42fc29e6ae76 net/avf: use contiguous allocation for DMA memory
5253e9b757c1855a296656d939f5c28e651fea69 crypto/qat: use contiguous allocation for DMA memory
297ab037b4c0d9d725aa6cfdd2c33f7cd9396899 ethdev: use contiguous allocation for DMA memory

Thanks.



More information about the dev mailing list