[dpdk-dev] mem: warn if address hint is not respected
Checks
Commit Message
Print a warning if the --base-virtaddr hint is not respected
since this might lead to problems when mapping memory in
the secondary process.
Signed-off-by: Jonas Pfefferle <jpf@zurich.ibm.com>
---
lib/librte_eal/linuxapp/eal/eal_memory.c | 23 +++++++++++++++++------
1 file changed, 17 insertions(+), 6 deletions(-)
Comments
31/10/2017 10:08, Jonas Pfefferle:
> Print a warning if the --base-virtaddr hint is not respected
> since this might lead to problems when mapping memory in
> the secondary process.
>
> Signed-off-by: Jonas Pfefferle <jpf@zurich.ibm.com>
Anatoly, please review this patch.
It does not seem to fix something, so it is candidate for 18.02.
On 06-Nov-17 8:26 PM, Thomas Monjalon wrote:
> 31/10/2017 10:08, Jonas Pfefferle:
>> Print a warning if the --base-virtaddr hint is not respected
>> since this might lead to problems when mapping memory in
>> the secondary process.
>>
>> Signed-off-by: Jonas Pfefferle <jpf@zurich.ibm.com>
>
> Anatoly, please review this patch.
> It does not seem to fix something, so it is candidate for 18.02.
>
For some reason my Thunderbird ate the original email, so i'll reply to
this one.
One nitpick would be that we're calling get_virtual_area many times and
it would probably be a good idea to make pagesize static and call
sysconf only once. Otherwise,
Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
"Burakov, Anatoly" <anatoly.burakov@intel.com> wrote on 11/07/2017 02:54:24
PM:
> From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
> To: Thomas Monjalon <thomas@monjalon.net>
> Cc: dev@dpdk.org, Jonas Pfefferle <jpf@zurich.ibm.com>,
jianfeng.tan@intel.com
> Date: 11/07/2017 02:54 PM
> Subject: Re: [dpdk-dev] [PATCH] mem: warn if address hint is not
respected
>
> On 06-Nov-17 8:26 PM, Thomas Monjalon wrote:
> > 31/10/2017 10:08, Jonas Pfefferle:
> >> Print a warning if the --base-virtaddr hint is not respected
> >> since this might lead to problems when mapping memory in
> >> the secondary process.
> >>
> >> Signed-off-by: Jonas Pfefferle <jpf@zurich.ibm.com>
> >
> > Anatoly, please review this patch.
> > It does not seem to fix something, so it is candidate for 18.02.
> >
>
> For some reason my Thunderbird ate the original email, so i'll reply to
> this one.
>
> One nitpick would be that we're calling get_virtual_area many times and
> it would probably be a good idea to make pagesize static and call
> sysconf only once. Otherwise,
We should address this in a separate patch and introduce a pagesize
function
for everyone to use. sysconf is used like this all over the place.
>
> Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
>
> --
> Thanks,
> Anatoly
>
Hi Jonas,
Seems you are trying to use --base-virtaddr to resolve address conflicts
in secondary, I'm wondering how this happened and how to avoid it:
1. what's your hugepage side? Hugepage mmap is size aligned, maybe 1G works?
2. is there more libs loaded in secondary process or memory usage before
EAL init?
3. Since address allocated in one direction, I'm thinking to reserve a
larger "hop" address space as MAP_ANONYMOUS, allocate hugepage, then release
"hop". That essentially reserve an address space big enough for secondary,
and most important the hop size is easy to estimate than --base-virtaddr.
Thanks,
Xueming(Steven)
> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Jonas Pfefferle1
> Sent: Wednesday, November 8, 2017 7:52 PM
> To: Burakov, Anatoly <anatoly.burakov@intel.com>
> Cc: dev@dpdk.org; jianfeng.tan@intel.com; Thomas Monjalon
> <thomas@monjalon.net>
> Subject: Re: [dpdk-dev] [PATCH] mem: warn if address hint is not respected
>
> "Burakov, Anatoly" <anatoly.burakov@intel.com> wrote on 11/07/2017
> 02:54:24
> PM:
>
> > From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
> > To: Thomas Monjalon <thomas@monjalon.net>
> > Cc: dev@dpdk.org, Jonas Pfefferle <jpf@zurich.ibm.com>,
> jianfeng.tan@intel.com
> > Date: 11/07/2017 02:54 PM
> > Subject: Re: [dpdk-dev] [PATCH] mem: warn if address hint is not
> respected
> >
> > On 06-Nov-17 8:26 PM, Thomas Monjalon wrote:
> > > 31/10/2017 10:08, Jonas Pfefferle:
> > >> Print a warning if the --base-virtaddr hint is not respected since
> > >> this might lead to problems when mapping memory in the secondary
> > >> process.
> > >>
> > >> Signed-off-by: Jonas Pfefferle <jpf@zurich.ibm.com>
> > >
> > > Anatoly, please review this patch.
> > > It does not seem to fix something, so it is candidate for 18.02.
> > >
> >
> > For some reason my Thunderbird ate the original email, so i'll reply
> > to this one.
> >
> > One nitpick would be that we're calling get_virtual_area many times
> > and it would probably be a good idea to make pagesize static and call
> > sysconf only once. Otherwise,
>
> We should address this in a separate patch and introduce a pagesize
> function for everyone to use. sysconf is used like this all over the place.
>
> >
> > Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
> >
> > --
> > Thanks,
> > Anatoly
> >
Hi Xueming,
Correct --base-virtaddr was introduced for that purpose. There are
multiple reasons why the address layout of the secondary process might
look different: reasons you mentioned in 2), ASLR etc. I believe there
is no way to avoid this in real world use cases. The reason for this
particular patch is that the address hint (--base-virtaddr) is
sometimes not respected and the kernel falls back to just reserving
any address it can find to satisfy the mapping (see discussion on the
patch), i.e. effectively rendering --base-virtaddr useless.
Regards,
Jonas (new email address)
On Tue, 26 Dec 2017 15:56:10 +0000
"Xueming(Steven) Li" <xuemingl@mellanox.com> wrote:
> Hi Jonas,
>
> Seems you are trying to use --base-virtaddr to resolve address
>conflicts
> in secondary, I'm wondering how this happened and how to avoid it:
> 1. what's your hugepage side? Hugepage mmap is size aligned, maybe
>1G works?
> 2. is there more libs loaded in secondary process or memory usage
>before
> EAL init?
> 3. Since address allocated in one direction, I'm thinking to reserve
>a
> larger "hop" address space as MAP_ANONYMOUS, allocate hugepage, then
>release
> "hop". That essentially reserve an address space big enough for
>secondary,
> and most important the hop size is easy to estimate than
>--base-virtaddr.
>
> Thanks,
> Xueming(Steven)
>
>> -----Original Message-----
>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Jonas
>>Pfefferle1
>> Sent: Wednesday, November 8, 2017 7:52 PM
>> To: Burakov, Anatoly <anatoly.burakov@intel.com>
>> Cc: dev@dpdk.org; jianfeng.tan@intel.com; Thomas Monjalon
>> <thomas@monjalon.net>
>> Subject: Re: [dpdk-dev] [PATCH] mem: warn if address hint is not
>>respected
>>
>> "Burakov, Anatoly" <anatoly.burakov@intel.com> wrote on 11/07/2017
>> 02:54:24
>> PM:
>>
>> > From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
>> > To: Thomas Monjalon <thomas@monjalon.net>
>> > Cc: dev@dpdk.org, Jonas Pfefferle <jpf@zurich.ibm.com>,
>> jianfeng.tan@intel.com
>> > Date: 11/07/2017 02:54 PM
>> > Subject: Re: [dpdk-dev] [PATCH] mem: warn if address hint is not
>> respected
>> >
>> > On 06-Nov-17 8:26 PM, Thomas Monjalon wrote:
>> > > 31/10/2017 10:08, Jonas Pfefferle:
>> > >> Print a warning if the --base-virtaddr hint is not respected
>>since
>> > >> this might lead to problems when mapping memory in the
>>secondary
>> > >> process.
>> > >>
>> > >> Signed-off-by: Jonas Pfefferle <jpf@zurich.ibm.com>
>> > >
>> > > Anatoly, please review this patch.
>> > > It does not seem to fix something, so it is candidate for 18.02.
>> > >
>> >
>> > For some reason my Thunderbird ate the original email, so i'll
>>reply
>> > to this one.
>> >
>> > One nitpick would be that we're calling get_virtual_area many
>>times
>> > and it would probably be a good idea to make pagesize static and
>>call
>> > sysconf only once. Otherwise,
>>
>> We should address this in a separate patch and introduce a pagesize
>> function for everyone to use. sysconf is used like this all over the
>>place.
>>
>> >
>> > Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
>> >
>> > --
>> > Thanks,
>> > Anatoly
>> >
So the idea of item 3 might sound and lead to seldom usage of '--base-virtaddress'.
Reserve an address hole big enough before hugepage almost cost nothing.
> -----Original Message-----
> From: Jonas Pfefferle [mailto:pepperjo@japf.ch]
> Sent: Wednesday, January 3, 2018 5:22 PM
> To: Xueming(Steven) Li <xuemingl@mellanox.com>; Burakov, Anatoly
> <anatoly.burakov@intel.com>
> Cc: dev@dpdk.org; jianfeng.tan@intel.com; Thomas Monjalon
> <thomas@monjalon.net>
> Subject: Re: [dpdk-dev] [PATCH] mem: warn if address hint is not respected
>
> Hi Xueming,
>
> Correct --base-virtaddr was introduced for that purpose. There are
> multiple reasons why the address layout of the secondary process might
> look different: reasons you mentioned in 2), ASLR etc. I believe there is
> no way to avoid this in real world use cases. The reason for this
> particular patch is that the address hint (--base-virtaddr) is sometimes
> not respected and the kernel falls back to just reserving any address it
> can find to satisfy the mapping (see discussion on the patch), i.e.
> effectively rendering --base-virtaddr useless.
>
> Regards,
> Jonas (new email address)
>
>
> On Tue, 26 Dec 2017 15:56:10 +0000
> "Xueming(Steven) Li" <xuemingl@mellanox.com> wrote:
> > Hi Jonas,
> >
> > Seems you are trying to use --base-virtaddr to resolve address
> >conflicts in secondary, I'm wondering how this happened and how to
> >avoid it:
> > 1. what's your hugepage side? Hugepage mmap is size aligned, maybe 1G
> >works?
> > 2. is there more libs loaded in secondary process or memory usage
> >before EAL init?
> > 3. Since address allocated in one direction, I'm thinking to reserve a
> >larger "hop" address space as MAP_ANONYMOUS, allocate hugepage, then
> >release "hop". That essentially reserve an address space big enough
> >for secondary, and most important the hop size is easy to estimate
> >than --base-virtaddr.
> >
> > Thanks,
> > Xueming(Steven)
> >
> >> -----Original Message-----
> >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Jonas
> >>Pfefferle1
> >> Sent: Wednesday, November 8, 2017 7:52 PM
> >> To: Burakov, Anatoly <anatoly.burakov@intel.com>
> >> Cc: dev@dpdk.org; jianfeng.tan@intel.com; Thomas Monjalon
> >><thomas@monjalon.net>
> >> Subject: Re: [dpdk-dev] [PATCH] mem: warn if address hint is not
> >>respected
> >>
> >> "Burakov, Anatoly" <anatoly.burakov@intel.com> wrote on 11/07/2017
> >> 02:54:24
> >> PM:
> >>
> >> > From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
> >> > To: Thomas Monjalon <thomas@monjalon.net>
> >> > Cc: dev@dpdk.org, Jonas Pfefferle <jpf@zurich.ibm.com>,
> >> jianfeng.tan@intel.com
> >> > Date: 11/07/2017 02:54 PM
> >> > Subject: Re: [dpdk-dev] [PATCH] mem: warn if address hint is not
> >> respected
> >> >
> >> > On 06-Nov-17 8:26 PM, Thomas Monjalon wrote:
> >> > > 31/10/2017 10:08, Jonas Pfefferle:
> >> > >> Print a warning if the --base-virtaddr hint is not respected
> >>since
> >> > >> this might lead to problems when mapping memory in the
> >>secondary
> >> > >> process.
> >> > >>
> >> > >> Signed-off-by: Jonas Pfefferle <jpf@zurich.ibm.com>
> >> > >
> >> > > Anatoly, please review this patch.
> >> > > It does not seem to fix something, so it is candidate for 18.02.
> >> > >
> >> >
> >> > For some reason my Thunderbird ate the original email, so i'll
> >>reply
> >> > to this one.
> >> >
> >> > One nitpick would be that we're calling get_virtual_area many
> >>times
> >> > and it would probably be a good idea to make pagesize static and
> >>call
> >> > sysconf only once. Otherwise,
> >>
> >> We should address this in a separate patch and introduce a pagesize
> >>function for everyone to use. sysconf is used like this all over the
> >>place.
> >>
> >> >
> >> > Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
> >> >
> >> > --
> >> > Thanks,
> >> > Anatoly
> >> >
I don't believe that it would work since you cannot assume that mmap
addresses will just increase. Especially with ASLR anything can
happen. I don't understand why you want to get rid of --base-virtaddr
I think it is a valid solution for the problem.
On Wed, 3 Jan 2018 09:37:00 +0000
"Xueming(Steven) Li" <xuemingl@mellanox.com> wrote:
> So the idea of item 3 might sound and lead to seldom usage of
>'--base-virtaddress'.
> Reserve an address hole big enough before hugepage almost cost
>nothing.
>
>> -----Original Message-----
>> From: Jonas Pfefferle [mailto:pepperjo@japf.ch]
>> Sent: Wednesday, January 3, 2018 5:22 PM
>> To: Xueming(Steven) Li <xuemingl@mellanox.com>; Burakov, Anatoly
>> <anatoly.burakov@intel.com>
>> Cc: dev@dpdk.org; jianfeng.tan@intel.com; Thomas Monjalon
>> <thomas@monjalon.net>
>> Subject: Re: [dpdk-dev] [PATCH] mem: warn if address hint is not
>>respected
>>
>> Hi Xueming,
>>
>> Correct --base-virtaddr was introduced for that purpose. There are
>> multiple reasons why the address layout of the secondary process
>>might
>> look different: reasons you mentioned in 2), ASLR etc. I believe
>>there is
>> no way to avoid this in real world use cases. The reason for this
>> particular patch is that the address hint (--base-virtaddr) is
>>sometimes
>> not respected and the kernel falls back to just reserving any
>>address it
>> can find to satisfy the mapping (see discussion on the patch), i.e.
>> effectively rendering --base-virtaddr useless.
>>
>> Regards,
>> Jonas (new email address)
>>
>>
>> On Tue, 26 Dec 2017 15:56:10 +0000
>> "Xueming(Steven) Li" <xuemingl@mellanox.com> wrote:
>> > Hi Jonas,
>> >
>> > Seems you are trying to use --base-virtaddr to resolve address
>> >conflicts in secondary, I'm wondering how this happened and how to
>> >avoid it:
>> > 1. what's your hugepage side? Hugepage mmap is size aligned, maybe
>>1G
>> >works?
>> > 2. is there more libs loaded in secondary process or memory usage
>> >before EAL init?
>> > 3. Since address allocated in one direction, I'm thinking to
>>reserve a
>> >larger "hop" address space as MAP_ANONYMOUS, allocate hugepage,
>>then
>> >release "hop". That essentially reserve an address space big
>>enough
>> >for secondary, and most important the hop size is easy to estimate
>> >than --base-virtaddr.
>> >
>> > Thanks,
>> > Xueming(Steven)
>> >
>> >> -----Original Message-----
>> >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Jonas
>> >>Pfefferle1
>> >> Sent: Wednesday, November 8, 2017 7:52 PM
>> >> To: Burakov, Anatoly <anatoly.burakov@intel.com>
>> >> Cc: dev@dpdk.org; jianfeng.tan@intel.com; Thomas Monjalon
>> >><thomas@monjalon.net>
>> >> Subject: Re: [dpdk-dev] [PATCH] mem: warn if address hint is not
>> >>respected
>> >>
>> >> "Burakov, Anatoly" <anatoly.burakov@intel.com> wrote on
>>11/07/2017
>> >> 02:54:24
>> >> PM:
>> >>
>> >> > From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
>> >> > To: Thomas Monjalon <thomas@monjalon.net>
>> >> > Cc: dev@dpdk.org, Jonas Pfefferle <jpf@zurich.ibm.com>,
>> >> jianfeng.tan@intel.com
>> >> > Date: 11/07/2017 02:54 PM
>> >> > Subject: Re: [dpdk-dev] [PATCH] mem: warn if address hint is
>>not
>> >> respected
>> >> >
>> >> > On 06-Nov-17 8:26 PM, Thomas Monjalon wrote:
>> >> > > 31/10/2017 10:08, Jonas Pfefferle:
>> >> > >> Print a warning if the --base-virtaddr hint is not respected
>> >>since
>> >> > >> this might lead to problems when mapping memory in the
>> >>secondary
>> >> > >> process.
>> >> > >>
>> >> > >> Signed-off-by: Jonas Pfefferle <jpf@zurich.ibm.com>
>> >> > >
>> >> > > Anatoly, please review this patch.
>> >> > > It does not seem to fix something, so it is candidate for
>>18.02.
>> >> > >
>> >> >
>> >> > For some reason my Thunderbird ate the original email, so i'll
>> >>reply
>> >> > to this one.
>> >> >
>> >> > One nitpick would be that we're calling get_virtual_area many
>> >>times
>> >> > and it would probably be a good idea to make pagesize static
>>and
>> >>call
>> >> > sysconf only once. Otherwise,
>> >>
>> >> We should address this in a separate patch and introduce a
>>pagesize
>> >>function for everyone to use. sysconf is used like this all over
>>the
>> >>place.
>> >>
>> >> >
>> >> > Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
>> >> >
>> >> > --
>> >> > Thanks,
>> >> > Anatoly
>> >> >
>
07/11/2017 14:54, Burakov, Anatoly:
> On 06-Nov-17 8:26 PM, Thomas Monjalon wrote:
> > 31/10/2017 10:08, Jonas Pfefferle:
> >> Print a warning if the --base-virtaddr hint is not respected
> >> since this might lead to problems when mapping memory in
> >> the secondary process.
> >>
> >> Signed-off-by: Jonas Pfefferle <jpf@zurich.ibm.com>
> >
> > Anatoly, please review this patch.
> > It does not seem to fix something, so it is candidate for 18.02.
> >
>
> For some reason my Thunderbird ate the original email, so i'll reply to
> this one.
>
> One nitpick would be that we're calling get_virtual_area many times and
> it would probably be a good idea to make pagesize static and call
> sysconf only once. Otherwise,
>
> Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
Applied, thanks
@@ -256,33 +256,44 @@ static void *
get_virtual_area(size_t *size, size_t hugepage_sz)
{
void *addr;
+ void *addr_hint;
int fd;
long aligned_addr;
if (internal_config.base_virtaddr != 0) {
- addr = (void*) (uintptr_t) (internal_config.base_virtaddr +
- baseaddr_offset);
+ int page_size = sysconf(_SC_PAGE_SIZE);
+ addr_hint = (void *) (uintptr_t)
+ (internal_config.base_virtaddr + baseaddr_offset);
+ addr_hint = RTE_PTR_ALIGN_FLOOR(addr_hint, page_size);
+ } else {
+ addr_hint = NULL;
}
- else addr = NULL;
RTE_LOG(DEBUG, EAL, "Ask a virtual area of 0x%zx bytes\n", *size);
+
fd = open("/dev/zero", O_RDONLY);
if (fd < 0){
RTE_LOG(ERR, EAL, "Cannot open /dev/zero\n");
return NULL;
}
do {
- addr = mmap(addr,
- (*size) + hugepage_sz, PROT_READ,
+ addr = mmap(addr_hint, (*size) + hugepage_sz, PROT_READ,
#ifdef RTE_ARCH_PPC_64
MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB,
#else
MAP_PRIVATE,
#endif
fd, 0);
- if (addr == MAP_FAILED)
+ if (addr == MAP_FAILED) {
*size -= hugepage_sz;
+ } else if (addr_hint != NULL && addr != addr_hint) {
+ RTE_LOG(WARNING, EAL, "WARNING! Base virtual address "
+ "hint (%p != %p) not respected!\n",
+ addr_hint, addr);
+ RTE_LOG(WARNING, EAL, " This may cause issues with "
+ "mapping memory into secondary processes\n");
+ }
} while (addr == MAP_FAILED && *size > 0);
if (addr == MAP_FAILED) {