[PATCH] common/mlx5: adjust fork call with the new kernel API

[Slava Ovsiienko]

> -----Original Message-----
> From: Stephen Hemminger <stephen at networkplumber.org>
> Sent: Thursday, May 25, 2023 6:28 PM
> To: Slava Ovsiienko <viacheslavo at nvidia.com>
> Cc: Erez Ferber <erezf at nvidia.com>; dev at dpdk.org; Matan Azrad
> <matan at nvidia.com>; Raslan Darawsheh <rasland at nvidia.com>;
> stable at dpdk.org
> Subject: Re: [PATCH] common/mlx5: adjust fork call with the new kernel API
> 
> On Thu, 25 May 2023 08:10:03 +0000
> Slava Ovsiienko <viacheslavo at nvidia.com> wrote:
> 
> > > -----Original Message-----
> > > From: Stephen Hemminger <stephen at networkplumber.org>
> > > Sent: Wednesday, May 24, 2023 5:50 PM
> > > To: Erez Ferber <erezf at nvidia.com>
> > > Cc: dev at dpdk.org; Slava Ovsiienko <viacheslavo at nvidia.com>; Matan
> > > Azrad <matan at nvidia.com>; Raslan Darawsheh <rasland at nvidia.com>;
> > > stable at dpdk.org
> > > Subject: Re: [PATCH] common/mlx5: adjust fork call with the new
> > > kernel API
> > >
> > > On Wed, 24 May 2023 15:01:40 +0300
> > > <erezf at nvidia.com> wrote:
> > >
> > > > From: Erez Ferber <erezf at nvidia.com>
> > > >
> > > > While doing process fork() the operating system remaps all the
> > > > parent process's memory to the address space of the child process
> > > > and activates the Copy-on-Write mechanics - it duplicates physical
> > > > pages once memory writing happens in the child process. Sometimes
> > > > memory duplication is not allowed - for example, if the page
> > > > contains hardware queue descriptors. To handle similar issues the
> > > > rdma-core library should be prepared for forking.
> > > >
> > > > The ibv_fork_init() prepares the library to track all the related
> > > > memory and prevent it from forking using madvise() system API.
> > > > This approach allows fork, but not all the memory is forked to the
> > > > child process and, application should care not to touch pages
> > > > where the parent application allocated the rdma-core objects.
> > > >
> > > > The newer kernels propose an option of copy-on-fork for DMA pages
> > > > and tracking all the memory and disabling it for the forking is no
> > > > longer needed. The new API routine ibv_is_fork_initialized()
> > > > should be involved to decide if library initialization for forking is
> required.
> > > >
> > > > Fixes: 0e83b8e536 ("net/mlx5: move rdma-core calls to separate
> > > > file")
> > > > Cc: stable at dpdk.org
> > > > Signed-off-by: Erez Ferber <erezf at nvidia.com>
> > >
> > Hi,
> >
> > > I don't think DPDK applications should fork(), and lots other parts
> > > of the shared huge pages will break if an application does this.
> >
> > I agree - application should not, we have the secondary/primary processes
> approach.
> > Nonetheless, we have the real use case - DPDK application does fork() and
> works well.
> > Without mlx5 PMD 😊. With mlx5 it ran into some troubles. Now we have
> the solution.
> 
> The problem is you are allowing fork(). And many other libraries may break.
> Imagine a DPDK library which has some local mutex and hugepages.
> If two forked processes use it then the locks won't work and hugepage data
> will be corrupted.

IMO, this problem is not fork() specific - applications/libraries supporting 
secondary/primary process approach should be developed with the similar
precautions.

And, IIRC, fork() is neither disallowed nor discouraged in DPDK documentation.
Moreover, some library unit tests ensure fork() works. No wonder users
may develop applications using fork(). They do, and app works well, beside mlx5 PMD.
Also, the recommendations to re-design with pri/sec approach was given.

With best regards,
Slava