[dpdk-dev] [RFC PATCH] replace DPDK config and build system

[Van Haaren, Harry]

> From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of Bruce Richardson
> Sent: Wednesday, June 7, 2017 11:48 AM
> To: dev at dpdk.org
> Cc: Richardson, Bruce <bruce.richardson at intel.com>
> Subject: [dpdk-dev] [RFC PATCH] replace DPDK config and build system
> 
> Hi all,
> 
> following on from the pressing need to add support in DPDK for detecting
> and managing external dependencies, I undertook to see what options we had.
> However, unrelated to this, over time, I have become increasingly
> frustrated by the complexity of the DPDK configuration and build system. As
> such, I feel that looking at some of the newer build tools that are out
> there might give us the additional functionality we want, along with other
> benefits. As such, I undertook a prototype using "meson"[1] for
> configuration, which uses "ninja" as a backend for doing the actual build.
> 
> With these tools we get the following benefits (not a complete list):
> * support for detecting dependencies on the system
> * support for detecting compiler features, including functions, defines
> * improved maintainability through a high-level language, which gives
>   decent error messages including line numbers (!!)
> * co-existence with the existing makefile system without making any changes
>   to it
> * faster builds using ninja - on my many-core system, the builds seem
>   significantly faster than with our existing system. Especially in the
>   nothing-has-changed case, builds with my prototype return instantly,
>   compared to taking a few seconds to recursively check each directory with
>   the current build system
> * the ability to switch to using a standard "ninja" + "ninja install" setup
> * the chance to rework our existing build-config files, and hopefully
>   pretty much remove them.
> * pkg-config support.
> * we get to move away from our bespoke build system
> * dependencies in each lib can be moved back to being tracked in the libs
>   files themselves, not up a level
> 
> 
> Of course, it's not a panacea, but having spent hours on the prototype thus
> far, I find working with meson and ninja far more user-friendly than
> working on our makefiles, and again the build speed is a really nice
> improvment too.
> 
> The prototype is incomplete, but it does build a reasonable number of our
> libraries, some unit tests, the i40e PMD and the testpmd binary, and I have
> successfully passed traffic using testpmd from the build. Some things are
> not fully correct, e.g. static builds aren't working right now, as I haven't
> correctly done all the dependency tracking, I think, and the cpu flag
> detection has issues. It also has only been tried on x86_64 linux, on a
> couple of systems, so YMMV. However, I feel it's a reasonable enough start
> point to show what we might be able to achieve.
> 
> Please take the prototype and test it out. I think it's a better
> alternative to trying to bolt on additional functionality to our existing
> config and build system.


Test drive done; here are my experiences / thoughts:

1) Understanding the Meson build files is much easier for me than the current build-files. I'll admit that bash scripting is not my forte, with the caveat that if my basic bash scripting doesn't suffice, others in the community are probably in a similar position.

2) I see huge value in pkg-config integration - linking made easy. From a developer usability POV, a project should provide a pkg-config file, and configure itself based on the pkg-config files available on the system. 

3) The speed of ninja is impressive - particularly in the small-amounts-of-work case. Try it on your own machine if you don't believe me :)

4) "build variants" are super easy, meson build && meson build_variant  will create two directories, with .o files contained under it (same as one possible usage of the current system - just calling out that that feature remains).

5) Vim users, there is a "mesonic" plugin[1] which provides syntax highlighting for Meson files, the Meson Options file, and Meson integration to the Vim compiler hooks. Aka typing  :make   will cause Ninja to be invoked and compile. There are more fancy features, check the script page.

6) Ninja uninstall will reverse a previous installation, and returns a clean system. Nice for testing and integrating DPDK with applications that expect DPDK to be installed system-wide.


Of course no build system is perfect - and I'd guess Meson also has a few gritty details that are not ideal - but to me this test-drive has shown promise. Opinions from non-developers may be of particular interest..

@DPDK Packagers, do you have experience with Meson? Does it integrate well with (your) existing infrastructure? Hope you don't mind me adding you to CC :)

-Harry

[1] http://www.vim.org/scripts/script.php?script_id=5378