[dpdk-dev] [PATCH v7 0/6] Reproducible build

[Bruce Richardson]

On Wed, Sep 06, 2017 at 10:10:34AM +0100, Luca Boccassi wrote:
> On Wed, 2017-09-06 at 09:46 +0100, Bruce Richardson wrote:
> > On Tue, Sep 05, 2017 at 10:15:29PM +0100, Luca Boccassi wrote:
> > > On Tue, 2017-09-05 at 14:00 -0700, Stephen Hemminger wrote:
> > > > On Tue,  5 Sep 2017 21:51:53 +0100
> > > > luca.boccassi at gmail.com wrote:
> > > > 
> > > > > From: Luca Boccassi <luca.boccassi at gmail.com>
> > > > > 
> > > > > In the past couple of years a concerted effort among almost all
> > > > > Linux
> > > > > distros has been striving toward achieving reproducible builds.
> > > > > [1]
> > > > > This involves changes to the toolchain, new tools and CI
> > > > > systems.
> > > > > [2]
> > > > > 
> > > > > v1 fixed the documentation, examples and linker script
> > > > > generation.
> > > > > v2 fixes all problems, which were caused by unstable order of
> > > > > headers
> > > > > inclusion, source files listing and object file listing when
> > > > > passing
> > > > > them to the compiler.
> > > > > DPDK's build, at least with the default configuration, is fully
> > > > > reproducible with this patch series as tested by the
> > > > > Reproducible
> > > > > Builds developers experimental toolchain. [3]
> > > > > 
> > > > > v3 restores the first patch, which was eaten by git send-email.
> > > > > 
> > > > > v4 drops the patch that reorders rebuilds, and adds a patch to
> > > > > make
> > > > > the inclusion of headers deterministic with regards to GCC
> > > > > embedding
> > > > > the full file path when expading __FILE__ and when writing the
> > > > > directory listing in the DWARF objects.
> > > > > It also drops the first 2 patches which have already been
> > > > > merged.
> > > > > 
> > > > > v5 adds the -I$(SRCDIR) workaround to librte_eal linuxapp's and
> > > > > librte_gro's Makefiles.
> > > > > 
> > > > > v6 fixes copypasta added in v5 - librte-gro's Makefile should
> > > > > have
> > > > > used
> > > > > $(SRCDIR) rather than $(SRCDIR)/include.
> > > > > 
> > > > > v7 dropped the big patch to change the CFLAGS in all libraries
> > > > > Makefiles
> > > > > and instead uses a much simpler approach suggested by Ferruh
> > > > > (thanks!),
> > > > > which consists in installing the library public headers
> > > > > symlinks
> > > > > before
> > > > > building. Although the drawback is that the DWARF directory
> > > > > listing
> > > > > will always include the user configured build output directory,
> > > > > it
> > > > > is
> > > > > a much smaller patch that will be rendered obsolete by the move
> > > > > to
> > > > > Meson
> > > > > anyway.
> > > > > 
> > > > > [1] https://reproducible-builds.org/
> > > > > [2] https://reproducible-builds.org/tools/
> > > > > [3] https://wiki.debian.org/ReproducibleBuilds/ExperimentalTool
> > > > > chai
> > > > > n#Us
> > > > > 
> > > > > Luca Boccassi (6):
> > > > >   mk: sort list of shared objects in linker script
> > > > >   mk: sort list of files in examples.dox
> > > > >   mk: sort headers before wildcard inclusion
> > > > >   mk: sort source files before passing them to the compiler
> > > > >   mk: sort object files when building deps lists
> > > > >   mk: install symlinks before build step
> > > > > 
> > > > >  drivers/net/cxgbe/Makefile                                 | 2
> > > > > +-
> > > > >  drivers/net/e1000/Makefile                                 | 2
> > > > > +-
> > > > >  drivers/net/fm10k/Makefile                                 | 2
> > > > > +-
> > > > >  drivers/net/i40e/Makefile                                  | 2
> > > > > +-
> > > > >  drivers/net/ixgbe/Makefile                                 | 2
> > > > > +-
> > > > >  drivers/net/qede/Makefile                                  | 2
> > > > > +-
> > > > >  drivers/net/sfc/Makefile                                   | 2
> > > > > +-
> > > > >  drivers/net/thunderx/Makefile                              | 2
> > > > > +-
> > > > >  examples/ip_pipeline/Makefile                              | 2
> > > > > +-
> > > > >  examples/multi_process/client_server_mp/mp_server/Makefile | 2
> > > > > +-
> > > > >  examples/server_node_efd/server/Makefile                   | 2
> > > > > +-
> > > > >  lib/librte_eal/common/Makefile                             | 2
> > > > > +-
> > > > >  mk/rte.app.mk                                              | 4
> > > > > ++-
> > > > > -
> > > > >  mk/rte.combinedlib.mk                                      | 2
> > > > > +-
> > > > >  mk/rte.hostapp.mk                                          | 4
> > > > > ++-
> > > > > -
> > > > >  mk/rte.lib.mk                                              | 7
> > > > > ++++++-
> > > > >  mk/rte.sdkdoc.mk                                           | 2
> > > > > +-
> > > > >  mk/rte.shared.mk                                           | 4
> > > > > ++-
> > > > > -
> > > > >  18 files changed, 26 insertions(+), 21 deletions(-)
> > > > > 
> > > > 
> > > > How does this interact with propose new build process.
> > > > Hate to have do everything twice.
> > > 
> > > Most of these patches, mainly for the custom bits that discover the
> > > source/headers files on the system (all the wildcards, etc) will
> > > not be
> > > necessary any longer.
> > > 
> > > The documentation build is not there yet in the patchset, so I'll
> > > check
> > > once that's available.
> > > 
> > > But I'm still testing the Meson patches, I had issues integrating
> > > it in
> > > the package builds as Meson support is still quite new in
> > > debhelper, I
> > > hope to finish that up next week and give feedback to Bruce.
> > > 
> > > The reproducible build testing infra I've got is picky and eats
> > > only
> > > deb packages so until I get that up and running I cannot test the
> > > reproducibility using Meson unfortunately.
> > > 
> > > -- 
> > 
> > It's also safer to assume that any replacement build system is going
> > to
> > take a while longer to complete, so it's as well to consider these
> > changes for integration, since they are not that large of patches.
> 
> I agree :-)
> 
> > Also, looking forward to your feedback on the meson build patches. I
> > was
> > going to do a v2 with minor updates, but I'll hold off until next
> > week
> > when I get your comments.
> > 
> > /Bruce
> 
> Ok, but please bear in mind that I'll be travelling from today to
> Monday, so it will probably be end of the week. So if needed feel free
> to send v2 already. Sorry for the delay!
> 
Ok, thanks for the info. I'll see how it goes and how many other changes
I might have to make. The only major issue right now is the one that you
already flagged about removing the check for a Kconfig file in the kernel.

/Bruce