[dpdk-dev] [PATCH] validate_abi: build faster by augmenting make with job count

[Neil Horman]

On Wed, Jul 20, 2016 at 07:47:32PM +0000, Wiles, Keith wrote:
> 
> > On Jul 20, 2016, at 12:48 PM, Neil Horman <nhorman at redhat.com> wrote:
> > 
> > On Wed, Jul 20, 2016 at 07:40:49PM +0200, Thomas Monjalon wrote:
> >> 2016-07-20 13:09, Neil Horman:
> >>> From: Neil Horman <nhorman at redhat.com>
> >>> 
> >>> John Mcnamara and I were discussing enhacing the validate_abi script to build
> >>> the dpdk tree faster with multiple jobs.  Theres no reason not to do it, so this
> >>> implements that requirement.  It uses a MAKE_JOBS variable that can be set by
> >>> the user to limit the job count.  By default the job count is set to the number
> >>> of online cpus.
> >> 
> >> Please could you use the variable name DPDK_MAKE_JOBS?
> >> This name is already used in scripts/test-build.sh.
> >> 
> > Sure
> > 
> >>> +if [ -z "$MAKE_JOBS" ]
> >>> +then
> >>> +	# This counts the number of cpus on the system
> >>> +	MAKE_JOBS=`lscpu -p=cpu | grep -v "#" | wc -l`
> >>> +fi
> >> 
> >> Is lscpu common enough?
> >> 
> > I'm not sure how to answer that.  lscpu is part of the util-linux package, which
> > is part of any base install.  Theres a variant for BSD, but I'm not sure how
> > common it is there.
> > Neil
> > 
> >> Another acceptable default would be just "-j" without any number.
> >> It would make the number of jobs unlimited.
> 
> I think the best is just use -j as it tries to use the correct number of jobs based on the number of cores, right?
> 
-j with no argument (or -j 0), is sort of, maybe what you want.  With either of
those options, make will just issue jobs as fast as it processes dependencies.
Dependent on how parallel the build is, that can lead to tons of waiting process
(i.e. more than your number of online cpus), which can actually hurt your build
time.

While its fine in los of cases, its not always fine, and with this
implementation you can still opt in to that behavior by setting DPDK_MAKE_JOBS=0

Neil

>