[v5,2/2] ci: Introduce travis builds for github repositories

  GitHub is a service used by developers to store repositories.  GitHub
provides service integrations that allow 3rd party services to access
developer repositories and perform actions.  One of these services is
Travis-CI, a simple continuous integration platform.

This is a simple initial implementation of a travis build for the DPDK
project.  It doesn't require any changes from individual developers to
enable, but will allow those developers who opt-in to GitHub and the
travis service to get automatic builds for every push they make.

Additionally, the travis service will send an email to the test-report
list informing anyone interested in the automated build (including a
result).

Signed-off-by: Aaron Conole <aconole@redhat.com>
Signed-off-by: Michael Santana <msantana@redhat.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
---
v5:
  Renamed ARM64 to AARCH64

 .ci/linux-build.sh                  |  88 +++++++++++++++
 .ci/linux-setup.sh                  |  31 ++++++
 .travis.yml                         | 159 ++++++++++++++++++++++++++++
 MAINTAINERS                         |   6 ++
 doc/guides/contributing/patches.rst |   4 +
 5 files changed, 288 insertions(+)
 create mode 100755 .ci/linux-build.sh
 create mode 100755 .ci/linux-setup.sh
 create mode 100644 .travis.yml
  

Hi,

07/02/2019 23:01, Michael Santana:
> GitHub is a service used by developers to store repositories.  GitHub
> provides service integrations that allow 3rd party services to access
> developer repositories and perform actions.  One of these services is
> Travis-CI, a simple continuous integration platform.
> 
> This is a simple initial implementation of a travis build for the DPDK
> project.  It doesn't require any changes from individual developers to
> enable, but will allow those developers who opt-in to GitHub and the
> travis service to get automatic builds for every push they make.
> 
> Additionally, the travis service will send an email to the test-report
> list informing anyone interested in the automated build (including a
> result).
> 
> Signed-off-by: Aaron Conole <aconole@redhat.com>
> Signed-off-by: Michael Santana <msantana@redhat.com>
> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>

I think it is a good idea to provide some recipes for some well known CI.

I have some concerns about the organization of what is generic/specific
and about the automation of reports.
Detailed comments below:

> --- /dev/null
> +++ b/.ci/linux-build.sh
> +# Just used for the 'classic' configuration system (ie: make)

I am not sure about supporting the legacy system in a new CI.

> +set_conf() {
> +    echo "[BUILT WITH $2 SET TO $3]"

In order to improve readability, I suggest to document the parameters
on the first line. Example:

set_conf() { # <dir> <key> <value>

> +    c="$1/.config"
> +    shift
> +
> +    if grep -q "$1" "$c"; then
> +        sed -i "s:^$1=.*$:$1=$2:g" $c

You don't need the "g" as there is only one option per line.

> +    else
> +        echo $1=$2 >> "$c"

The else branch is useless because .config includes all options.

> +    fi
> +}
> +
> +BUILD_ARCH="x86_64-native-linuxapp-"

We could have some native Arm compilation.

> +
> +if [ "${AARCH64}" == "1" ]; then
> +    # convert the arch specifier
> +    BUILD_ARCH="arm64-armv8a-linuxapp-"
> +    AARCH64_TOOL="linaro-arm-tool"

What is this directory? It looks really specific to Travis.

> +    export PATH=$PATH:$(pwd)/${AARCH64_TOOL}/bin
> +fi
> +
> +
> +if [ "${NINJABUILD}" == "1" ]; then
> +    OPTS=""
> +
> +    DEF_LIB="static"
> +    if [ "${SHARED}" == "1" ]; then
> +        DEF_LIB="shared"
> +    fi
> +
> +    if [ "${DISABLE_KERNEL_MODULES}" == "1" ]; then
> +        OPTS="-Denable_kmods=false"
> +    fi

Isn't it possible to directly provide the meson options in travis.yml
instead of doing a translation with new option names?

> +
> +    if [ "${AARCH64}" == "1" ]; then
> +        OPTS="${OPTS} --cross-file config/arm/arm64_armv8_linuxapp_gcc"
> +    fi
> +
> +    OPTS="$OPTS --default-library=$DEF_LIB"
> +    meson build --werror -Dexamples=all ${OPTS}
> +    ninja -C build
> +else
> +    EXTRA_OPTS=""
> +
> +    make config T="${BUILD_ARCH}${CC}"
> +
> +    set_conf build CONFIG_RTE_KNI_KMOD n
> +    set_conf build CONFIG_RTE_EAL_IGB_UIO n

Why these options are fixed?

> +
> +    if dpkg --list | grep -q zlib1g ; then
> +        set_conf build CONFIG_RTE_LIBRTE_PMD_ZLIB y
> +    fi

This is Debian-specific.

> +
> +    if dpkg --list | grep -q libpcap-dev ; then
> +        set_conf build CONFIG_RTE_PORT_PCAP y
> +    fi
> +
> +    if [ "${SHARED}" == "1" ]; then
> +        set_conf build CONFIG_RTE_BUILD_SHARED_LIB y
> +    fi
> +
> +    if [ "${DISABLE_KERNEL_MODULES}" == "1" ]; then
> +        echo Unsupported kernel builds at the moment
> +    fi
> +
> +    if [ "${AARCH64}" == "1" ]; then
> +        EXTRA_OPTS="CROSS=aarch64-linux-gnu-"
> +
> +        # need to turn off these extras
> +        set_conf build CONFIG_RTE_PORT_PCAP n
> +        set_conf build CONFIG_RTE_LIBRTE_PMD_ZLIB n
> +
> +        # convert the CC/CXX variables
> +        export CC=aarch64-linux-gnu-${CC}
> +        export CXX=aarch64-linux-gnu-${CXX}
> +        export AR=aarch64-linux-gnu-ar
> +        export STRIP=aarch64-linux-gnu-strip
> +    fi
> +
> +    make all ${EXTRA_OPTS}
> +fi
> diff --git a/.ci/linux-setup.sh b/.ci/linux-setup.sh
> new file mode 100755
> index 000000000..8dfd5aa49
> --- /dev/null
> +++ b/.ci/linux-setup.sh
> @@ -0,0 +1,31 @@
> +#!/bin/bash
> +
> +python3.5 -m pip install --upgrade meson --user

Which distributions have python3.5?
It looks very specific.

> +
> +echo "AARCH64 is [ ${AARCH64} ]"
> +
> +if [ "${AARCH64}" == "1" ]; then
> +    # need to build & install libnuma

Why is it needed? linbnuma is optional.
I think this file can be dropped or renamed to
something like install-libnuma-for-cross-arm.sh

> +    # This will only be minimal support for now.
> +    AARCH64_TOOL_URL='https://releases.linaro.org/components/toolchain/binaries/latest-7/aarch64-linux-gnu/gcc-linaro-7.4.1-2019.02-x86_64_aarch64-linux-gnu.tar.xz'
> +    AARCH64_TOOL="linaro-arm-tool"
> +    NUMA_GIT_URL="https://github.com/numactl/numactl.git"
> +
> +    wget -O "${AARCH64_TOOL}.tar.xz" "${AARCH64_TOOL_URL}"
> +    tar -xf "${AARCH64_TOOL}.tar.xz"
> +    mv gcc-linaro* "${AARCH64_TOOL}"
> +    export PATH=$PATH:$(pwd)/${AARCH64_TOOL}/bin
> +    git clone "${NUMA_GIT_URL}"
> +    cd numactl
> +    git checkout v2.0.11
> +    ./autogen.sh
> +    autoconf -i
> +    mkdir numa_bin
> +    ./configure --host=aarch64-linux-gnu CC=aarch64-linux-gnu-gcc \
> +                --prefix=$(pwd)/numa_bin
> +    make install # install numa
> +    cd ..
> +    cp numactl/numa_bin/include/numa*.h "${AARCH64_TOOL}/aarch64-linux-gnu/libc/usr/include/"
> +    cp numactl/numa_bin/lib/libnuma.* "${AARCH64_TOOL}/aarch64-linux-gnu/lib64/"
> +    cp numactl/numa_bin/lib/libnuma.* "${AARCH64_TOOL}/lib/"
> +fi
> diff --git a/.travis.yml b/.travis.yml
> new file mode 100644
> index 000000000..0faa35770
> --- /dev/null
> +++ b/.travis.yml
> @@ -0,0 +1,159 @@
> +language: c
> +compiler:
> +  - gcc
> +  - clang
> +
> +os:
> +  - linux
> +
> +addons:
> +  apt:
> +    sources:
> +      - deadsnakes  #Repo for python 3.5
> +      - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
> +    packages:
> +      - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip, ninja-build]
> +
> +before_install: ./.ci/${TRAVIS_OS_NAME}-setup.sh
> +
> +sudo: false
> +
> +env:
> +  - SHARED=1
> +  - DISABLE_KERNEL_MODULES=1
> +  - SHARED=1 DISABLE_KERNEL_MODULES=1
> +  - NINJABUILD=1
> +  - NINJABUILD=1 SHARED=1
> +  - NINJABUILD=1 DISABLE_KERNEL_MODULES=1
> +  - NINJABUILD=1 SHARED=1 DISABLE_KERNEL_MODULES=1
> +
> +matrix:
> +  include:
> +  - env: SHARED=1
> +    compiler: gcc
> +    addons:
> +      apt:
> +        sources:
> +          - deadsnakes  #Repo for python 3.5
> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
> +        packages:
> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
> +          - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip, ninja-build]
> +  - env: DISABLE_KERNEL_MODULES=1
> +    compiler: gcc
> +    addons:
> +      apt:
> +        sources:
> +          - deadsnakes  #Repo for python 3.5
> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
> +        packages:
> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
> +          - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip, ninja-build]
> +  - env: SHARED=1 DISABLE_KERNEL_MODULES=1
> +    compiler: gcc
> +    addons:
> +      apt:
> +        sources:
> +          - deadsnakes  #Repo for python 3.5
> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
> +        packages:
> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
> +          - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip, ninja-build]
> +  - env: SHARED=1
> +    compiler: clang
> +    addons:
> +      apt:
> +        sources:
> +          - deadsnakes  #Repo for python 3.5
> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
> +        packages:
> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
> +          - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip, ninja-build]
> +  - env: DISABLE_KERNEL_MODULES=1
> +    compiler: clang
> +    addons:
> +      apt:
> +        sources:
> +          - deadsnakes  #Repo for python 3.5
> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
> +        packages:
> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
> +          - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip, ninja-build]
> +  - env: SHARED=1 DISABLE_KERNEL_MODULES=1
> +    compiler: clang
> +    addons:
> +      apt:
> +        sources:
> +          - deadsnakes  #Repo for python 3.5
> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
> +        packages:
> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
> +          - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip, ninja-build]
> +  - env: AARCH64=1
> +    compiler: gcc
> +    addons:
> +      apt:
> +        sources:
> +          - deadsnakes  #Repo for python 3.5
> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
> +        packages:
> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
> +          - [libnuma-dev, linux-headers-$(uname -r), libtool, python3.5, python3-pip]
> +  - env: AARCH64=1 NINJABUILD=1
> +    compiler: gcc
> +    addons:
> +      apt:
> +        sources:
> +          - deadsnakes  #Repo for python 3.5
> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
> +        packages:
> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
> +          - [linux-headers-$(uname -r), libtool, python3.5, python3-pip, ninja-build]
> +  - env: NINJABUILD=1
> +    compiler: gcc
> +    addons:
> +      apt:
> +        sources:
> +          - deadsnakes  #Repo for python 3.5
> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
> +        packages:
> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
> +          - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip, ninja-build]
> +  - env: NINJABUILD=1 SHARED=1
> +    compiler: gcc
> +    addons:
> +      apt:
> +        sources:
> +          - deadsnakes  #Repo for python 3.5
> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
> +        packages:
> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
> +          - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip, ninja-build]
> +  - env: NINJABUILD=1 DISABLE_KERNEL_MODULES=1
> +    compiler: gcc
> +    addons:
> +      apt:
> +        sources:
> +          - deadsnakes  #Repo for python 3.5
> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
> +        packages:
> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
> +          - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip, ninja-build]
> +  - env: NINJABUILD=1 SHARED=1 DISABLE_KERNEL_MODULES=1
> +    compiler: gcc
> +    addons:
> +      apt:
> +        sources:
> +          - deadsnakes  #Repo for python 3.5
> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
> +        packages:
> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
> +          - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip, ninja-build]
> +
> +
> +script: ./.ci/${TRAVIS_OS_NAME}-build.sh
> +
> +notifications:
> +  email:
> +    recipients:
> +      - test-report@dpdk.org

The idea of this mailing list is to receive reports about
the upstream development. When doing a private development,
reports should not be sent. How can it be disabled?
  

Thomas Monjalon <thomas@monjalon.net> writes:

> Hi,
>
> 07/02/2019 23:01, Michael Santana:
>> GitHub is a service used by developers to store repositories.  GitHub
>> provides service integrations that allow 3rd party services to access
>> developer repositories and perform actions.  One of these services is
>> Travis-CI, a simple continuous integration platform.
>> 
>> This is a simple initial implementation of a travis build for the DPDK
>> project.  It doesn't require any changes from individual developers to
>> enable, but will allow those developers who opt-in to GitHub and the
>> travis service to get automatic builds for every push they make.
>> 
>> Additionally, the travis service will send an email to the test-report
>> list informing anyone interested in the automated build (including a
>> result).
>> 
>> Signed-off-by: Aaron Conole <aconole@redhat.com>
>> Signed-off-by: Michael Santana <msantana@redhat.com>
>> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
>> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
>
> I think it is a good idea to provide some recipes for some well known CI.
>
> I have some concerns about the organization of what is generic/specific
> and about the automation of reports.
> Detailed comments below:
>
>> --- /dev/null
>> +++ b/.ci/linux-build.sh
>> +# Just used for the 'classic' configuration system (ie: make)
>
> I am not sure about supporting the legacy system in a new CI.

For now, documentation all says that the legacy system is the supported
system.  I think it's appropriate to continue to support it until such
time as it is eliminated.  Otherwise, when an end user builds we don't
know what to say about support - IE: if they have problems with the
classic system for whatever reason, do we tell them "sorry, we cannot
help"?

The patches.rst mentions that all patches must pass with the Makefile
system, and the contributing/documentation.rst calls it the "standard
DPDK build system."  If you want to change those things to reflect
something different please do, and we can drop all of the stuff related
to it, but until that time we won't.

>> +set_conf() {
>> +    echo "[BUILT WITH $2 SET TO $3]"
>
> In order to improve readability, I suggest to document the parameters
> on the first line. Example:
>
> set_conf() { # <dir> <key> <value>
>
>> +    c="$1/.config"
>> +    shift
>> +
>> +    if grep -q "$1" "$c"; then
>> +        sed -i "s:^$1=.*$:$1=$2:g" $c
>
> You don't need the "g" as there is only one option per line.
>
>> +    else
>> +        echo $1=$2 >> "$c"
>
> The else branch is useless because .config includes all options.

Okay.

>> +    fi
>> +}
>> +
>> +BUILD_ARCH="x86_64-native-linuxapp-"
>
> We could have some native Arm compilation.

Sure.  Is this just commentary?  Do you suggest a change here?  This is
a default, and will be adjusted later by other parameters.

>> +
>> +if [ "${AARCH64}" == "1" ]; then
>> +    # convert the arch specifier
>> +    BUILD_ARCH="arm64-armv8a-linuxapp-"
>> +    AARCH64_TOOL="linaro-arm-tool"
>
> What is this directory? It looks really specific to Travis.

It's specific to the AARCH64 toolchain that was pulled in as part of
linux-prepare.sh - do you think something should change?

>> +    export PATH=$PATH:$(pwd)/${AARCH64_TOOL}/bin
>> +fi
>> +
>> +
>> +if [ "${NINJABUILD}" == "1" ]; then
>> +    OPTS=""
>> +
>> +    DEF_LIB="static"
>> +    if [ "${SHARED}" == "1" ]; then
>> +        DEF_LIB="shared"
>> +    fi
>> +
>> +    if [ "${DISABLE_KERNEL_MODULES}" == "1" ]; then
>> +        OPTS="-Denable_kmods=false"
>> +    fi
>
> Isn't it possible to directly provide the meson options in travis.yml
> instead of doing a translation with new option names?

Yes and no.  It would be possible, but we try to support both build
systems and they have different options afaict.  So we need something
common.  Maybe we missed something?

>> +
>> +    if [ "${AARCH64}" == "1" ]; then
>> +        OPTS="${OPTS} --cross-file config/arm/arm64_armv8_linuxapp_gcc"
>> +    fi
>> +
>> +    OPTS="$OPTS --default-library=$DEF_LIB"
>> +    meson build --werror -Dexamples=all ${OPTS}
>> +    ninja -C build
>> +else
>> +    EXTRA_OPTS=""
>> +
>> +    make config T="${BUILD_ARCH}${CC}"
>> +
>> +    set_conf build CONFIG_RTE_KNI_KMOD n
>> +    set_conf build CONFIG_RTE_EAL_IGB_UIO n
>
> Why these options are fixed?

There was a problem with the Travis system when trying to build some of
the kernel modules, but I don't remember the details.  Maybe Michael does.

>> +
>> +    if dpkg --list | grep -q zlib1g ; then
>> +        set_conf build CONFIG_RTE_LIBRTE_PMD_ZLIB y
>> +    fi
>
> This is Debian-specific.

Will address this in v6.

>> +
>> +    if dpkg --list | grep -q libpcap-dev ; then
>> +        set_conf build CONFIG_RTE_PORT_PCAP y
>> +    fi
>> +
>> +    if [ "${SHARED}" == "1" ]; then
>> +        set_conf build CONFIG_RTE_BUILD_SHARED_LIB y
>> +    fi
>> +
>> +    if [ "${DISABLE_KERNEL_MODULES}" == "1" ]; then
>> +        echo Unsupported kernel builds at the moment
>> +    fi
>> +
>> +    if [ "${AARCH64}" == "1" ]; then
>> +        EXTRA_OPTS="CROSS=aarch64-linux-gnu-"
>> +
>> +        # need to turn off these extras
>> +        set_conf build CONFIG_RTE_PORT_PCAP n
>> +        set_conf build CONFIG_RTE_LIBRTE_PMD_ZLIB n
>> +
>> +        # convert the CC/CXX variables
>> +        export CC=aarch64-linux-gnu-${CC}
>> +        export CXX=aarch64-linux-gnu-${CXX}
>> +        export AR=aarch64-linux-gnu-ar
>> +        export STRIP=aarch64-linux-gnu-strip
>> +    fi
>> +
>> +    make all ${EXTRA_OPTS}
>> +fi
>> diff --git a/.ci/linux-setup.sh b/.ci/linux-setup.sh
>> new file mode 100755
>> index 000000000..8dfd5aa49
>> --- /dev/null
>> +++ b/.ci/linux-setup.sh
>> @@ -0,0 +1,31 @@
>> +#!/bin/bash
>> +
>> +python3.5 -m pip install --upgrade meson --user
>
> Which distributions have python3.5?
> It looks very specific.

I agree, could probably just be python3 we need to check and see if we
can just use that.

But, we did need the upgrade.

Travis environment comes up with ubuntu 14.04, which includes python3.4,
and the requisite version of meson needs python3.5 or higher.

>> +
>> +echo "AARCH64 is [ ${AARCH64} ]"
>> +
>> +if [ "${AARCH64}" == "1" ]; then
>> +    # need to build & install libnuma
>
> Why is it needed? linbnuma is optional.
> I think this file can be dropped or renamed to
> something like install-libnuma-for-cross-arm.sh

We needed this because it broke the meson build when cross compiling,
IIRC.  I will investigate it further to be sure.

>> +    # This will only be minimal support for now.
>> +
>> AARCH64_TOOL_URL='https://releases.linaro.org/components/toolchain/binaries/latest-7/aarch64-linux-gnu/gcc-linaro-7.4.1-2019.02-x86_64_aarch64-linux-gnu.tar.xz'
>> +    AARCH64_TOOL="linaro-arm-tool"
>> +    NUMA_GIT_URL="https://github.com/numactl/numactl.git"
>> +
>> +    wget -O "${AARCH64_TOOL}.tar.xz" "${AARCH64_TOOL_URL}"
>> +    tar -xf "${AARCH64_TOOL}.tar.xz"
>> +    mv gcc-linaro* "${AARCH64_TOOL}"
>> +    export PATH=$PATH:$(pwd)/${AARCH64_TOOL}/bin
>> +    git clone "${NUMA_GIT_URL}"
>> +    cd numactl
>> +    git checkout v2.0.11
>> +    ./autogen.sh
>> +    autoconf -i
>> +    mkdir numa_bin
>> +    ./configure --host=aarch64-linux-gnu CC=aarch64-linux-gnu-gcc \
>> +                --prefix=$(pwd)/numa_bin
>> +    make install # install numa
>> +    cd ..
>> + cp numactl/numa_bin/include/numa*.h
>> "${AARCH64_TOOL}/aarch64-linux-gnu/libc/usr/include/"
>> +    cp numactl/numa_bin/lib/libnuma.* "${AARCH64_TOOL}/aarch64-linux-gnu/lib64/"
>> +    cp numactl/numa_bin/lib/libnuma.* "${AARCH64_TOOL}/lib/"
>> +fi
>> diff --git a/.travis.yml b/.travis.yml
>> new file mode 100644
>> index 000000000..0faa35770
>> --- /dev/null
>> +++ b/.travis.yml
>> @@ -0,0 +1,159 @@
>> +language: c
>> +compiler:
>> +  - gcc
>> +  - clang
>> +
>> +os:
>> +  - linux
>> +
>> +addons:
>> +  apt:
>> +    sources:
>> +      - deadsnakes  #Repo for python 3.5
>> +      - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
>> +    packages:
>> +      - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip, ninja-build]
>> +
>> +before_install: ./.ci/${TRAVIS_OS_NAME}-setup.sh
>> +
>> +sudo: false
>> +
>> +env:
>> +  - SHARED=1
>> +  - DISABLE_KERNEL_MODULES=1
>> +  - SHARED=1 DISABLE_KERNEL_MODULES=1
>> +  - NINJABUILD=1
>> +  - NINJABUILD=1 SHARED=1
>> +  - NINJABUILD=1 DISABLE_KERNEL_MODULES=1
>> +  - NINJABUILD=1 SHARED=1 DISABLE_KERNEL_MODULES=1
>> +
>> +matrix:
>> +  include:
>> +  - env: SHARED=1
>> +    compiler: gcc
>> +    addons:
>> +      apt:
>> +        sources:
>> +          - deadsnakes  #Repo for python 3.5
>> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
>> +        packages:
>> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
>> + - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip,
>> ninja-build]
>> +  - env: DISABLE_KERNEL_MODULES=1
>> +    compiler: gcc
>> +    addons:
>> +      apt:
>> +        sources:
>> +          - deadsnakes  #Repo for python 3.5
>> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
>> +        packages:
>> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
>> + - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip,
>> ninja-build]
>> +  - env: SHARED=1 DISABLE_KERNEL_MODULES=1
>> +    compiler: gcc
>> +    addons:
>> +      apt:
>> +        sources:
>> +          - deadsnakes  #Repo for python 3.5
>> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
>> +        packages:
>> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
>> + - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip,
>> ninja-build]
>> +  - env: SHARED=1
>> +    compiler: clang
>> +    addons:
>> +      apt:
>> +        sources:
>> +          - deadsnakes  #Repo for python 3.5
>> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
>> +        packages:
>> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
>> + - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip,
>> ninja-build]
>> +  - env: DISABLE_KERNEL_MODULES=1
>> +    compiler: clang
>> +    addons:
>> +      apt:
>> +        sources:
>> +          - deadsnakes  #Repo for python 3.5
>> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
>> +        packages:
>> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
>> + - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip,
>> ninja-build]
>> +  - env: SHARED=1 DISABLE_KERNEL_MODULES=1
>> +    compiler: clang
>> +    addons:
>> +      apt:
>> +        sources:
>> +          - deadsnakes  #Repo for python 3.5
>> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
>> +        packages:
>> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
>> + - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip,
>> ninja-build]
>> +  - env: AARCH64=1
>> +    compiler: gcc
>> +    addons:
>> +      apt:
>> +        sources:
>> +          - deadsnakes  #Repo for python 3.5
>> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
>> +        packages:
>> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
>> +          - [libnuma-dev, linux-headers-$(uname -r), libtool, python3.5, python3-pip]
>> +  - env: AARCH64=1 NINJABUILD=1
>> +    compiler: gcc
>> +    addons:
>> +      apt:
>> +        sources:
>> +          - deadsnakes  #Repo for python 3.5
>> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
>> +        packages:
>> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
>> +          - [linux-headers-$(uname -r), libtool, python3.5, python3-pip, ninja-build]
>> +  - env: NINJABUILD=1
>> +    compiler: gcc
>> +    addons:
>> +      apt:
>> +        sources:
>> +          - deadsnakes  #Repo for python 3.5
>> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
>> +        packages:
>> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
>> + - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip,
>> ninja-build]
>> +  - env: NINJABUILD=1 SHARED=1
>> +    compiler: gcc
>> +    addons:
>> +      apt:
>> +        sources:
>> +          - deadsnakes  #Repo for python 3.5
>> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
>> +        packages:
>> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
>> + - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip,
>> ninja-build]
>> +  - env: NINJABUILD=1 DISABLE_KERNEL_MODULES=1
>> +    compiler: gcc
>> +    addons:
>> +      apt:
>> +        sources:
>> +          - deadsnakes  #Repo for python 3.5
>> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
>> +        packages:
>> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
>> + - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip,
>> ninja-build]
>> +  - env: NINJABUILD=1 SHARED=1 DISABLE_KERNEL_MODULES=1
>> +    compiler: gcc
>> +    addons:
>> +      apt:
>> +        sources:
>> +          - deadsnakes  #Repo for python 3.5
>> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
>> +        packages:
>> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
>> + - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip,
>> ninja-build]
>> +
>> +
>> +script: ./.ci/${TRAVIS_OS_NAME}-build.sh
>> +
>> +notifications:
>> +  email:
>> +    recipients:
>> +      - test-report@dpdk.org
>
> The idea of this mailing list is to receive reports about
> the upstream development. When doing a private development,
> reports should not be sent. How can it be disabled?

There are a few variables related to controlling this that we can
investigate so that we only alert on the main repository, if you want.

We could also just remove it.  Not a problem either way.
  

27/02/2019 15:35, Aaron Conole:
> Thomas Monjalon <thomas@monjalon.net> writes:
> > 07/02/2019 23:01, Michael Santana:
> >> +# Just used for the 'classic' configuration system (ie: make)
> >
> > I am not sure about supporting the legacy system in a new CI.
> 
> For now, documentation all says that the legacy system is the supported
> system.  I think it's appropriate to continue to support it until such
> time as it is eliminated.  Otherwise, when an end user builds we don't
> know what to say about support - IE: if they have problems with the
> classic system for whatever reason, do we tell them "sorry, we cannot
> help"?

It is tested by others and me.

> The patches.rst mentions that all patches must pass with the Makefile
> system, and the contributing/documentation.rst calls it the "standard
> DPDK build system."  If you want to change those things to reflect
> something different please do, and we can drop all of the stuff related
> to it, but until that time we won't.

If it is forcing to have a translation of options, I think it is better
to skip the legacy system in this CI.

> >> +BUILD_ARCH="x86_64-native-linuxapp-"
> >
> > We could have some native Arm compilation.
> 
> Sure.  Is this just commentary?  Do you suggest a change here?  This is
> a default, and will be adjusted later by other parameters.

OK, I missed it is the default. Maybe a comment would help.

> >> +if [ "${AARCH64}" == "1" ]; then
> >> +    # convert the arch specifier
> >> +    BUILD_ARCH="arm64-armv8a-linuxapp-"
> >> +    AARCH64_TOOL="linaro-arm-tool"
> >
> > What is this directory? It looks really specific to Travis.
> 
> It's specific to the AARCH64 toolchain that was pulled in as part of
> linux-prepare.sh - do you think something should change?

I think it should be a parameter somewhere else. In the .yml file?

> >> +    DEF_LIB="static"
> >> +    if [ "${SHARED}" == "1" ]; then
> >> +        DEF_LIB="shared"
> >> +    fi
> >> +
> >> +    if [ "${DISABLE_KERNEL_MODULES}" == "1" ]; then
> >> +        OPTS="-Denable_kmods=false"
> >> +    fi
> >
> > Isn't it possible to directly provide the meson options in travis.yml
> > instead of doing a translation with new option names?
> 
> Yes and no.  It would be possible, but we try to support both build
> systems and they have different options afaict.  So we need something
> common.  Maybe we missed something?

It was my understanding.
That's why I think we should drop the legacy support
and focus on a simpler meson support.

> >> +    set_conf build CONFIG_RTE_KNI_KMOD n
> >> +    set_conf build CONFIG_RTE_EAL_IGB_UIO n
> >
> > Why these options are fixed?
> 
> There was a problem with the Travis system when trying to build some of
> the kernel modules, but I don't remember the details.  Maybe Michael does.

OK, please add a comment if kept.

> >> +python3.5 -m pip install --upgrade meson --user
> >
> > Which distributions have python3.5?
> > It looks very specific.
> 
> I agree, could probably just be python3 we need to check and see if we
> can just use that.
> 
> But, we did need the upgrade.
> 
> Travis environment comes up with ubuntu 14.04, which includes python3.4,
> and the requisite version of meson needs python3.5 or higher.

It could be a python --version check then?

> >> +if [ "${AARCH64}" == "1" ]; then
> >> +    # need to build & install libnuma
> >
> > Why is it needed? linbnuma is optional.
> > I think this file can be dropped or renamed to
> > something like install-libnuma-for-cross-arm.sh
> 
> We needed this because it broke the meson build when cross compiling,
> IIRC.  I will investigate it further to be sure.

OK thanks

> >> +notifications:
> >> +  email:
> >> +    recipients:
> >> +      - test-report@dpdk.org
> >
> > The idea of this mailing list is to receive reports about
> > the upstream development. When doing a private development,
> > reports should not be sent. How can it be disabled?
> 
> There are a few variables related to controlling this that we can
> investigate so that we only alert on the main repository, if you want.
> 
> We could also just remove it.  Not a problem either way.

I think it should be disabled by default,
and documented how to enable it.

Then we may need a way to filter which travis report should be allowed
in the mailing list, in order to avoid flooding it.
I think we cannot filter by sender. Do we need procmail?
  

Thomas Monjalon <thomas@monjalon.net> writes:

> 27/02/2019 15:35, Aaron Conole:
>> Thomas Monjalon <thomas@monjalon.net> writes:
>> > 07/02/2019 23:01, Michael Santana:
>> >> +# Just used for the 'classic' configuration system (ie: make)
>> >
>> > I am not sure about supporting the legacy system in a new CI.
>> 
>> For now, documentation all says that the legacy system is the supported
>> system.  I think it's appropriate to continue to support it until such
>> time as it is eliminated.  Otherwise, when an end user builds we don't
>> know what to say about support - IE: if they have problems with the
>> classic system for whatever reason, do we tell them "sorry, we cannot
>> help"?
>
> It is tested by others and me.

I test it, too. :)

>> The patches.rst mentions that all patches must pass with the Makefile
>> system, and the contributing/documentation.rst calls it the "standard
>> DPDK build system."  If you want to change those things to reflect
>> something different please do, and we can drop all of the stuff related
>> to it, but until that time we won't.
>
> If it is forcing to have a translation of options, I think it is better
> to skip the legacy system in this CI.

We can do that.

>> >> +BUILD_ARCH="x86_64-native-linuxapp-"
>> >
>> > We could have some native Arm compilation.
>> 
>> Sure.  Is this just commentary?  Do you suggest a change here?  This is
>> a default, and will be adjusted later by other parameters.
>
> OK, I missed it is the default. Maybe a comment would help.

Okay.

>> >> +if [ "${AARCH64}" == "1" ]; then
>> >> +    # convert the arch specifier
>> >> +    BUILD_ARCH="arm64-armv8a-linuxapp-"
>> >> +    AARCH64_TOOL="linaro-arm-tool"
>> >
>> > What is this directory? It looks really specific to Travis.
>> 
>> It's specific to the AARCH64 toolchain that was pulled in as part of
>> linux-prepare.sh - do you think something should change?
>
> I think it should be a parameter somewhere else. In the .yml file?
>
>> >> +    DEF_LIB="static"
>> >> +    if [ "${SHARED}" == "1" ]; then
>> >> +        DEF_LIB="shared"
>> >> +    fi
>> >> +
>> >> +    if [ "${DISABLE_KERNEL_MODULES}" == "1" ]; then
>> >> +        OPTS="-Denable_kmods=false"
>> >> +    fi
>> >
>> > Isn't it possible to directly provide the meson options in travis.yml
>> > instead of doing a translation with new option names?
>> 
>> Yes and no.  It would be possible, but we try to support both build
>> systems and they have different options afaict.  So we need something
>> common.  Maybe we missed something?
>
> It was my understanding.
> That's why I think we should drop the legacy support
> and focus on a simpler meson support.

Okay.

>> >> +    set_conf build CONFIG_RTE_KNI_KMOD n
>> >> +    set_conf build CONFIG_RTE_EAL_IGB_UIO n
>> >
>> > Why these options are fixed?
>> 
>> There was a problem with the Travis system when trying to build some of
>> the kernel modules, but I don't remember the details.  Maybe Michael does.
>
> OK, please add a comment if kept.

I think the new plan is to drop it.

>> >> +python3.5 -m pip install --upgrade meson --user
>> >
>> > Which distributions have python3.5?
>> > It looks very specific.
>> 
>> I agree, could probably just be python3 we need to check and see if we
>> can just use that.
>> 
>> But, we did need the upgrade.
>> 
>> Travis environment comes up with ubuntu 14.04, which includes python3.4,
>> and the requisite version of meson needs python3.5 or higher.
>
> It could be a python --version check then?

Well, I don't know what the purpose of the version check would be.  We
can tailor the environments.  Just need to pass the right information,
that's all.  I think we'll make it look nicer, though.  We can do any
environment specific fix ups in the prepare scripts, which I think makes
sense.  Then the build script can be a bit more generic.

>> >> +if [ "${AARCH64}" == "1" ]; then
>> >> +    # need to build & install libnuma
>> >
>> > Why is it needed? linbnuma is optional.
>> > I think this file can be dropped or renamed to
>> > something like install-libnuma-for-cross-arm.sh
>> 
>> We needed this because it broke the meson build when cross compiling,
>> IIRC.  I will investigate it further to be sure.
>
> OK thanks
>
>> >> +notifications:
>> >> +  email:
>> >> +    recipients:
>> >> +      - test-report@dpdk.org
>> >
>> > The idea of this mailing list is to receive reports about
>> > the upstream development. When doing a private development,
>> > reports should not be sent. How can it be disabled?
>> 
>> There are a few variables related to controlling this that we can
>> investigate so that we only alert on the main repository, if you want.
>> 
>> We could also just remove it.  Not a problem either way.
>
> I think it should be disabled by default,
> and documented how to enable it.
>
> Then we may need a way to filter which travis report should be allowed
> in the mailing list, in order to avoid flooding it.
> I think we cannot filter by sender. Do we need procmail?

Maybe it would make sense to have a build list that we can use?  Then we
just let that absorb all the various build from different CI services.
Just thinking out loud.  That way, anyone interested in the build status
can look there, and it keeps the test-report clean.  Again, just a
half-baked idea.
  

On Wed, 2019-02-27 at 10:53 -0500, Aaron Conole wrote:
> Thomas Monjalon <thomas@monjalon.net> writes:
> 
> > 27/02/2019 15:35, Aaron Conole:
> > > Thomas Monjalon <thomas@monjalon.net> writes:
> > > > 07/02/2019 23:01, Michael Santana:
> > > > > +python3.5 -m pip install --upgrade meson --user
> > > > 
> > > > Which distributions have python3.5?
> > > > It looks very specific.
> > > 
> > > I agree, could probably just be python3 we need to check and see
> > > if we
> > > can just use that.
> > > 
> > > But, we did need the upgrade.
> > > 
> > > Travis environment comes up with ubuntu 14.04, which includes
> > > python3.4,
> > > and the requisite version of meson needs python3.5 or higher.
> > 
> > It could be a python --version check then?
> 
> Well, I don't know what the purpose of the version check would
> be.  We
> can tailor the environments.  Just need to pass the right
> information,
> that's all.  I think we'll make it look nicer, though.  We can do any
> environment specific fix ups in the prepare scripts, which I think
> makes
> sense.  Then the build script can be a bit more generic.

Travis supports Ubuntu 16.04 images now, which have python 3.5, I'd
suggest to just use them (dist: xenial) so you can skip all the checks
and the manual installations.
  

Luca Boccassi <bluca@debian.org> writes:

> On Wed, 2019-02-27 at 10:53 -0500, Aaron Conole wrote:
>> Thomas Monjalon <thomas@monjalon.net> writes:
>> 
>> > 27/02/2019 15:35, Aaron Conole:
>> > > Thomas Monjalon <thomas@monjalon.net> writes:
>> > > > 07/02/2019 23:01, Michael Santana:
>> > > > > +python3.5 -m pip install --upgrade meson --user
>> > > > 
>> > > > Which distributions have python3.5?
>> > > > It looks very specific.
>> > > 
>> > > I agree, could probably just be python3 we need to check and see
>> > > if we
>> > > can just use that.
>> > > 
>> > > But, we did need the upgrade.
>> > > 
>> > > Travis environment comes up with ubuntu 14.04, which includes
>> > > python3.4,
>> > > and the requisite version of meson needs python3.5 or higher.
>> > 
>> > It could be a python --version check then?
>> 
>> Well, I don't know what the purpose of the version check would
>> be.  We
>> can tailor the environments.  Just need to pass the right
>> information,
>> that's all.  I think we'll make it look nicer, though.  We can do any
>> environment specific fix ups in the prepare scripts, which I think
>> makes
>> sense.  Then the build script can be a bit more generic.
>
> Travis supports Ubuntu 16.04 images now, which have python 3.5, I'd
> suggest to just use them (dist: xenial) so you can skip all the checks
> and the manual installations.

Great news, thanks, Luca!