ci: hook to Github Actions

  With the recent changes in terms of free access to the Travis CI, let's
offer an alternative with Github Actions.
Running jobs on ARM is not supported unless using external runners, so
this commit only adds builds for x86_64 and cross compiling for i386 and
aarch64.

Differences with the Travis CI integration:
- All jobs generate documentation.
  This is not that heavy and the default timeout on actions is never
  reached so no reason splitting this into multiple jobs.
- Error logs are not dumped to the console when something goes wrong.
  Instead, they are gathered in a "catch-all" step and attached as
  artifacts.
- A cache entry is stored once and for all, but if no cache is found you
  can inherit from the default branch cache. The cache is 5GB large, for
  the whole git repository.
- The maximum retention of logs and artifacts is 3 months.
- /home/runner is world writable, so a workaround has been added for
  starting dpdk processes.

Signed-off-by: David Marchand <david.marchand@redhat.com>
---
 .ci/linux-build.sh          |  4 +-
 .github/workflows/build.yml | 98 +++++++++++++++++++++++++++++++++++++
 MAINTAINERS                 |  1 +
 3 files changed, 102 insertions(+), 1 deletion(-)
 create mode 100644 .github/workflows/build.yml
  

David Marchand <david.marchand@redhat.com> writes:

> With the recent changes in terms of free access to the Travis CI, let's
> offer an alternative with Github Actions.
> Running jobs on ARM is not supported unless using external runners, so
> this commit only adds builds for x86_64 and cross compiling for i386 and
> aarch64.
>
> Differences with the Travis CI integration:
> - All jobs generate documentation.
>   This is not that heavy and the default timeout on actions is never
>   reached so no reason splitting this into multiple jobs.
> - Error logs are not dumped to the console when something goes wrong.
>   Instead, they are gathered in a "catch-all" step and attached as
>   artifacts.
> - A cache entry is stored once and for all, but if no cache is found you
>   can inherit from the default branch cache. The cache is 5GB large, for
>   the whole git repository.
> - The maximum retention of logs and artifacts is 3 months.
> - /home/runner is world writable, so a workaround has been added for
>   starting dpdk processes.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---

Thanks for working on this.  Sadly, I think we will have to abandon
Travis soon - given the new changes it is looking very awful.  Robot
already is starved for job time.

Since we don't have ARM test runs, I guess we will have to rely on
something else for that coverage now, but I like that there is coverage
included at least to compile.

I will need to update the robot to pull information from github actions,
so for now it will need to be manually checked (but here's an example of
a run: https://github.com/ovsrobot/dpdk/actions/runs/382073265).  What's
nice is the robot is already primed to run the jobs, so that's good.

Acked-by: Aaron Conole <aconole@redhat.com>

>  .ci/linux-build.sh          |  4 +-
>  .github/workflows/build.yml | 98 +++++++++++++++++++++++++++++++++++++
>  MAINTAINERS                 |  1 +
>  3 files changed, 102 insertions(+), 1 deletion(-)
>  create mode 100644 .github/workflows/build.yml
>
> diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh
> index d079801d78..a2a0e5bf42 100755
> --- a/.ci/linux-build.sh
> +++ b/.ci/linux-build.sh
> @@ -12,7 +12,9 @@ on_error() {
>          fi
>      done
>  }
> -trap on_error EXIT
> +# We capture the error logs as artifacts in Github Actions, no need to dump
> +# them via a EXIT handler.
> +[ -n "$GITHUB_WORKFLOW" ] || trap on_error EXIT
>  
>  install_libabigail() {
>      version=$1
> diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
> new file mode 100644
> index 0000000000..e0a8f1ed52
> --- /dev/null
> +++ b/.github/workflows/build.yml
> @@ -0,0 +1,98 @@
> +name: build
> +
> +on: push
> +
> +defaults:
> +  run:
> +    shell: bash --noprofile --norc -exo pipefail {0}
> +
> +jobs:
> +  build:
> +    name: ${{ join(matrix.config.*, '-') }}
> +    runs-on: ${{ matrix.config.os }}
> +    env:
> +      PKGS: |
> +        ccache libnuma-dev python3-setuptools python3-wheel python3-pip \
> +        ninja-build libbsd-dev libpcap-dev libibverbs-dev libcrypto++-dev \
> +        libfdt-dev libjansson-dev doxygen graphviz python3-sphinx \
> +        python3-sphinx-rtd-theme
> +      CC: ccache ${{ matrix.config.compiler }}
> +      JOBNAME: ${{ join(matrix.config.*, '-') }}
> +
> +    strategy:
> +      fail-fast: false
> +      matrix:
> +        config:
> +          - os: ubuntu-18.04
> +            compiler: gcc
> +            library: static
> +          - os: ubuntu-18.04
> +            compiler: gcc
> +            library: shared
> +          - os: ubuntu-18.04
> +            compiler: clang
> +            library: static
> +          - os: ubuntu-18.04
> +            compiler: clang
> +            library: shared
> +          - os: ubuntu-18.04
> +            compiler: gcc
> +            library: static
> +            cross: i386
> +          - os: ubuntu-18.04
> +            compiler: gcc
> +            library: static
> +            cross: aarch64
> +          - os: ubuntu-18.04
> +            compiler: gcc
> +            library: shared
> +            cross: aarch64
> +
> +    steps:
> +    - uses: actions/checkout@v2
> +    - uses: actions/cache@v2
> +      with:
> +        path: ~/.ccache
> +        key: ${{ env.JOBNAME }}-${{ github.ref }}
> +        restore-keys: |
> +          ${{ env.JOBNAME }}-refs/heads/main
> +    - name: Install packages
> +      run: sudo apt install -y ${{ env.PKGS }}
> +    - name: Install i386 cross compiling packages
> +      if: matrix.config.cross == 'i386'
> +      run: sudo apt install -y gcc-multilib
> +    - name: Install aarch64 cross compiling packages
> +      if: matrix.config.cross == 'aarch64'
> +      run: |
> +        sudo apt install -y gcc-aarch64-linux-gnu libc6-dev-arm64-cross \
> +          pkg-config-aarch64-linux-gnu
> +    - name: Prepare environment
> +      run: |
> +         .ci/linux-setup.sh
> +         # Workaround on $HOME permissions as EAL checks them for plugin loading
> +         chmod o-w $HOME
> +    - name: Build and test
> +      run: |
> +        export DEF_LIB=${{ matrix.config.library }}
> +        export BUILD_DOCS=1
> +        case '${{ matrix.config.cross }}' in
> +        'i386')
> +            export BUILD_32BIT=1
> +        ;;
> +        'aarch64')
> +            export AARCH64=1
> +        ;;
> +        '')
> +            export RUN_TESTS=1
> +        ;;
> +        esac
> +        .ci/linux-build.sh
> +    - name: Upload logs on failure
> +      if: failure()
> +      uses: actions/upload-artifact@v2
> +      with:
> +        name: meson-logs-${{ env.JOBNAME }}
> +        path: |
> +          build/meson-logs/testlog.txt
> +          build/.ninja_log
> +          build/meson-logs/meson-log.txt
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 214515060a..95b61085b7 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -109,6 +109,7 @@ Public CI
>  M: Aaron Conole <aconole@redhat.com>
>  M: Michael Santana <maicolgabriel@hotmail.com>
>  F: .travis.yml
> +F: .github/workflows/build.yml
>  F: .ci/
>  
>  ABI Policy & Versioning
  

On Wed, Nov 25, 2020 at 2:45 PM Aaron Conole <aconole@redhat.com> wrote:
> Thanks for working on this.  Sadly, I think we will have to abandon
> Travis soon - given the new changes it is looking very awful.  Robot
> already is starved for job time.
>
> Since we don't have ARM test runs, I guess we will have to rely on
> something else for that coverage now, but I like that there is coverage
> included at least to compile.

For ARM test runs, UNH is a good candidate but nothing prevents other
ARM based CI from being added.


> I will need to update the robot to pull information from github actions,
> so for now it will need to be manually checked (but here's an example of
> a run: https://github.com/ovsrobot/dpdk/actions/runs/382073265).  What's
> nice is the robot is already primed to run the jobs, so that's good.

Thanks.
I added a bookmark to https://github.com/ovsrobot/dpdk/actions for now.
  

<snip>

> 
> On Wed, Nov 25, 2020 at 2:45 PM Aaron Conole <aconole@redhat.com> wrote:
> > Thanks for working on this.  Sadly, I think we will have to abandon
> > Travis soon - given the new changes it is looking very awful.  Robot
> > already is starved for job time.
I am looking at [1], is DPDK not considered as open source project?

[1] https://blog.travis-ci.com/oss-announcement

> >
> > Since we don't have ARM test runs, I guess we will have to rely on
> > something else for that coverage now, but I like that there is
> > coverage included at least to compile.
> 
> For ARM test runs, UNH is a good candidate but nothing prevents other ARM
> based CI from being added.
Is it possible to keep Travis CI for Arm?

> 
> 
> > I will need to update the robot to pull information from github
> > actions, so for now it will need to be manually checked (but here's an
> > example of a run:
> > https://github.com/ovsrobot/dpdk/actions/runs/382073265).  What's nice is
> the robot is already primed to run the jobs, so that's good.
Is there any guarantee that GitHub actions will be free forever?

> 
> Thanks.
> I added a bookmark to https://github.com/ovsrobot/dpdk/actions for now.
> 
> 
> --
> David Marchand
  

On Thu, Nov 26, 2020 at 5:47 AM Honnappa Nagarahalli
<Honnappa.Nagarahalli@arm.com> wrote:
> > On Wed, Nov 25, 2020 at 2:45 PM Aaron Conole <aconole@redhat.com> wrote:
> > > Thanks for working on this.  Sadly, I think we will have to abandon
> > > Travis soon - given the new changes it is looking very awful.  Robot
> > > already is starved for job time.
> I am looking at [1], is DPDK not considered as open source project?
>
> [1] https://blog.travis-ci.com/oss-announcement

Ilya (@OVS) contacted the Travis support.
The reply is that a project that has sponsored contributors can not
ask for free tokens on Travis CI.

I personally did not try to contact their support given this response.


> > > Since we don't have ARM test runs, I guess we will have to rely on
> > > something else for that coverage now, but I like that there is
> > > coverage included at least to compile.
> >
> > For ARM test runs, UNH is a good candidate but nothing prevents other ARM
> > based CI from being added.
> Is it possible to keep Travis CI for Arm?

For individuals, the 10k credits with the current DPDK jobs get burned
in something like 4 runs (read: 4 runs a month).
Even if we narrow the configuration to only ARM, this will at best
give us x3, so let's say 12 runs a month.

Now consider the ovsrobot and the number of series that hit the list
on a worst^Wbest day like a week before rc1.


> > > I will need to update the robot to pull information from github
> > > actions, so for now it will need to be manually checked (but here's an
> > > example of a run:
> > > https://github.com/ovsrobot/dpdk/actions/runs/382073265).  What's nice is
> > the robot is already primed to run the jobs, so that's good.
> Is there any guarantee that GitHub actions will be free forever?

There is no "forever".
At least, UNH lab which the project sponsors seems viable on the mid/long term.
  

<snip>

> 
> On Thu, Nov 26, 2020 at 5:47 AM Honnappa Nagarahalli
> <Honnappa.Nagarahalli@arm.com> wrote:
> > > On Wed, Nov 25, 2020 at 2:45 PM Aaron Conole <aconole@redhat.com>
> wrote:
> > > > Thanks for working on this.  Sadly, I think we will have to
> > > > abandon Travis soon - given the new changes it is looking very
> > > > awful.  Robot already is starved for job time.
> > I am looking at [1], is DPDK not considered as open source project?
> >
> > [1] https://blog.travis-ci.com/oss-announcement
> 
> Ilya (@OVS) contacted the Travis support.
> The reply is that a project that has sponsored contributors can not ask for free
> tokens on Travis CI.
> 
> I personally did not try to contact their support given this response.
> 
> 
> > > > Since we don't have ARM test runs, I guess we will have to rely on
> > > > something else for that coverage now, but I like that there is
> > > > coverage included at least to compile.
> > >
> > > For ARM test runs, UNH is a good candidate but nothing prevents
> > > other ARM based CI from being added.
> > Is it possible to keep Travis CI for Arm?
> 
> For individuals, the 10k credits with the current DPDK jobs get burned in
> something like 4 runs (read: 4 runs a month).
> Even if we narrow the configuration to only ARM, this will at best give us x3, so
> let's say 12 runs a month.
> 
> Now consider the ovsrobot and the number of series that hit the list on a
> worst^Wbest day like a week before rc1.
> 
> 
> > > > I will need to update the robot to pull information from github
> > > > actions, so for now it will need to be manually checked (but
> > > > here's an example of a run:
> > > > https://github.com/ovsrobot/dpdk/actions/runs/382073265).  What's
> > > > nice is
> > > the robot is already primed to run the jobs, so that's good.
> > Is there any guarantee that GitHub actions will be free forever?
> 
> There is no "forever".
I think we are spending our efforts on things that will not work for the community in the long run (unless the project spends money to buy credits)

> At least, UNH lab which the project sponsors seems viable on the mid/long term.
Yes, agree, all the more reason to continue to maintain the lab.

> 
> 
> --
> David Marchand
  

On Thu, Nov 26, 2020 at 6:01 PM Honnappa Nagarahalli
<Honnappa.Nagarahalli@arm.com> wrote:
> > > Is there any guarantee that GitHub actions will be free forever?
> >
> > There is no "forever".
> I think we are spending our efforts on things that will not work for the community in the long run (unless the project spends money to buy credits)

That was not the initial goal of the patch, but GHA can be used by
developers who work on their github forks too, like for testing before
submitting to the public ml.

On spending efforts, the lab should be the priority.
My main concern was to get ABI checks back quickly since 21.02
proposals started to hit the list.
A ticket has been opened for the lab to handle this, but this can take
some time so in the interim we have GHA support.

I sent the v2 with ABI checks.


--
David Marchand