[PATCH v4] devtools: parallelize ABI check

[Ferruh Yigit]

On 1/18/2023 10:45 AM, David Marchand wrote:
> On Thu, Jan 12, 2023 at 3:15 PM Ferruh Yigit <ferruh.yigit at amd.com> wrote:
>>>>         abidiff $ABIDIFF_OPTIONS $dump $dump2 || {
>>>>                 abiret=$?
>>>> -               echo "Error: ABI issue reported for 'abidiff $ABIDIFF_OPTIONS $dump $dump2'" >&2
>>>> -               error=1
>>>> -               echo
>>>> +               echo "Error: ABI issue reported for abidiff $ABIDIFF_OPTIONS $dump $dump2" >&2
>>>>                 if [ $(($abiret & 3)) -ne 0 ]; then
>>>>                         echo "ABIDIFF_ERROR|ABIDIFF_USAGE_ERROR, this could be a script or environment issue." >&2
>>>>                 fi
>>>> @@ -57,8 +55,13 @@ for dump in $(find $refdir -name "*.dump"); do
>>>>                 if [ $(($abiret & 8)) -ne 0 ]; then
>>>>                         echo "ABIDIFF_ABI_INCOMPATIBLE_CHANGE, this change breaks the ABI." >&2
>>>>                 fi
>>>> -               echo
>>>> +               return 1
>>>>         }
>>>> -done
>>>> +}'
>>>> +
>>>> +error=
>>>> +find $refdir -name "*.dump" |
>>>> +xargs -n1 -P0 sh -c 'eval "$diff_func"; run_diff $0' ||
>>>> +error=1
>>>>
>>>>  [ -z "$error" ] || [ -n "$warnonly" ]
>>>
>>> For the record, on my system, calling this script is ~5 times faster:
>>> - before
>>> real    0m5,447s
>>> user    0m4,497s
>>> sys    0m0,937s
>>>
>>> - after
>>> real    0m1,202s
>>> user    0m10,784s
>>> sys    0m2,027s
>>>
>>>
>>>> diff --git a/devtools/gen-abi.sh b/devtools/gen-abi.sh
>>>> index f15a3b9aaf..61f7510ea1 100755
>>>> --- a/devtools/gen-abi.sh
>>>> +++ b/devtools/gen-abi.sh
>>>> @@ -22,5 +22,6 @@ for f in $(find $installdir -name "*.so.*"); do
>>>>         fi
>>>>
>>>>         libname=$(basename $f)
>>>> -       abidw --out-file $dumpdir/${libname%.so*}.dump $f
>>>> -done
>>>> +       echo $dumpdir/${libname%.so*}.dump $f
>>>> +done |
>>>> +xargs -n2 -P0 abidw --out-file
>>>> --
>>>> 2.39.0
>>>>
>>>
>>> - before
>>> real    0m8,237s
>>> user    0m7,704s
>>> sys    0m0,504s
>>>
>>> - after
>>> real    0m2,517s
>>> user    0m14,145s
>>> sys    0m0,766s
>>>
>>>
>>> Ferruh, I am seeing quite different numbers for running those scripts
>>> (clearly not of the minute order).
>>> I switched to testing/building in tmpfs some time ago.
>>> It requires a good amount of memory (I empirically allocated 40G), but
>>> maybe worth a try for you?
>>>
>>
>> I run 'test-meson-builds.sh' script directly and yes I am getting
>> different numbers although there is still improvement, not in scale with
>> what you are getting, with v4 I have following:
>>
>> - before
>> real    10m3.248s
>> user    39m8.664s
>> sys     14m52.870s
>>
>> - after
>> real    7m49.086s
>> user    39m59.507s
>> sys     15m0.598s
> 
> Well, yes, I did not realise which apples you were looking at :-).
> The change looks good in any case.
> 

Ack