Bug 1005 - Build failure with gcc11 in When Using Environment Modules
Summary: Build failure with gcc11 in When Using Environment Modules
Status: CONFIRMED
Alias: None
Product: DPDK
Classification: Unclassified
Component: meson (show other bugs)
Version: 22.03
Hardware: POWER Linux
: Normal normal
Target Milestone: ---
Assignee: dev
URL:
Depends on:
Blocks:
 
Reported: 2022-05-05 18:59 CEST by David Christensen
Modified: 2022-05-10 11:23 CEST (History)
1 user (show)



Attachments

Description David Christensen 2022-05-05 18:59:48 CEST
Install and enable IBM Advanced Toolchain 15.0 using environment modules:

$ module load at15.0
$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/opt/at15.0/libexec/gcc/powerpc64le-linux-gnu/11.2.1/lto-wrapper
Target: powerpc64le-linux-gnu
Configured with: /build/at15.0_RHEL8_ppc64le-ppc64le/114/at15.0-2.redhat-8_ppc64le_ppc64le/sources/gcc/configure --build=powerpc64le-linux-gnu --host=powerpc64le-linux-gnu --target=powerpc64le-linux-gnu --with-cpu=default64 --prefix=/opt/at15.0 --with-long-double-128 --enable-secureplt --disable-multilib --with-advance-toolchain=at15.0 --with-glibc-version=2.34 --with-local-prefix=/opt/at15.0 --enable-threads=posix --enable-languages=c,c++,fortran,go --enable-__cxa_atexit --enable-shared --enable-checking=release --enable-lto --enable-gnu-indirect-function --enable-initfini-array --enable-linker-build-id --with-system-zlib --with-gmp-include=/opt/at15.0/include --with-gmp-lib=/opt/at15.0/lib64 --with-mpfr-include=/opt/at15.0/include --with-mpfr-lib=/opt/at15.0/lib64 --with-mpc-include=/opt/at15.0/include --with-mpc-lib=/opt/at15.0/lib64 --without-ppl --without-cloog --without-libelf --with-host-libstdcxx='-L/opt/at15.0/lib64 -lstdc++ -lsupc++ -lgmp -lgmpxx -lm' --with-cpu=power8 --with-tune=power10
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 11.2.1 20220211 (Advance-Toolchain 15.0-2) [64575dfb22ae] (GCC)

Compile DPDK:

$ meson -v
0.61.2
$ meson build
$ ninja -C build

Observe link errors:

ninja: Entering directory `build'
[3324/3341] Linking target app/dpdk-test-acl
FAILED: app/dpdk-test-acl
cc  -o app/dpdk-test-acl app/dpdk-test-acl.p/test-acl_main.c.o -Wl,--as-needed -Wl,--no-undefined -Wl,-O1 -Wl,--whole-archive -Wl,--start-group lib/librte_node.a lib/librte_graph.a lib/librte_flow_classify.a lib/librte_pipeline.a lib/librte_table.a lib/librte_pdump.a lib/librte_port.a lib/librte_fib.a lib/librte_ipsec.a lib/librte_vhost.a lib/librte_stack.a lib/librte_security.a lib/librte_sched.a lib/librte_reorder.a lib/librte_rib.a lib/librte_dmadev.a lib/librte_regexdev.a lib/librte_rawdev.a lib/librte_power.a lib/librte_pcapng.a lib/librte_member.a lib/librte_lpm.a lib/librte_latencystats.a lib/librte_kni.a lib/librte_jobstats.a lib/librte_ip_frag.a lib/librte_gso.a lib/librte_gro.a lib/librte_gpudev.a lib/librte_eventdev.a lib/librte_efd.a lib/librte_distributor.a lib/librte_cryptodev.a lib/librte_compressdev.a lib/librte_cfgfile.a lib/librte_bpf.a lib/librte_bitratestats.a lib/librte_bbdev.a lib/librte_acl.a lib/librte_timer.a lib/librte_hash.a lib/librte_metrics.a lib/librte_cmdline.a lib/librte_pci.a lib/librte_ethdev.a lib/librte_meter.a lib/librte_net.a lib/librte_mbuf.a lib/librte_mempool.a lib/librte_rcu.a lib/librte_ring.a lib/librte_eal.a lib/librte_telemetry.a lib/librte_kvargs.a drivers/librte_common_cpt.a drivers/librte_common_dpaax.a drivers/librte_common_iavf.a drivers/librte_common_octeontx.a drivers/librte_bus_auxiliary.a drivers/librte_bus_dpaa.a drivers/librte_bus_fslmc.a drivers/librte_bus_ifpga.a drivers/librte_bus_pci.a drivers/librte_bus_vdev.a drivers/librte_bus_vmbus.a drivers/librte_common_cnxk.a drivers/librte_common_mlx5.a drivers/librte_common_qat.a drivers/librte_mempool_bucket.a drivers/librte_mempool_cnxk.a drivers/librte_mempool_dpaa.a drivers/librte_mempool_dpaa2.a drivers/librte_mempool_octeontx.a drivers/librte_mempool_ring.a drivers/librte_mempool_stack.a drivers/librte_dma_cnxk.a drivers/librte_dma_dpaa.a drivers/librte_dma_skeleton.a drivers/librte_net_af_packet.a drivers/librte_net_ark.a drivers/librte_net_atlantic.a drivers/librte_net_avp.a drivers/librte_net_axgbe.a drivers/librte_net_bnx2x.a drivers/librte_net_bnxt.a drivers/librte_net_bond.a drivers/librte_net_cnxk.a drivers/librte_net_cxgbe.a drivers/librte_net_dpaa.a drivers/librte_net_dpaa2.a drivers/librte_net_e1000.a drivers/librte_net_ena.a drivers/librte_net_enetc.a drivers/librte_net_enetfec.a drivers/librte_net_enic.a drivers/librte_net_failsafe.a drivers/librte_net_fm10k.a drivers/librte_net_hinic.a drivers/librte_net_i40e.a drivers/librte_net_iavf.a drivers/librte_net_ice.a drivers/librte_net_igc.a drivers/librte_net_ionic.a drivers/librte_net_ipn3ke.a drivers/librte_net_ixgbe.a drivers/librte_net_kni.a drivers/librte_net_liquidio.a drivers/librte_net_memif.a drivers/librte_net_mlx4.a drivers/librte_net_mlx5.a drivers/librte_net_netvsc.a drivers/librte_net_nfp.a drivers/librte_net_ngbe.a drivers/librte_net_null.a drivers/librte_net_octeontx.a drivers/librte_net_octeontx_ep.a drivers/librte_net_pcap.a drivers/librte_net_pfe.a drivers/librte_net_qede.a drivers/librte_net_ring.a drivers/librte_net_softnic.a drivers/librte_net_tap.a drivers/librte_net_thunderx.a drivers/librte_net_txgbe.a drivers/librte_net_vdev_netvsc.a drivers/librte_net_vhost.a drivers/librte_net_virtio.a drivers/librte_net_vmxnet3.a drivers/librte_raw_cnxk_bphy.a drivers/librte_raw_cnxk_gpio.a drivers/librte_raw_dpaa2_cmdif.a drivers/librte_raw_dpaa2_qdma.a drivers/librte_raw_ifpga.a drivers/librte_raw_ntb.a drivers/librte_raw_skeleton.a drivers/librte_crypto_bcmfs.a drivers/librte_crypto_caam_jr.a drivers/librte_crypto_ccp.a drivers/librte_crypto_cnxk.a drivers/librte_crypto_dpaa_sec.a drivers/librte_crypto_dpaa2_sec.a drivers/librte_crypto_mlx5.a drivers/librte_crypto_nitrox.a drivers/librte_crypto_null.a drivers/librte_crypto_octeontx.a drivers/librte_crypto_openssl.a drivers/librte_crypto_scheduler.a drivers/librte_crypto_virtio.a drivers/librte_compress_mlx5.a drivers/librte_compress_octeontx.a drivers/librte_compress_zlib.a drivers/librte_regex_mlx5.a drivers/librte_regex_cn9k.a drivers/librte_vdpa_ifc.a drivers/librte_vdpa_mlx5.a drivers/librte_event_cnxk.a drivers/librte_event_dpaa.a drivers/librte_event_dpaa2.a drivers/librte_event_dsw.a drivers/librte_event_opdl.a drivers/librte_event_skeleton.a drivers/librte_event_sw.a drivers/librte_event_octeontx.a drivers/librte_baseband_acc100.a drivers/librte_baseband_fpga_5gnr_fec.a drivers/librte_baseband_fpga_lte_fec.a drivers/librte_baseband_la12xx.a drivers/librte_baseband_null.a drivers/librte_baseband_turbo_sw.a -Wl,--no-whole-archive -Wl,--no-as-needed -pthread -lm -ldl -lnuma -lfdt -Wl,-rpath,/lib/../lib64:XXXXXXXXXXXXXXXXXX -Wl,-rpath-link,/lib/../lib64 -Wl,--export-dynamic /usr/lib64/libpcap.so /usr/lib64/libelf.so /lib/../lib64/libmlx5.so /lib/../lib64/libibverbs.so /usr/lib64/libcrypto.so /usr/lib64/libz.so /lib/../lib64/libmlx4.so -lrt -Wl,--end-group
/opt/at15.0/lib/gcc/powerpc64le-linux-gnu/11.2.1/../../../../powerpc64le-linux-gnu/bin/ld: /lib/../lib64/libpthread.so.0: undefined reference to `__libc_dlopen_mode@GLIBC_PRIVATE'
/opt/at15.0/lib/gcc/powerpc64le-linux-gnu/11.2.1/../../../../powerpc64le-linux-gnu/bin/ld: /lib/../lib64/libpthread.so.0: undefined reference to `__nanosleep_nocancel@GLIBC_PRIVATE'
/opt/at15.0/lib/gcc/powerpc64le-linux-gnu/11.2.1/../../../../powerpc64le-linux-gnu/bin/ld: /lib/../lib64/libpthread.so.0: undefined reference to `__libc_current_sigrtmin_private@GLIBC_PRIVATE'
/opt/at15.0/lib/gcc/powerpc64le-linux-gnu/11.2.1/../../../../powerpc64le-linux-gnu/bin/ld: /lib/../lib64/libpthread.so.0: undefined reference to `__libc_dlclose@GLIBC_PRIVATE'
/opt/at15.0/lib/gcc/powerpc64le-linux-gnu/11.2.1/../../../../powerpc64le-linux-gnu/bin/ld: /lib/../lib64/libpthread.so.0: undefined reference to `__libc_siglongjmp@GLIBC_PRIVATE'
/opt/at15.0/lib/gcc/powerpc64le-linux-gnu/11.2.1/../../../../powerpc64le-linux-gnu/bin/ld: /lib/../lib64/libdl.so.2: undefined reference to `_dl_sym@GLIBC_PRIVATE'
/opt/at15.0/lib/gcc/powerpc64le-linux-gnu/11.2.1/../../../../powerpc64le-linux-gnu/bin/ld: /lib/../lib64/libpthread.so.0: undefined reference to `__libc_longjmp@GLIBC_PRIVATE'
/opt/at15.0/lib/gcc/powerpc64le-linux-gnu/11.2.1/../../../../powerpc64le-linux-gnu/bin/ld: /lib/../lib64/libpthread.so.0: undefined reference to `__libc_allocate_rtsig_private@GLIBC_PRIVATE'
/opt/at15.0/lib/gcc/powerpc64le-linux-gnu/11.2.1/../../../../powerpc64le-linux-gnu/bin/ld: /lib/../lib64/libpthread.so.0: undefined reference to `__libc_thread_freeres@GLIBC_PRIVATE'
/opt/at15.0/lib/gcc/powerpc64le-linux-gnu/11.2.1/../../../../powerpc64le-linux-gnu/bin/ld: /lib/../lib64/libpthread.so.0: undefined reference to `__libc_dlsym@GLIBC_PRIVATE'
/opt/at15.0/lib/gcc/powerpc64le-linux-gnu/11.2.1/../../../../powerpc64le-linux-gnu/bin/ld: /lib/../lib64/libpthread.so.0: undefined reference to `__libc_current_sigrtmax_private@GLIBC_PRIVATE'
/opt/at15.0/lib/gcc/powerpc64le-linux-gnu/11.2.1/../../../../powerpc64le-linux-gnu/bin/ld: /lib/../lib64/libpthread.so.0: undefined reference to `__pause_nocancel@GLIBC_PRIVATE'
/opt/at15.0/lib/gcc/powerpc64le-linux-gnu/11.2.1/../../../../powerpc64le-linux-gnu/bin/ld: /lib/../lib64/libdl.so.2: undefined reference to `_dl_addr@GLIBC_PRIVATE'
/opt/at15.0/lib/gcc/powerpc64le-linux-gnu/11.2.1/../../../../powerpc64le-linux-gnu/bin/ld: /lib/../lib64/libdl.so.2: undefined reference to `_dl_vsym@GLIBC_PRIVATE'
/opt/at15.0/lib/gcc/powerpc64le-linux-gnu/11.2.1/../../../../powerpc64le-linux-gnu/bin/ld: /lib/../lib64/libpthread.so.0: undefined reference to `__libc_pthread_init@GLIBC_PRIVATE'
/opt/at15.0/lib/gcc/powerpc64le-linux-gnu/11.2.1/../../../../powerpc64le-linux-gnu/bin/ld: /lib/../lib64/libpthread.so.0: undefined reference to `_dl_make_stack_executable@GLIBC_PRIVATE'
collect2: error: ld returned 1 exit status
...

Issue appears to be related to glibc 2.34 reorganization (https://developers.redhat.com/articles/2021/12/17/why-glibc-234-removed-libpthread#the_developer_view) and library path enabled by meson (-Wl,-rpath-link,/lib/../lib64).
Comment 1 David Christensen 2022-05-09 23:33:57 CEST
I can resolve the build problem with the following steps:

$ meson build
$ cd build
$ sed --in-place=.BAK1 's|-Wl,-rpath,/lib/../lib64||' build.ninja
$ sed --in-place=.BAK2 's|-Wl,-rpath-link,/lib/../lib64||' build.ninja
$ cd ..
$ ninja -C build
ninja: Entering directory `build'
[3341/3341] Linking target app/test/dpdk-test
$

It's not clear to me where the rpath/rpath-link arguments are being added.
Comment 2 Bruce Richardson 2022-05-10 11:23:13 CEST
The rpath entries are automatically added by meson to the build so that the binaries generated can be run directly from the build directory. When "ninja install" is run, these extra rpaths are removed.

From https://mesonbuild.com/Release-notes-for-0-55-0.html 
"On Linux-like systems, Meson adds rpath entries to allow running apps in the build tree, and then removes those build-time-only rpath entries when installing."

Note You need to log in before you can comment on or make changes to this bug.