HIP module doesn't provide a working rpath or LD_LIBRARY_PATH for libamdhip64.so.6 at runtime (accel/amd/gfx90a, 2025.06)

[alexschroeter]

This has been created with the help of AI

Environment

• EESSI version: 2025.06
• Accelerator target: accel/amd/gfx90a (AMD Instinct MI210)
• EESSI_ACCELERATOR_TARGET_OVERRIDE=accel/amd/gfx90a
• Module: HIP/6.4.1-rocm-compilers-19.0.0-ROCm-6.4.1 (via accel/amd/gfx90a MODULEPATH)

Description

A binary compiled with hipcc --offload-arch=gfx90a against the EESSI-provided
HIP module fails to run because the dynamic linker cannot locate
libamdhip64.so.6 at runtime. Neither LD_LIBRARY_PATH nor a working rpath
points at the directory containing it, even though the module sets
LIBRARY_PATH (used at link time) and exports HIP_LIB_PATH.

I raised this originally as a comment on easybuild-easyconfigs#23542; @Thyre
asked that it be re-verified against the actual EESSI-merged stack (rather
than that draft PR) and filed here instead, since PR comments get lost. I've
now reproduced it directly against the production 2025.06 CVMFS repo.

I understand EESSI's general policy is to rely on rpath rather than
LD_LIBRARY_PATH for this kind of thing — so the bug as I see it isn't
"LD_LIBRARY_PATH should be set," it's that neither mechanism currently
works for binaries built against HIP outside of the EESSI build
environment (i.e. as an end user compiling their own HIP code on top of the
loaded module).

Steps to reproduce

export EESSI_ACCELERATOR_TARGET_OVERRIDE=accel/amd/gfx90a
source /cvmfs/software.eessi.io/versions/2025.06/init/bash
module load HIP
hipcc --offload-arch=gfx90a vecadd.cpp -o vecadd
./vecadd

Expected behavior

The resulting binary runs without needing any manual environment changes
beyond what module load HIP already sets up.

Actual behavior

./vecadd fails at startup because libamdhip64.so.6 cannot be found by the
dynamic linker.

Workaround

export LD_LIBRARY_PATH=$HIP_LIB_PATH:$LD_LIBRARY_PATH

This gets the linker past the libamdhip64.so.6 error (the module does
export HIP_LIB_PATH, it's just not consulted by default), but on its own it
does not produce a working binary — it immediately hits a second, separate
bug: a CXXABI_1.3.15 not found error from gflags/Z3 (pulled in via
comgr) caused by a DT_RPATH/DT_RUNPATH interaction that makes the dynamic
linker fall through to the compat layer's own ld.so.cache and pick up an
older, incompatible libstdc++. Full details in this comment.
The actual full workaround needed to get a running binary is:

export LD_LIBRARY_PATH=/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/GCCcore/14.2.0/lib64:$HIP_LIB_PATH:$LD_LIBRARY_PATH

I'm keeping this issue focused on the LD_LIBRARY_PATH/module problem, since
the libstdc++ issue is a distinct bug with a different root cause — just
flagging it here so anyone verifying this report doesn't get confused when
the "simple" workaround alone still fails.

Suggested fix directions

• Either have the HIP module set LD_LIBRARY_PATH (if rpath-only is not
  achievable for externally-compiled code), or
• Ensure hipcc/the HIP compiler driver embeds a correct rpath (e.g.
  pointing at $HIP_LIB_PATH) into binaries it links, so no environment
  variable is needed at all.

Happy to test candidate fixes on real gfx90a hardware (8x MI210) if useful.

[alexschroeter]

Update: dug further into the CXXABI_1.3.15 issue mentioned above (real hardware, gpu09-004, gfx90a) and have both a corrected diagnosis and a working fix — not just for that secondary issue, but for the primary LD_LIBRARY_PATH problem in this issue too, from the same root cause.

The DT_RUNPATH claim above was wrong

Checked every library in the chain with readelf -d (libamdhip64.so.6, libamd_comgr.so.3, libhsa-runtime64.so.1, librocprofiler-register.so.0, libgflags.so.2.2, libz3.so.4.13) — zero DT_RUNPATH anywhere. That mechanism isn't in play.

What's actually happening

hipcc-compiled binaries carry no rpath at all (confirmed via readelf -d on the compiled test binary — only NEEDED entries, nothing else). Since glibc resolves each SONAME once per process and shares that binding with every later consumer, and processes the main binary's own direct dependencies before descending into libamdhip64.so.6's dependency tree, the unrpathed binary's own request for libstdc++.so.6 is the first one in the process — falls through to the compat layer's old gcc-13 copy, and that binding sticks process-wide. gflags/Z3's own correct RPATH entries never get consulted.

This also directly explains the primary bug in this issue: hipcc doesn't embed a working path to libamdhip64.so.6 either, for the same reason — it just doesn't rpath anything for user-compiled code.

The fix already exists in EESSI — it's just not built for ROCm yet

EESSI's buildenv mechanism gives user-compiled code correct rpathing for every other toolchain (confirmed: buildenv/default-foss-2025a's wrapped g++ embeds a full, correct RPATH automatically, GCCcore lib64 included). module avail buildenv shows nothing for ROCm/rocm-compilers on the deployed 2025.06 stack.

Built and validated buildenv-default-rocm-compilers-19.0.0-ROCm-6.4.1.eb (installed via EESSI-extend into a user prefix, no changes to the shared CVMFS repo), completing the HIP_CLANG_PATH approach discussed on easybuild-easyconfigs#23542:

​python easyblock = 'BuildEnv' name = 'buildenv' version = 'default' homepage = 'None' description = """...""" toolchain = {'name': 'rocm-compilers', 'version': '19.0.0-ROCm-6.4.1'} modextravars = {'HIP_CLANG_PATH': '%(installdir)s/bin/rpath_wrappers/amdclangxx_wrapper'} moduleclass = 'devel' ​

With module load HIP then this buildenv loaded (order matters — HIP's own modulefile resets HIP_CLANG_PATH on load, so buildenv must load after), a plain hipcc compile — zero manual flags — produces a correctly-rpathed binary that runs with zero LD_LIBRARY_PATH workarounds. Fixes both issues in this ticket at once.

Given this, might be worth deciding whether the module-level prepend_path("LD_LIBRARY_PATH", ...) fix originally suggested is still
wanted as a fallback for people not using buildenv, or whether shipping the buildenv module is the preferred fix on its own — happy to help either way.

[ocaisa]

The key thing in buildenv is the RPATH-wrappers for the compilers. We've had an idea for a long time to ship the wrappers directly with the compilers in such a way that they are only activated in a user session (not in an EasyBuild session). The machinery we needed for that has been shipped in EasyBuild for a while, see https://gitlab.com/eessi/support/-/work_items/213#note_3043654152 for a discussion.

This should mean that things always work both for EasyBuild (which creates the wrappers on the fly) and ordinary users compiling things.

The one complication is ld, which is probably the most important wrapper. This is shipped by binutils, which is a filtered dependency for EESSI. I'm thinking of just making an exception for that that and wrapping it in GCCcore

[ocaisa]

I looked again into how EasyBuild does the wrappers. It always adds wrappers for the linkers, regardless of what it sees in the toolchain. I don't necessarily like that as we will have those repeated for every compiler, but ok.

So the general design would be when installing a compiler (identified by the moduleclass I guess):

• initialise a toolchain with the new compiler (this is a bit hard as I think you need to module to initialise the toolchain)
• create the wrappers in the installation directory
• add the wrappers to the end of the module file in a protected way (they are added to PATH only if the envvar ___EASYBUILD___ is not set)