Skip to content

Add hook for TensorFlow v2.18.1 #177

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
TopRichard wants to merge 8 commits into
base: main
Choose a base branch
from
Open

Add hook for TensorFlow v2.18.1 #177

Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
d510e7e
eb_hooks modifications for {2025.06}[2024a] TensorFlow v2.18.1
Feb 28, 2026
6451f30
removed CUDA entries
Feb 28, 2026
79446cb
removed unnecessary changes
Feb 28, 2026
b99c09e
fixed typo
Feb 28, 2026
0ad9b6c
Added hash for x86_64
Feb 28, 2026
e020a96
Fix KleidiAI -march/-mcpu conflict
Mar 2, 2026
9a7f39c
Merge branch 'main' of github.com:EESSI/software-layer-scripts into T…
Mar 10, 2026
3518cda
Add hook for TensorFlow v2.18.1
Mar 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 46 additions & 0 deletions eb_hooks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -650,6 +650,51 @@ def parse_hook_maturin(ec, eprefix):
raise EasyBuildError("maturin-specific hook triggered for non-maturin easyconfig?!")


def parse_hook_tensorflow_h5py_glibc(ec, eprefix):
"""
Fix the Python and environment used while building and running tests for TensorFlow-2.18.1
"""
if ec.name == 'TensorFlow' and ec.version == '2.18.1':
ec['preconfigopts'] = ec.get('preconfigopts', '') + (
'export GCC_HOST_COMPILER_PATH=$EBROOTGCC/bin/gcc && '
'sed -i \'s|--define=PREFIX=/usr|--define=PREFIX=\\$EESSI_EPREFIX|g\' .bazelrc && '
'cat > /tmp/fix_h5py.py << \'EOF\'\n'
Copy link
Contributor

@boegel boegel Mar 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't like the use of a hardcoded path in a fixed location...

Can we at least use /tmp/$USER/, or even better mktemp -d ?

Some comments in this hook to explain exactly what all of this does and why it's required would be helpful I think, since it looks quite involved...

'with open("requirements_lock_3_12.txt", "r") as f:\n'
' content = f.read()\n'
'content = content.replace("h5py==3.11.0 \\\\", "h5py==3.15.1 \\\\")\n'
'content = content.replace(\n'
Copy link
Contributor

@boegel boegel Mar 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use re.sub here instead, especially below to avoid the long repetition.

Something like this (totally untested):

regex = re.compile("(--hash=sha256:f4e025e852754ca833401777c25888acb96889ee2c27e7e629a19aee288833f0)", re.M)
extra_hashes = "...'
content = regex.sub(extra_hashes, content)

' " --hash=sha256:f4e025e852754ca833401777c25888acb96889ee2c27e7e629a19aee288833f0",\n'
' " --hash=sha256:f4e025e852754ca833401777c25888acb96889ee2c27e7e629a19aee288833f0 \\\\\\n --hash=sha256:25c8843fec43b2cc368aa15afa1cdf83fc5e17b1c4e10cd3771ef6c39b72e5ce \\\\\\n --hash=sha256:8a33bfd5dfcea037196f7778534b1ff7e36a7f40a89e648c8f2967292eb6898e"\n'
')\n'
'with open("requirements_lock_3_12.txt", "w") as f:\n'
' f.write(content)\n'
'EOF\n'
'python3 /tmp/fix_h5py.py && '
)
if get_eessi_envvar('EESSI_CPU_FAMILY') == 'aarch64':
ec['prebuildopts'] = ec.get('prebuildopts', '') + (
# KleidiAI in TF 2.18 has similar -march/-mcpu conflict as XNNPACK
# The easyblock already excludes XNNPACK from -mcpu=native, extend the same exclusion to KleidiAI
Copy link
Contributor

@boegel boegel Mar 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't this something that should be fixed in the TensorFlow easyblock, or in the easyconfig file for TensorFlow 2.18.1?

Now that I think of it, same question w.r.t. the hash stuff above...
Would that be a lot cleaner by adding a patch file (seems like that could work, since it's done pre-configure)?

'sed -i \'s|--per_file_copt=-.*XNNPACK/.*@-mcpu=native|--per_file_copt=-.*XNNPACK/.*,-.*KleidiAI/.*@-mcpu=native|g\' .tf_configure.bazelrc && '
)
current_opts = ec.get('buildopts', [])
if isinstance(current_opts, str):
current_opts = current_opts.split()

ec['buildopts'] = current_opts + [
'--linkopt=-Wl,--disable-new-dtags --host_linkopt=-Wl,--disable-new-dtags --action_env=GCC_HOST_COMPILER_PATH=$EBROOTGCC/bin/gcc --host_action_env=GCC_HOST_COMPILER_PATH=$EBROOTGCC/bin/gcc',
]

ec['pretestopts'] = (
"""interppath=$(find "$EESSI_EPREFIX/lib64" -name 'ld-*' | grep -E 'so\\.1|so\\.2' | head -n1) && """
"""pybin=$(find "%(builddir)s/%(name)s/bazel-root/" -type f -path "*/external/python_%(arch)s-unknown-linux-gnu/bin/python%(pyshortver)s" | head -n1) && """
"""patchelf --set-interpreter "$interppath" "$pybin" && """
)
print_msg("TensorFlow-h5py-glibc related changes have been applied")
else:
raise EasyBuildError("TensorFlow-h5py-glibc specific hook triggered for non-TensorFlow easyconfig?!")


def parse_hook_ucx_eprefix(ec, eprefix):
"""Make UCX aware of compatibility layer via additional configuration options."""
if ec.name == 'UCX':
Expand Down Expand Up @@ -1857,6 +1902,7 @@ def post_easyblock_hook(self, *args, **kwargs):
'OpenBLAS': parse_hook_openblas_relax_lapack_tests_num_errors,
'pybind11': parse_hook_pybind11_replace_catch2,
'Qt5': parse_hook_qt5_check_qtwebengine_disable,
'TensorFlow': parse_hook_tensorflow_h5py_glibc,
'UCX': parse_hook_ucx_eprefix,
}

Expand Down
Loading