Skip to content

[build] Ship the clang builtin headers and prefer them at runtime #33

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
aaronj0 wants to merge 1 commit into
base: main
Choose a base branch
from
+32 −8
Open

[build] Ship the clang builtin headers and prefer them at runtime #33

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,7 @@ target_compile_definitions(cppjit PRIVATE
CPPINTEROP_LIBRARY="cppjit_backend/lib/libclangCppInterOp${CMAKE_SHARED_LIBRARY_SUFFIX}"
CPPINTEROP_INCLUDE_DIR="cppjit_backend/include"
CPPJIT_CLANG_MAJOR="${LLVM_VERSION_MAJOR}"
CPPJIT_CLANG_INCLUDE_DIR="cppjit_backend/lib/clang/${LLVM_VERSION_MAJOR}"
)

target_include_directories(cppjit PRIVATE
Expand Down Expand Up @@ -175,6 +176,18 @@ install(CODE "
file(INSTALL \"${CPPINTEROP_INSTALL_DIR}/include/\" DESTINATION \${CMAKE_INSTALL_PREFIX}/cppjit_backend/include)
")

# ship the builtin headers of the build clang, laid out as a headers-only
# resource dir: only include/ ships
set(_clang_resource_dir "${LLVM_LIBRARY_DIR}/clang/${LLVM_VERSION_MAJOR}")
if(NOT EXISTS "${_clang_resource_dir}/include")
message(FATAL_ERROR
"No builtin headers at ${_clang_resource_dir}/include; the LLVM at "
"${LLVM_DIR} carries no clang resource directory")
endif()
install(DIRECTORY "${_clang_resource_dir}/include/"
DESTINATION "cppjit_backend/lib/clang/${LLVM_VERSION_MAJOR}/include"
)
Comment thread
aaronj0 marked this conversation as resolved.

# the public cpyrt API headers keep their installed cpyrt/ prefix
install(FILES
src/cpyrt/API.h
Expand Down
27 changes: 19 additions & 8 deletions src/interop/interop_wrapper.cxx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,7 @@ static inline bool is_integral(std::string& s) {
struct InterOpPaths {
std::string Library;
std::string IncludeDir;
std::string ClangIncludeDir; // empty when the bundled headers are absent
};

// One relative layout, two anchors: prefer CppInterOp next to our own load
Expand All @@ -94,8 +95,17 @@ static InterOpPaths cppinterop_paths() {
anchor = here;
}
#endif
return {(anchor / CPPINTEROP_LIBRARY).string(),
(anchor / CPPINTEROP_INCLUDE_DIR).string()};
InterOpPaths Paths{(anchor / CPPINTEROP_LIBRARY).string(),
(anchor / CPPINTEROP_INCLUDE_DIR).string(),
{}};
// The builtin headers of the build clang ship with every installed
// package (see the CMake install rule); a raw build tree has none and
// falls back to resource-dir detection.
const std::filesystem::path bundled = anchor / CPPJIT_CLANG_INCLUDE_DIR;
std::error_code ec;
if (std::filesystem::exists(bundled / "include", ec))
Paths.ClangIncludeDir = bundled.string();
return Paths;
}

// The one place libclangCppInterOp is dlopen'd.
Expand All @@ -109,7 +119,8 @@ static bool loadDispatchAPI(const InterOpPaths& Paths) {

// CppInterOp itself appends CPPINTEROP_EXTRA_INTERPRETER_ARGS inside
// CreateInterpreter, so nothing needs to be forwarded from here.
static interop::TInterp_t acquireOrCreateInterpreter() {
static interop::TInterp_t
acquireOrCreateInterpreter(const InterOpPaths& Paths) {
if (auto existingInterp = Cpp::GetInterpreter())
return existingInterp;

Expand All @@ -119,10 +130,10 @@ static interop::TInterp_t acquireOrCreateInterpreter() {
args.push_back("-march=native");
#endif
// Without clang's builtin headers the interpreter fails at its first
// #include. CppInterOp probes only bare `clang`; when just
// clang-<major> is installed, resolve and pass it explicitly.
std::string resourceDir;
if (Cpp::DetectResourceDir("clang").empty())
// #include. Prefer the bundled copy: it matches the build clang and
// needs no LLVM on the host.
std::string resourceDir = Paths.ClangIncludeDir;
if (resourceDir.empty() && Cpp::DetectResourceDir("clang").empty())

@Vipul-Cariappa Vipul-Cariappa Aug 22, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we prioritize the system's resource directory.
First, check clang-version, then the one we include, then simple clang.

@aaronj0 aaronj0 Aug 22, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No that would not work. If we build a binary for manylinux or OS X based on llvm22 and a user installs it on their system which happens to have a lower LLVM version (apple always has clang) that would crash the interpreter. The point is that cppjit should not depend on system LLVM.

@aaronj0 aaronj0 Aug 22, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is exactly the problem I described in last weeks meeting.

@vgvassilev vgvassilev Aug 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No that would not work. If we build a binary for manylinux or OS X based on llvm22 and a user installs it on their system which happens to have a lower LLVM version (apple always has clang) that would crash the interpreter. The point is that cppjit should not depend on system LLVM.

That would fail the match of the resource dir folder name which incorporates the version. We can also protect against older clangs such a check...

@aaronj0 aaronj0 Aug 22, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would fail the match of the resource dir folder name which incorporates the version. We can also protect against older clangs such a check...

Can you point me to which match fail you are referring to? Yes we can incorporate a check if the major version matches but just to clarify, we decided to bundle the headers so that we don't run into this problem and introduce mechanisms to ensure that system or package manager LLVM's agree with what cppjit needs.

If we always prioritize system-installed LLVM, that goes against the very solution you proposed (and we agreed on) to bundle the headers, leading to a self contained binary and not depend on an arbitrary LLVM picked up at runtime which can differ based on user environments

@aaronj0 aaronj0 Aug 22, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What the review suggestion here proposes is to prioritize the arbitrary clang headers detected at runtime over the one that we bundle and I disagree with that. If anything, system clang can be used as a fallback but not as the first pick for resource dir since there is no guarantee that exists or even work if it does.

@vgvassilev vgvassilev Aug 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pip install --no-build-isolation -ve .

That's 1 package manager. Does that solution work for non-pip setups?

What the review suggestion here proposes is to prioritize the arbitrary clang headers detected at runtime over the one that we bundle and I disagree with that. If anything, system clang can be used as a fallback but not as the first pick for resource dir since there is no guarantee that exists or even work if it does.

Where my comments suggested that?

@aaronj0 aaronj0 Aug 22, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's 1 package manager. Does that solution work for non-pip setups?

Yes. Development mode works because this is not a pip-specific solution but in the the underlying build system. The installation layout is fixed and under our control irrespective of the package manager. Here is a pure CMake build, without pip, against an uninstalled developer build tree of LLVM:

$ python3 -m venv demo-venv          # python to build against
$ cmake -S cppjit -B build \
      -DCMAKE_BUILD_TYPE=Release \
      -DLLVM_DIR=$HOME/llvm-project/release_build/lib/cmake/llvm \
      -DPython_EXECUTABLE=$PWD/demo-venv/bin/python
$ cmake --build build -j$(nproc)
$ cmake --install build --prefix $PWD/cppjit-install
$ cp -r cppjit/python/cppjit cppjit-install/   # normally done by the wheel

The installed tree carries the builtin headers of that LLVM:

$ diff -r $HOME/llvm-project/release_build/lib/clang/21/include \
          cppjit-install/cppjit_backend/lib/clang/21/include && echo IDENTICAL
IDENTICAL

So irrespective of what LLVM we develop against, the runtime prefers the headers libclangCppInterOp was compiled against. In a developer's uninstalled build tree there are no bundled headers, and the loader falls back to DetectResourceDir, which is unchanged in this patch.

The result is self-contained either way. With an empty PATH (no clang visible to the process):

$ PYTHONPATH=$PWD/cppjit-install PATH=/nonexistent demo-venv/bin/python \
      -c "import cppjit; cppjit.cppdef('#include <vector>\nint f(){ std::vector<int> v{41}; return v[0]+1; }'); print('f() =', cppjit.gbl.f())"
f() = 42

The point: This does not depend on a package manager, but a property of the CMake install rules which is run by every install channel. At the end of the day, cppjit is a Python package and any PEP 517 (Python standard that separates the build frontend from the build backend) frontend, drives it through scikit-build-core. So this satisfies conda recipes, conda build environments, a linux distro specific package manager, and a bare cmake --install in a developers setup (with no Python packaging at all). We are satisfying the Python build standard. If any distribution or install does not contain the bundled headers (which is impossible unless someone removes them on purpose) we try the DetectResourceDir fallback with clang. We can improve that fallback to restrict to the supported LLVM version but that is outside the scope of this PR.

@aaronj0 aaronj0 Aug 22, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where my comments suggested that?

I was referring to the original review comment in this thread:

Can we prioritize the system's resource directory.
First, check clang-version, then the one we include, then simple clang.

cc @Vipul-Cariappa

@vgvassilev vgvassilev Aug 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's 1 package manager. Does that solution work for non-pip setups?

Yes. Development mode works because this is not a pip-specific solution but in the the underlying build system. The installation layout is fixed and under our control irrespective of the package manager. Here is a pure CMake build, without pip, against an uninstalled developer build tree of LLVM:

$ python3 -m venv demo-venv          # python to build against
$ cmake -S cppjit -B build \
      -DCMAKE_BUILD_TYPE=Release \
      -DLLVM_DIR=$HOME/llvm-project/release_build/lib/cmake/llvm \
      -DPython_EXECUTABLE=$PWD/demo-venv/bin/python
$ cmake --build build -j$(nproc)
$ cmake --install build --prefix $PWD/cppjit-install
$ cp -r cppjit/python/cppjit cppjit-install/   # normally done by the wheel

The installed tree carries the builtin headers of that LLVM:

$ diff -r $HOME/llvm-project/release_build/lib/clang/21/include \
          cppjit-install/cppjit_backend/lib/clang/21/include && echo IDENTICAL
IDENTICAL

So irrespective of what LLVM we develop against, the runtime prefers the headers libclangCppInterOp was compiled against. In a developer's uninstalled build tree there are no bundled headers, and the loader falls back to DetectResourceDir, which is unchanged in this patch.

The result is self-contained either way. With an empty PATH (no clang visible to the process):

$ PYTHONPATH=$PWD/cppjit-install PATH=/nonexistent demo-venv/bin/python \
      -c "import cppjit; cppjit.cppdef('#include <vector>\nint f(){ std::vector<int> v{41}; return v[0]+1; }'); print('f() =', cppjit.gbl.f())"
f() = 42

The point: This does not depend on a package manager, but a property of the CMake install rules which is run by every install channel. At the end of the day, cppjit is a Python package and any PEP 517 (Python standard that separates the build frontend from the build backend) frontend, drives it through scikit-build-core. So this satisfies conda recipes, conda build environments, a linux distro specific package manager, and a bare cmake --install in a developers setup (with no Python packaging at all). We are satisfying the Python build standard. If any distribution or install does not contain the bundled headers (which is impossible unless someone removes them on purpose) we try the DetectResourceDir fallback with clang. We can improve that fallback to restrict to the supported LLVM version but that is outside the scope of this PR.

Ok, that clarifies it -- can you update the PR description and the commit message? That would have helped me.

resourceDir = Cpp::DetectResourceDir("clang-" CPPJIT_CLANG_MAJOR);
if (!resourceDir.empty()) {
args.push_back("-resource-dir");
Expand Down Expand Up @@ -219,7 +230,7 @@ extern "C" int LoadCppInterOp() {
if (!loadDispatchAPI(Paths))
return;

acquireOrCreateInterpreter();
acquireOrCreateInterpreter(Paths);
configureInterpreter(Paths);
preloadHeaders();
defineRuntimeHelpers();
Expand Down
Loading