feat(doctor): nvcc's unreachable back-end, and the post-release verification that found it - #560
Merged
Merged
Conversation
…i from a comment Section 14 was written when the engine-side change was merged but not yet released. Section 15 records what the released 2026.9.5.1 was measured to do: six criteria run against the published binary in a fresh subos, and a control establishing that the driver becomes reachable because of the index package rather than because of host leakage. It also records that the first run of those criteria was a false green. The script was correct; the invocation removed its first line, which was `set -euo pipefail`, in order to work around a quoting problem. Every assertion then became advisory and the closing banner printed regardless. The rule this yields is that assertions must not rest on `set -e` alone. The comment added to the runtime.artifacts whitelist carried a warning emoji. Emphasis is carried by a capitalised phrase instead.
Verifying the release inside a sandbox found the CUDA example failing where it succeeds on the host. The difference is one line: nvcc runs cicc, cudafe++, ptxas and fatbinary as bare names on a PATH it prepends itself from an `nvcc.profile` beside its binary, Debian-family packaging makes that profile a symlink into /etc, and a sandbox that replaces /etc removes it. nvcc then keeps the ambient PATH and reports `sh: 1: cicc: not found`, which names neither nvcc nor the profile. Nothing about the toolkit is missing, so every obvious check passes. The plan is asked for rather than assumed. `nvcc --dryrun` prints the stages and the PATH nvcc will use without compiling anything; parse_dryrun reads that plan, and the doctor resolves each bare name against the path nvcc states, reporting the first that does not resolve. A dryrun that yields no plan -- there is no nvcc, or the output is not one -- produces no finding, on the same principle as the host-compiler bound: a probe that reaches no answer must not invent one. Measured on both sides, because a check that is always silent and a check that is always loud look alike from one run. With a working nvcc the warning does not appear; with nvcc copied to a directory holding no profile it appears and names `cicc` -- the same stage the real failure names, because Debian ships cudafe++, ptxas and fatbinary in /usr/bin and cicc only in the toolkit's own directory. No e2e accompanies this. It would need nvcc, which CI does not have, and a `# requires: nvcc` script skips and exits 0 on every shard. The coverage is five unit tests over two real dryrun transcripts differing in exactly that PATH line, plus the two-sided control recorded in the design's section 15.7.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two things, in the order they happened.
1. A check: whether nvcc can reach its own back-end
Verifying the release inside a sandbox found the CUDA example failing where it
succeeds on the host, with:
Single-variable isolation on the same nvcc command line:
sh: 1: cicc: not foundPATH=/usr/lib/nvidia-cuda-toolkit/bin:$PATH.oNVVMIR_LIBRARY_DIR.oOne PATH entry is the whole of the difference. nvcc runs
cicc,cudafe++,ptxasandfatbinaryas bare names on a PATH it prepends itself from annvcc.profilebeside its own binary; Debian-family packaging makes thatprofile a symlink into
/etc; a sandbox that replaces/etcremoves it. Thetoolkit is complete, nvcc is on PATH,
crt/host_config.his readable, and thedoctor's device section prints as usual — every obvious check passes, and the
message the user gets names neither nvcc nor the profile.
The plan is asked for rather than assumed.
nvcc --dryrunprints the stagesand the PATH nvcc will use without compiling anything.
parse_dryrunreadsthat plan; the doctor resolves each bare name against the path nvcc states and
reports the first that does not:
This is the same principle as reading the host-compiler bound out of
crt/host_config.hinstead of tabulating it: the vendor answers, mcpp does notguess. A dryrun that yields no plan — there is no nvcc, or the output is not a
plan — produces no finding.
Measured on both sides
A check that is always silent and a check that is always loud look alike from
one run, so both were produced:
cannot reach its own back-endciccIt names
ciccrather than the earliercudafe++because Debian shipscudafe++,ptxasandfatbinaryin/usr/binandcicconly in thetoolkit's own directory — which is the stage the real failure names.
Coverage
No e2e. It would need nvcc, which CI does not have, and a
# requires: nvccscript skips and exits 0 on every shard, which is an assertion that never runs.
The coverage is five unit tests over two real
--dryruntranscripts differingin exactly that PATH line, plus the two-sided control above.
2. Section 15: what the released 2026.9.5.1 was measured to do
Section 14 was written at merge time, before the release. Section 15 records
the six criteria run against the published binary in a fresh subos:
mcpp 2026.9.5.1Configured xlings mirror = CNself doctorreports the host/device compiler pairinggcc 13 exceeds the bound of 12 stated in /usr/include/crt/host_config.hacceldimension refuses a mismatched artifact, naming the dimension, both sides, and a remedy on that axisaccel,sm_90f,sm_86,--no-accel--no-acceladmits itcompat.cuda-runtimemakes the driver reachable with no environment variableOK: driver reachable, cuInit=0x7a828447cbc0Plus a control for F: removing the dependency and changing nothing else yields
CONTROL-OK: driver NOT reachable without the package, so F measures thepackage rather than host leakage.
The first run of those criteria was a false green
Fitting the script into
xlings subos use <name> --sandbox --cmd "$(...)"raninto a quoting problem, and the workaround deleted the script's first line —
which was
set -euo pipefail. Nothing was installed, every failure continued,three sections asserted nothing, and the closing banner printed anyway. Two
rules follow: assertions carry their own
|| fail "..."rather than resting onset -e, and a transport problem is fixed in the quoting, not by deleting aline from the script.
One further finding
Inside the sandbox
xlings install mcpp@2026.9.5.1succeeds while thebare-name shim still resolves to the previous release; xlings reports this
itself. Release artifacts are therefore verified through the store path. This
is the same directory as the shim-pruning defect in section 14.4
(openxlings/xlings#582).
Also
The comment added to the
runtime.artifactswhitelist carried a warning emoji.Emphasis is now carried by a capitalised phrase.