Enable Cargo's new build-dir layout by ranger-ross · Pull Request #155439 · rust-lang/rust

ranger-ross · 2026-04-17T13:50:00Z

This PR enables the new Cargo build-dir layout in boostrap builds with -Zbuild-dir-new-layout.

See: #t-infra/bootstrap > Has anyone tested `./x` with the new build-dir layout?

weihanglo · 2026-04-17T13:59:14Z

    // got a list of prefix/extensions and we basically just need to find the
-    // most recent file in the `deps` folder corresponding to each one.
-    let contents = target_deps_dir
+    // most recent file in the `build` folder corresponding to each one.


Not yet read the code by myself. Just wonder why bootstrap are doing this? And if bootstrap needs to do it, it should be a feature in Cargo, at least a (perma-)unstable feature.

View changes since the review

See the commit block around line 2374. Basically we need the non-uplifted name that contains the unique hash. The artifact notifications provide the uplifted name where possible.

ranger-ross · 2026-04-17T15:28:43Z

+    //
+    // Cargo's build folder is structured as `build/<pkg>/<hash>/out/<artifacts>` so
+    // we need to traverse multiple directory layers to get to actual files.
+    let read_dir = |path: &Path| path.read_dir().ok().into_iter().flatten().filter_map(Result::ok);


checks failed because if there are any lingering files/dirs in <build-dir>/<target>/build that are not using the new layout it panics.

Update the original code to not panic if we call .read_dir() on a non-directory. Lets see if CI is happy with this

View changes since the review

ranger-ross · 2026-04-19T12:01:59Z

Looks like the miri job timed out.
Out of curiosity, how stable is this job?

(Sorry, haven't touch the rl/r repo much so I wondering if the failure is related to the new layout or if its normal for spurious failures)

mati865 · 2026-04-19T12:48:31Z

x86_64 Linux jobs rarely hang, I've retried it to be sure.

ranger-ross · 2026-04-19T15:34:48Z

Okay it's hanged on both runs so it's probably the changes.

Is there a way for me to run this check on my machine? I was trying to parse through the GitHub actions to see what command is being run but I was struggling to find it.

weihanglo · 2026-04-19T15:46:14Z

@ranger-ross see https://rustc-dev-guide.rust-lang.org/tests/docker.html#testing-with-docker

cargo run --manifest-path src/ci/citool/Cargo.toml run-local x86_64-gnu-miri

This is the easiest way to run it locally.

ranger-ross · 2026-04-20T14:09:41Z

Okay, so I did some testing and locally it runs fine. (takes around ~20 mins on my machine)

After looking a bit closer at the job logs, it actually never got stuck.
It just take WAY longer than normal....
According to the github stats x86_64-gnu-miri normally takes ~58 mins to run, but we took over 6 hrs and hit the timeout.

Looking at the other jobs, most of them did not have a large perf regression (except x86_64-gnu-tools which was also pretty bad)

That is concerning... 😅

rustbot · 2026-04-27T15:15:47Z

Some changes occurred in compiler/rustc_codegen_cranelift

cc @bjorn3

The Miri subtree was changed

cc @rust-lang/miri

Some changes occurred in src/tools/rustfmt

cc @rust-lang/rustfmt

Some changes occurred in src/tools/compiletest

cc @jieyouxu

ranger-ross · 2026-04-27T15:27:49Z

I realized the issue is not with the speed of compilation, but rather that for some reason in CI the compiler (and many other units) are being re-built multiple times which slows everything down and times out.

I haven't figured out why yet. While testing the CI on my fork, I've noticed its inconsistent and sometimes everything only builds once. I'm still investigating why this could be happening. rust-lang/cargo#16345 comes to mind as something that might change this behavior, but I am still unsure whats happening.

I went ahead and pushed some new changes with more fixes for tests and cranelift.

I'll continue investigating this week, but any rl/r experts see any obvious issues with my changes lmk
@rustbot author

ytmimi · 2026-04-27T21:40:03Z

-    // Chop off `deps`.
+    // Chop off `out`.
+    me.pop();
+    // Chop off `<hash>`.
+    me.pop();
+    // Chop off `<pkgname>`.
+    me.pop();
+    // Chop off `build`.


Is this going to break things in rustfmt's CI when we do a subtree push?

View changes since the review

ahh yeah it would probably break, maybe better for me to remove that from this PR and raise a separate PR in the rustfmt repo?

I'm still trying to understand what changes we need in this PR verse what changes can be split out into smaller more targeted changes

In that case, I'd prefer we open up a PR to add the new cargo build dir layout support to rustfmt directly and verify that things are working before making any changes to the rustfmt subtree here in r-l/rust

fair enough. I tested on my fork and I think rust-lang/rust CI fails with out rustfmt changes.

I went ahead and opened rust-lang/rustfmt#6879

@ranger-ross If you're able to duplicate the rustfmt code changes you made in rust-lang/rustfmt#6879 here that would be great.

I added those commits to this PR. I also noticed that RUSTFLAGS: -D warning was missing from the github actions workflows in rust-lang/rust so I went ahead and added that while I was at it. I guess it doesn't matter for this repo but oh well.

We've recently made some changes to those CI files in rustfmt so it's possible that the RUSTFLAGS: -D warning changes just haven't been synced up yet, but I don't think there's any issue in explicitly including that here.

Enable Cargo's new build-dir layout try-job: x86_64-msvc-1

Kobzol · 2026-05-31T09:46:29Z

(Can't do that right now because of the pending try build, but please also do a "normal" try build and run perf. on this PR, to ensure that it doesn't break rustc-perf).

ranger-ross · 2026-05-31T09:58:01Z

Sure.
From reading the rustc contributor guide I take it that the command should be @bors try @rust-timer queue, correct?

jieyouxu · 2026-05-31T10:01:28Z

Hm, I forgot if rust-timer was gated behind a separate perm.
@bors delegate=try
@bors try @rust-timer queue

rust-bors · 2026-05-31T10:01:31Z

✌️ @ranger-ross, you can now perform try builds on this pull request!

You can now post @bors try to start a try build.

Enable Cargo's new build-dir layout

jieyouxu · 2026-05-31T10:02:34Z

... AAAA. I forgot it cancels the msvc job, d'oh. Sorry. We can just let the normal try job run w/ perf, then we'll just queue another run of msvc. 🤦

rust-bors · 2026-05-31T12:10:52Z

☀️ Try build successful (CI)
Build commit: f8202da (f8202da0c3bf8db1ff27204fd160db6594220b76, parent: c315891193c35827c2d789adce686f8a5481778f)

rust-timer · 2026-05-31T13:25:53Z

Finished benchmarking commit (f8202da): comparison URL.

Overall result: no relevant changes - no action needed

Benchmarking means the PR may be perf-sensitive. Consider adding rollup=never if this change is not fit for rolling up.

@rustbot label: -S-waiting-on-perf -perf-regression

Instruction count

This perf run didn't have relevant results for this metric.

Max RSS (memory usage)

Results (primary 3.3%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	3.3%	[2.8%, 4.2%]	3
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	3.3%	[2.8%, 4.2%]	3

Cycles

Results (secondary 22.0%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	22.0%	[22.0%, 22.0%]	1
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-	-	0

Binary size

This perf run didn't have relevant results for this metric.

Bootstrap: 509.639s -> 510.066s (0.08%)
Artifact size: 400.69 MiB -> 400.72 MiB (0.01%)

ranger-ross · 2026-05-31T13:59:29Z

@bors try jobs=x86_64-msvc-1

Enable Cargo's new build-dir layout try-job: x86_64-msvc-1

rust-bors · 2026-05-31T16:33:06Z

💔 Test for 9da18bc failed: CI. Failed job:

try - x86_64-msvc-1 (web logs, enhanced plaintext logs)

This allows compiletest to support the new Cargo `build-dir` layout which passes more `-L` flags as the `deps` dir has been split per build unit. This can be an issue on Windows as the max command size is fairly small.

rustbot · 2026-06-01T13:06:34Z

These commits modify the Cargo.lock file. Unintentional changes to Cargo.lock can be introduced when switching branches and rebasing PRs.

If this was unintentional then you should revert the changes before this PR is merged.
Otherwise, you can ignore this comment.

ranger-ross · 2026-06-01T13:11:19Z

Okay, slowly but surely making progress. The latest test failure was a new error, this time due to compiletest not using arg files when invoking rustc.

Added another commit with the changes to move compiletests to use arg files. (I duplicated the ArgFileCommand code from bootstrap to compiletest, if there is smart way to share code between these, I'd be happy to refactor it)

Hopefully this is the last blocker before the windows job is happy 🙏

@bors try jobs=x86_64-msvc-1

Enable Cargo's new build-dir layout try-job: x86_64-msvc-1

rust-bors · 2026-06-01T16:09:38Z

☀️ Try build successful (CI)
Build commit: 10faafd (10faafd107c8233cd0797381cc78ab8fcf62ef21, parent: 4804ad7e93e1b31f4605b7083871d0d3d85a2afe)

ranger-ross · 2026-06-01T16:22:06Z

Okay window CI is passing, I think this is finally ready for a re-review :)

@rustbot ready

rustbot assigned bjorn3 Apr 17, 2026

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels Apr 17, 2026

ranger-ross mentioned this pull request Apr 17, 2026

Stabilize build-dir layout v2 rust-lang/cargo#16807

Open

3 tasks

bjorn3 reviewed Apr 17, 2026

View reviewed changes

Comment thread src/bootstrap/src/core/build_steps/compile.rs Outdated

weihanglo reviewed Apr 17, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

ranger-ross force-pushed the cargo-new-build-dir-layout branch from bf07fb8 to eae2cbf Compare April 17, 2026 15:27

ranger-ross commented Apr 17, 2026

View reviewed changes

This comment was marked as outdated.

Sign in to view

ranger-ross force-pushed the cargo-new-build-dir-layout branch from eae2cbf to ad47c85 Compare April 27, 2026 15:15

rustbot added A-compiletest Area: The compiletest test runner A-testsuite Area: The testsuite used to check the correctness of rustc T-rustfmt Relevant to the rustfmt team, which will review and decide on the PR/issue. labels Apr 27, 2026

This comment has been minimized.

Sign in to view

ranger-ross changed the title ~~Enable Cargo's new build-dir layout in boostrap~~ Enable Cargo's new build-dir layout Apr 27, 2026

RalfJung reviewed Apr 27, 2026

View reviewed changes

Comment thread src/tools/miri/test-cargo-miri/run-test.py Outdated

RalfJung reviewed Apr 27, 2026

View reviewed changes

Comment thread src/tools/compiletest/src/runtest/run_make.rs Outdated

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 27, 2026

bjorn3 reviewed Apr 27, 2026

View reviewed changes

Comment thread compiler/rustc_codegen_cranelift/build_system/build_sysroot.rs

ytmimi reviewed Apr 27, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

rust-bors Bot pushed a commit that referenced this pull request May 31, 2026

Auto merge of #155439 - ranger-ross:cargo-new-build-dir-layout, r=<try>

4b494bc

Enable Cargo's new build-dir layout try-job: x86_64-msvc-1

jieyouxu self-assigned this May 31, 2026

This comment was marked as off-topic.

Sign in to view

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label May 31, 2026

This comment has been minimized.

Sign in to view

rust-bors Bot pushed a commit that referenced this pull request May 31, 2026

Auto merge of #155439 - ranger-ross:cargo-new-build-dir-layout, r=<try>

f8202da

Enable Cargo's new build-dir layout

This comment has been minimized.

Sign in to view

rustbot removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label May 31, 2026

This comment has been minimized.

Sign in to view

rust-bors Bot pushed a commit that referenced this pull request May 31, 2026

Auto merge of #155439 - ranger-ross:cargo-new-build-dir-layout, r=<try>

9da18bc

Enable Cargo's new build-dir layout try-job: x86_64-msvc-1

fix: Use arg files in compiletest

461b37f

This allows compiletest to support the new Cargo `build-dir` layout which passes more `-L` flags as the `deps` dir has been split per build unit. This can be an issue on Windows as the max command size is fairly small.

This comment has been minimized.

Sign in to view

rust-bors Bot pushed a commit that referenced this pull request Jun 1, 2026

Auto merge of #155439 - ranger-ross:cargo-new-build-dir-layout, r=<try>

10faafd

Enable Cargo's new build-dir layout try-job: x86_64-msvc-1

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jun 1, 2026

jieyouxu removed their assignment Jun 2, 2026

Uh oh!

Conversation

ranger-ross commented Apr 17, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

weihanglo Apr 17, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

ranger-ross Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ranger-ross commented Apr 19, 2026

Uh oh!

mati865 commented Apr 19, 2026

Uh oh!

ranger-ross commented Apr 19, 2026

Uh oh!

weihanglo commented Apr 19, 2026

Uh oh!

ranger-ross commented Apr 20, 2026

Uh oh!

This comment was marked as outdated.

rustbot commented Apr 27, 2026

Uh oh!

This comment has been minimized.

Uh oh!

Uh oh!

ranger-ross commented Apr 27, 2026

Uh oh!

Uh oh!

ytmimi Apr 27, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ytmimi Apr 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

Kobzol commented May 31, 2026

Uh oh!

ranger-ross commented May 31, 2026

Uh oh!

This comment was marked as off-topic.

jieyouxu commented May 31, 2026

Uh oh!

This comment has been minimized.

rust-bors Bot commented May 31, 2026

Uh oh!

This comment has been minimized.

jieyouxu commented May 31, 2026

Uh oh!

rust-bors Bot commented May 31, 2026

Uh oh!

This comment has been minimized.

rust-timer commented May 31, 2026

Overall result: no relevant changes - no action needed

Uh oh!

ranger-ross commented May 31, 2026

Uh oh!

This comment has been minimized.

rust-bors Bot commented May 31, 2026

Uh oh!

rustbot commented Jun 1, 2026

ranger-ross commented Apr 17, 2026 •

edited by rustbot

Loading

weihanglo Apr 17, 2026 •

edited by rustbot

Loading

ranger-ross Apr 17, 2026 •

edited

Loading

ytmimi Apr 27, 2026 •

edited by rustbot

Loading

ytmimi Apr 29, 2026 •

edited

Loading