[quantconnect-kronos]

Closes #9970

Description

Makes the Benchmarks workflow cheaper without changing the benchmark algorithms.

• Master's results are stored and reused. A run on master stores its benchmark_results.json as the benchmark-results workflow artifact. A pull request's run finds the latest successful master run (push event) through the Actions API and downloads that artifact as its reference. It builds and runs only the change and no longer builds or runs master up front. Master's runs only write the reference and pull request runs only read it. Each master run stores its own immutable artifact, so concurrent runs can't interfere.
• Only what falls below the reference is re-checked. The stored reference comes from another run and machine, so the algorithms below it (the existing failed_benchmarks.json from Re-check failed benchmarks before failing the Benchmarks workflow #9963) run again on master and on the change, on the same machine after a 30 second pause. The run fails only when the regression repeats. Master is checked out and built only in that case. If no reference is available (the lookup or download fails, for example before the first master run stores one), master is checked out, built and run in full as the reference, as before. The same compare and re-check of failures follow, so a missing artifact never fails the run on its own.
• No more warm-up run. run_benchmarks.py runs each algorithm once instead of a discarded warm-up run followed by the measured one. Measured locally, the first run's samples differ from the warm ones no more than two warm runs differ from each other (table below), and failures are re-checked anyway. The results file keeps the same format.
• What needs a run: push now triggers only on master, so pushes to other branches and tags no longer run the workflow. Pull requests are still benchmarked through pull_request. In the workflow's last 200 runs, 53 were pushes to non-master branches, mostly the push side of branches that also had a pull request run. I didn't skip documentation-only changes with a paths filter: if the check is required, a filtered-out run leaves it pending and blocks the merge.

Workflow file changes (.github/workflows/benchmarks.yml):

• on.push only covers branches: [master] (it used to cover '*' branches and tags).
• New permissions: contents: read, actions: read, so the token can read master's runs and artifacts, including from forks.
• The up-front Build Lean Master step is removed.
• New Store Master Benchmarks step (push only) using actions/upload-artifact@v4.
• New Find Master Benchmarks step (curl + python, both already in the image) and Fetch Master Benchmarks step (actions/download-artifact@v4 with run-id), on pull requests only, both continue-on-error.
• Checkout Lean Master now runs on every pull request, after the reference lookup. It is only a checkout: master is built only when it is needed.
• Run Benchmarks Master now runs only when the reference couldn't be fetched: it builds master, runs every algorithm and uses the result as the reference.
• Compare Benchmarks compares against reference/benchmark_results.json.
• Re-check Failed Benchmarks (unchanged trigger: the compare failed) builds master (a no-op when already built), then runs the failed algorithms on both sides after the pause.

compare_benchmarks.py and the benchmark algorithms are unchanged.

How it was tested

Everything ran in quantconnect/lean:foundation with the workflow's container limits (--cpus 12 --memory 12g). The market data the benchmarks read was copied to local disk and mounted at /Data. Each benchmark processed the same number of data points and had the same failed data request count as when reading from the full data folder.

Before and after times (seconds, local):

	current flow	new flow, pull request	new flow, master
builds	2 × ~111	1 × ~111 (+1 when re-checking)	1 × ~111
benchmark passes	2 × 986 (warm-up + measured)	1 × ~470	1 × 478
re-check	as needed	~170-200 when needed (master build + pause + failed algorithms on both sides)	none
total	~2,200 plus re-check	468-676	~590

For reference, a full pass with the warm-up currently takes 19-20 minutes on CI, and a whole run about 45 minutes.

Consistency, 6 consecutive pull request runs on unchanged code against a stored master reference:

run	first compare	below reference	re-check	total
1	failed	Py EmptySingleSecuritySecondEquity (170 vs 170.1 expected)	passed	658s
2	failed	C# HistoryRequest, Py HistoryRequest, Py StatelessCoarseUniverseSelection	passed	666s
3	passed	–	–	468s
4	passed	–	–	470s
5	failed	Py BasicTemplate	passed	640s
6	failed	C# HistoryRequest, Py HistoryRequest, Py StatefulCoarseUniverseSelection	passed	676s

All 6 passed, and the re-check never needed more than 3 of the 22 algorithms.

Regressions are still caught: with a temporary busy-wait in the engine's time slice loop (about 25-33% slower on the smaller benchmarks, not committed), the first compare flagged 9 algorithms. The same-machine re-check confirmed 7 of them (for example C# BasicTemplate 226 vs 343.8 expected, Py BasicTemplate 187 vs 275.4), so the run fails.

Warm-up, the same 20 benchmarks run 3 times each in a row after dropping the OS page cache (EmptyEquityAndOptions400 was left out because its data was still incomplete locally at that point). The first, cold run was between -11% and +23% of the measured second run, median -1%. A third warm run was between -12% and +19% of the second, median +1%. A full pass took 328s, 329s and 325s.

Other checks:

• actionlint reports only the existing warnings (the custom benchmark runner label, actions/checkout@v2).
• The run lookup query returns the latest successful master push run.
• The failed_benchmarks.json / run-all argument handling in the re-check step was checked in bash.

The Lean build passes. The C# test suite was not run, because the change touches only the workflow file and run_benchmarks.py, which nothing in the solution uses.

Caveats:

• The first master run after this merges stores the first reference. Until then, pull request runs take the no-reference path: master is run in full, as before, and only the failures are re-checked.
• A stored reference that happened to run slow can hide a regression smaller than that gap in the first compare. That gap is the cost of not running master in every pull request. Anything that does fall below the reference is still confirmed against master on the same machine.

🤖 Generated with Claude Code

[quantconnect-kronos]

As the issue asks, I cancelled this pull request's runs of the other workflows (Build & Test, Regression, Research Regression, Syntax, Report Generator, Python Virtual Environments) so the shared self-hosted runners stay free. This change only affects the Benchmarks workflow and run_benchmarks.py, which none of them use. This first Benchmarks run has no stored master reference yet (no master run has stored one), so it takes the fallback path and re-checks every algorithm against master on the same machine.

[quantconnect-kronos]

The six failing build checks are the runs of the other workflows I cancelled on purpose: Build & Test Lean, Regression Tests, Research Regression Tests, Python Virtual Environments, Report Generator Tests and Syntax Tests. Each run's conclusion is cancelled, after about 1 minute. The change didn't make them fail. The issue asks to keep the shared runners free on this pull request, so I'm not re-running them. They can be re-run if wanted, but they don't use anything this change touches. The Benchmarks run, the one this change affects, is still in progress.

[quantconnect-kronos]

The Benchmarks failure was caused by this change, now fixed in c5bdc06. No master run has stored a reference yet, so the run took the no-reference path. That path ran all 22 algorithms once on master and once on the change, then failed on that single comparison: Python HistoryRequestBenchmark was 108 against master's 145, on identical engine code. That is the single-sample noise #9961 describes, with no re-check behind it. Without a stored reference, master is now run in full as the reference, as before. The usual compare follows, and only the algorithms below the reference are re-checked on both sides on the same machine. This run also confirmed that a missing artifact is handled: the lookup and download steps carried on, and the run fell back to master as intended. The six cancelled checks are the other workflows, cancelled again on this commit as the issue asks.

[quantconnect-kronos-reviewer]

Requesting changes: the Benchmarks run on this pull request failed, and the change caused it.

The fallback path fails on noise. No master run has stored a reference yet, so run 40807448038 took the fallback path. The lookup found master run 40766358261, the download failed with Artifact not found for name: benchmark-results as expected, Compare was skipped and the re-check ran all 22 algorithms. One single-sample comparison of everything, with nothing re-checked after it, then failed: Py HistoryRequestBenchmark measured 108 against master's 145 (floor 130.5). That is noise. This change doesn't touch the engine, and the same build measured 142 for that algorithm in its first pass a few minutes earlier, which passes. This is the #9961 failure mode that #9963 fixed, and the fallback brings it back. It also does one more pass than it needs, because the change's first full pass is thrown away. The description says a missing artifact never fails the run, and this run shows it does. Any later run whose lookup or download fails, such as an API error or an expired artifact, gets the same flaky check.

Suggested fix: make the fallback produce the reference, and leave the rest as it is. When there's no stored reference, build master and run all algorithms on it, then point Compare Benchmarks at whichever reference exists (reference/ or LeanMaster/) so it writes failed_benchmarks.json. The re-check then always runs only the failed algorithms. It never re-runs all of them, and the no-reference path does the same work #9963 does today, minus the warm-up.

What's fine:

• The steps.*.outcome chaining works on CI. The download failed with continue-on-error, steps.reference.outcome was failure, Compare was skipped and the re-check ran.
• The single pass took 7m42s on CI (02:39:00–02:46:42), against 19–20 minutes for a pass with the warm-up.
• Its samples were within a few percent of the master pass on the same machine for 21 of 22 algorithms. Py EmptySingleSecuritySecondEquity 289 vs 272 and C# EmptyFutures 439 vs 394 were the widest.
• actions/checkout's default clean (git clean -ffdx) removes a stale failed_benchmarks.json, reference/ or LeanMaster/ before each run, so ALGORITHMS can't pick up a leftover file.
• Over the last 40 runs, every Benchmarks job ran on the same benchmarks runner, so the cross-machine gap in the stored reference matters less than the description's caveat suggests.

Still unproven: the reference path, which finds and downloads a stored artifact and compares against it, can't run on CI until a master run stores one. Its consistency evidence is the 6 local runs. Because every failure there falls back with continue-on-error, the fallback has to be reliable before this merges. After the fix, please re-run Benchmarks on this pull request (it still takes the fallback path) and add the outcome to the description, together with the first-compare and re-check results.

Other repositories: none affected. Only the workflow and run_benchmarks.py change, and nothing outside this repository uses them.

CI: the six cancelled build checks are the other workflows, cancelled on purpose as the issue asks (each cancelled after ~1 minute). They're unrelated to this change. The failing Benchmarks check is caused by it.

Risk: risk:high. The changed workflow's own check fails, and the reference path has never run on CI. A green Benchmarks run with the fixed fallback would support lowering it. The change itself is CI-only and easy to revert.

[quantconnect-kronos-reviewer]

With no failed_benchmarks.json, this re-runs all 22 algorithms on both sides and the compare on line 89 is final, so one noisy sample fails the run. That's what happened on this pull request's own run: Py HistoryRequestBenchmark measured 108 vs 145, after 142 in the first pass. When there's no stored reference, could master's full run happen before Compare Benchmarks, so it becomes the reference, compares against the first pass and writes failed_benchmarks.json? Then this step only ever re-checks the failures.

[quantconnect-kronos-reviewer]

Approving. c5bdc06 fixes the fallback the way my last review asked, and the Benchmarks run on it passes.

What changed since my last review: when the stored reference can't be fetched, Run Benchmarks Master now builds master, runs every algorithm and copies the result to reference/benchmark_results.json. Compare Benchmarks then always compares against reference/ and writes failed_benchmarks.json, so Re-check Failed Benchmarks only ever runs the failures. The re-check no longer runs every algorithm on both sides with a single final compare.

CI evidence, run 40810140894 on c5bdc06:

• The lookup found the latest master run. The download failed with Artifact not found for name: benchmark-results, as expected before any master run stores one, and continue-on-error let the run carry on. Run Benchmarks Master ran as the reference.
• The compare passed all 22 algorithms, so the re-check was skipped. Py HistoryRequestBenchmark, the one that failed last time, measured 148 against a floor of 131.4. The closest was Py IndicatorRibbonBenchmark at 75 against 74.7, which would have triggered a re-check of that algorithm only.
• The whole job took 18m32s. Today a run takes about 45 minutes, and the change's single pass took 7m40s.

Checked in the code:

• Every way the reference path can fail (the lookup, the download, a run from before this merges that has no artifact, an expired artifact) leaves steps.reference.outcome other than success, so master is run in full as before. A missing artifact can't fail the run on its own.
• In the re-check, dotnet build on LeanMaster is incremental after the fallback build, and it's needed on the reference path, where master hasn't been built yet. compare_benchmarks.py iterates the reference's keys, so the re-check's partial results compare only the failed algorithms.
• Master pushes only build, run and upload. They skip the master checkout, the compare and the re-check. Before this change, a master push compared master against itself.

Still unproven on CI: the reference path, where an artifact is found, downloaded and compared against, can only run after a master run stores one. Its evidence is the 6 local consecutive runs and the local regression check in the description. Any failure there falls back to the path proven above. Please watch the first pull request run after the first master run stores a reference.

Nit (description): the second bullet says "Master is checked out and built only in that case", but Checkout Lean Master runs on every pull request. Only the build is conditional, as the workflow-changes list says correctly.

CI: the six failing build checks are the Research Regression, Report Generator, Regression, Python Virtual Environments, Syntax and Build & Test runs on c5bdc06. Each was cancelled on purpose, as the issue asks, and none of them uses the workflow or run_benchmarks.py. If they are required to merge, they need to be re-run first.

Other repositories: none affected.

Risk: I set risk:low. The change touches only CI, isn't user-facing and is easy to revert. Each new failure mode falls back to the old behavior, and the run on this pull request proves that fallback works.