Skip to content

Run the slow AlgorithmHistoryTests cases in parallel - #9977

Merged
Martin-Molinero merged 1 commit into
masterfrom
kronos/parallelize-algorithm-history-tests
Oct 29, 2026
Merged

Martin-Molinero merged 1 commit into
masterfrom
kronos/parallelize-algorithm-history-tests

Conversation

@quantconnect-kronos

Copy link
Copy Markdown
Contributor

Part of #9966 (stage 2 of the plan posted on the issue).

Description

AlgorithmHistoryTests ran its 345 tests one after another (164s of test time). It was the last fixture still running in the parallel phase, alone for ~150s. Five methods account for ~140s of that: HistoryRequestWithExtendedMarketHoursTickResolution, TickHistoryReturnsConsistentResultsWithOrWithoutEquity, HistoryRequestWithContracDepthOffsets, HistoryRequestWithDataMappingMode and HistoryRequestWithFillForward.

  • AlgorithmHistoryTests: those five methods are now [Parallelizable(ParallelScope.All)], so their cases run in parallel with each other and with the rest of the fixture. The rest of the fixture still runs one test at a time, as before (checked in the results: no two of its other tests overlap). The fixture now uses FixtureLifeCycle(LifeCycle.InstancePerTestCase), so each test gets its own _algorithm and _testHistoryProvider. Its OneTimeSetUp and the providers it sets are now static, as that life cycle requires. No test or assertion changes.
    I didn't make the whole fixture parallel. Many of its other tests change process-wide state, such as Config (the history warning tests), SymbolCache.Clear(), custom data registered under the same ticker in its C# and Python cases, and the static counters of CustomFundamentalTestData. One attempt failed GetHistoryWithCustomDataType(Python) this way. The five methods above only read the standard test data through their own algorithm instance.
  • Python initialization in AssemblyInitialize: running Python tests at the same time deadlocked the test host twice. Both times it was a lazily initialized value that imports a Python module while it holds a lock: Python.NET's Runtime.PyDecimalType (a Lazy that imports _pydecimal), and the PandasData static constructor (which imports PandasMapper). During the import, the interpreter hands the GIL to another thread. If that thread needs the same value, it waits on the lock while holding the GIL, and the importing thread never gets the GIL back. Stack dumps of the stuck host showed exactly that. Both values are now initialized once in the assembly setup, before any test runs. For Python.NET's value that takes reflection, since it's internal. I reported the root cause in Deadlock when two threads first resolve PyDecimalType at the same time pythonnet#153. Once that ships, the reflection line can go. The PandasData static constructor is library code: the same deadlock can happen in any process where two threads first use PandasData at the same time while one holds the GIL. This PR doesn't change it.
  • StreamReaderExtensionsTests.Performance: it compares two readers' timings (streamReaderMs * 1.5 < getLineReaderMs). Its method-level [Parallelizable(ParallelScope.None)] didn't keep it from running next to other tests. In the results it ran at the very start of the parallel phase, on master too. It already fails now and then on CI (for example run 39981629984). In my first full run of this branch it failed while overlapping with the slow history cases. It now lives in its own [NonParallelizable] fixture, StreamReaderExtensionsPerformanceTests, which runs with nothing else in the same test run. The test and its assertion are unchanged.

Only test code changes. The workflow file is not changed.

Testing

Measured in quantconnect/lean:foundation (--cpus 12 --memory 12g) with the workflow's build and test commands. Each master (a3bf4cc) / branch pair ran at the same time on the same host, which was shared and loaded.

master this branch
AlgorithmHistoryTests alone 2m 44s 1m 26s – 1m 30s (3 runs)
AlgorithmHistoryTests inside the full run (first start to last end) 194s 121s
Full test step, run 1 (dotnet test duration) 10m 12s 9m 33s
Full test step, run 2 10m 26s 9m 30s
Full run 1 result 39,177 passed, 1 failed, 100 skipped 39,177 passed, 1 failed, 100 skipped
Full run 2 result 39,178 passed, 100 skipped 39,178 passed, 100 skipped

Run 1 failures: on master, LiveTradingDataFeedTests.HandlesAllTypes<TradeBar>(SPY ... Minute ...), the known fixed-wait test that stage 3 covers. On the branch, StreamReaderExtensionsTests.Performance(QuoteBar), which led to the last change above. Run 2 includes that change, and there Performance ran with no other test overlapping it.

Consistency: the changed fixtures (AlgorithmHistoryTests and both StreamReaderExtensions fixtures, 387 tests) ran 32 times, 8 in a row in each of 4 containers running at the same time (3 CPUs each).

  • AlgorithmHistoryTests passed all 32 runs, with no hangs (2m 9s – 2m 34s per run, against 3m 1s – 3m 11s on master in the same setup).
  • Performance failed in 5 of the 32 runs. The same stress on master failed it in 3 of 32, with the same assertion. In this setup the noise comes from the other containers on the host, which no test attribute can isolate. That is also true on shared CI runners.

What's left for the next stages

  1. LiveTradingDataFeedTests: wait on conditions instead of fixed time spans.
  2. Fixed delays in the sequential phase (RemoteFileSubscriptionStreamReaderTests, LiveTradingResultHandlerTests, LiveTradingRealTimeHandlerTests, KeyStringSynchronizerTests, RateGateTests, ...).
  3. Parallel phase follow-ups (BrokerageTransactionHandlerTests, CustomLiveDataFeedTests, ...). With AlgorithmHistoryTests at ~120s, it's still among the longest fixtures in the parallel phase. Most of what's left in its parallel cases is waiting for the GIL, which every history call takes for each slice once Python is initialized.
  4. Python, research and heavy computation fixtures.

🤖 Generated with Claude Code

- Run the five slowest AlgorithmHistoryTests methods in parallel, with a
  fixture instance per test case so they don't share the algorithm field
- Initialize Python.NET's decimal type and PandasData before tests run in
  parallel: both import Python modules lazily while holding a lock, which
  deadlocks with a test that holds the GIL and waits on that lock
- Move the StreamReaderExtensions performance test to its own
  non-parallelizable fixture: it compares two timings, and the method-level
  ParallelScope.None didn't keep it from running next to other tests

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@quantconnect-kronos

Copy link
Copy Markdown
Contributor Author

The failing checks other than Build & Test Lean are runs I cancelled, as #9966 asks, to keep the shared runners free for Build & Test Lean: Research Regression Tests, Report Generator Tests, Python Virtual Environments, Regression Tests, Syntax Tests and Benchmarks. Only test code changes here, so they say nothing about the change, and I'm not re-running them.

@quantconnect-kronos quantconnect-kronos Bot added the kronos:develop:working develop: in progress label Oct 29, 2026
@quantconnect-kronos

Copy link
Copy Markdown
Contributor Author

I checked the six failing build checks. Each is a run I cancelled, as #9966 asks: Benchmarks, Research Regression Tests, Report Generator Tests, Python Virtual Environments, Syntax Tests and Regression Tests. Each job's conclusion is cancelled and no step failed, so they say nothing about this change, and I'm not re-running them. API Tests was skipped. Build & Test Lean, the workflow this change affects, passed: 39,178 passed, 0 failed, 100 skipped, test step 12m 50s.

@quantconnect-kronos quantconnect-kronos Bot added the kronos:review Request review label Oct 29, 2026
@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot added kronos:review:queued review: queued kronos:review:working review: in progress and removed kronos:review Request review kronos:review:queued review: queued labels Oct 29, 2026
@quantconnect-kronos-reviewer
quantconnect-kronos-reviewer Bot marked this pull request as ready for review October 29, 2026 02:08

@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving: only test code changes, and I found no defects.

What I checked

  • AlgorithmHistoryTests life cycle. FixtureLifeCycle(LifeCycle.InstancePerTestCase) needs OneTimeSetUp to be static, and the providers it sets are now static too. They're read-only after setup, so sharing them is safe. Of the five parallel methods, only TickHistoryReturnsConsistentResultsWithOrWithoutEquity uses the _algorithm field, and each test now gets its own instance. The other four build their own algorithm with GetAlgorithm. None of the five touch Config, SymbolCache, the custom data statics or _testHistoryProvider. Each case runs PyModule.FromString("testModule", ...), which creates a separate module object, so cases running at the same time don't share one.
  • State the rest of the fixture changes while these cases run. TickResolutionOpenInterestHistoryRequestIsFilteredByDefault_MultipleSymbols(Python) calls SymbolCache.Clear(), and the warning tests call Config.Set("history-request-cells-warning-threshold", ...). The parallel cases assert only bar counts and prices, never DebugMessages or symbol strings, so neither change can make them fail. Before this PR, other fixtures already ran alongside that SymbolCache.Clear().
  • Python initialization in AssemblyInitialize. In the checked-out pythonnet, Runtime.PyDecimalType is an internal static Lazy<PyType> field (src/runtime/Runtime.cs:482), so the reflection reaches the right member. PandasData's static constructor takes the GIL itself, and taking it again inside the outer Py.GIL() is fine. QuantConnect/pythonnet#153 is open and public.
  • Moved Performance test. I compared it with the removed code: it's identical apart from the new fixture and its usings, and TradeBarTest/QuoteBarTest moved with it. StreamReaderExtensionsTests has no unused usings left. NUnit runs a [NonParallelizable] fixture in the non-parallel work shift, so nothing else runs during it.

CI. Build & Test Lean passed: 39,178 passed, 0 failed. The six red checks are the other workflows, which were cancelled as #9966 asks. I confirmed each run's conclusion is cancelled and no step failed. Saying they tell us nothing about this change is wrong for three of them, though. Regression Tests, Research Regression Tests and Python Virtual Environments all run QuantConnect.Tests.dll, so they run the changed AssemblyInitialize. RegressionTests and ResearchRegressionTests also call AssemblyInitialize.AdjustCurrentDirectory() from their test case sources. So I ran those three workflows' commands myself on this PR's head commit, in the workflow image, and all passed:

  • Regression Tests: 1,101 passed, 0 failed
  • Research Regression Tests: 5 passed, 0 failed
  • Python Virtual Environments: AssertVirtualEnvironment passed with and without system site packages, and PythonPackagesTests passed 138 of 138

Report Generator Tests, Syntax Tests and Benchmarks don't load the test assembly's setup.

Non-blocking note. The reflection in AssemblyInitialize is all null-conditional. If a later pythonnet upgrade renames or removes PyDecimalType before #153 ships, the warm-up silently does nothing and the parallel Python deadlock can come back. Since #153 is meant to make that line removable, I'm fine leaving it as is.

Other repositories: none affected. The change is limited to the test assembly.

Risk: I set risk:low. The change is test-only and easy to revert. The evidence is the green main workflow, 32 stress runs of the changed fixtures with no hangs, and my passing runs of the three affected workflows that were cancelled.

@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot added risk:low Low risk and removed kronos:review:working review: in progress labels Oct 29, 2026
@quantconnect-kronos quantconnect-kronos Bot added kronos:develop:idle develop: paused and removed kronos:develop:working develop: in progress labels Oct 29, 2026
@Martin-Molinero
Martin-Molinero merged commit eff85cb into master Oct 29, 2026
2 of 8 checks passed
@Martin-Molinero
Martin-Molinero deleted the kronos/parallelize-algorithm-history-tests branch October 29, 2026 12:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kronos:develop:idle develop: paused risk:low Low risk

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant