Skip to content

Run the Python test suite in parallel with pytest-xdist - #17520

Merged
stevejalim merged 2 commits into
mainfrom
ci-pytest-xdist
Nov 12, 2026
Merged

stevejalim merged 2 commits into
mainfrom
ci-pytest-xdist

Conversation

@stevejalim

Copy link
Copy Markdown
Contributor

The main pytest run takes six to eight minutes in CI in a single process. With -n auto on the four-vCPU hosted runners each worker gets its own in-memory test database and pytest-cov combines the per-worker data. PYTEST_XDIST_WORKERS=0 runs the suite in one process again when debugging; pytest bedrock outside Docker was already serial.

What parallel ordering exposes

Running in parallel changes which test precedes which, and that surfaces state one test leaves behind for the next. Doing the same change in Springfield turned up three such leaks, and the fixes come across in bedrock/conftest.py as autouse fixtures for everything under bedrock/:

  • The Django cache is cleared before each test. Wagtail keeps its site root paths there and waffle keeps switch state there. A test's database rollback invalidates neither, so a later test can build URLs from root pages that no longer exist.
  • The URL resolvers are reset before each test. Django caches reversed URLs per language, keyed on its own get_language(), while the locale prefix inside them comes from LocalePrefixPattern, which reads the language through lib.l10n_utils.translation. A test that mocks django.utils.translation.get_language while a URL is reversed fills the cache entry for the real language with the mocked language's prefix, and every later test on that worker reverses wrong URLs. Serial runs never see it because an earlier test has always populated the cache first. Clearing costs under 9 ms per test.
  • The active language is deactivated after each test. Requests to locale-prefixed URLs activate a language inside the URL resolver and nothing resets it.

AcceptedLocalesTest used to move the real data/www-l10n checkout aside for the class's duration, which under xdist can leave another worker reading an empty Fluent repo. It now builds its stand-in locale tree in a temporary directory and patches FLUENT_REPO_PATH to point at it.

Integration tests

The integration runner moves from pytest-parallel, which has had no release since 2021 and hangs on macOS, to xdist as well. PYTEST_PROCESSES maps straight onto -n and accepts the same values the workflows already set (1, auto). pytest-parallel and its only dependant, tblib, leave the lock. Worth a run-integration-tests push before merge.

Ported from mozmeao/springfield#2027.

The main pytest run has been taking six to eight minutes in CI in a
single process. With -n auto on the four-vCPU hosted runners each
worker gets its own in-memory test database and pytest-cov combines
the per-worker data. PYTEST_XDIST_WORKERS=0 runs the suite in one
process again when debugging.

Running in parallel changes which test precedes which, and that exposes
state one test leaves for the next. Three autouse fixtures in
bedrock/conftest.py close the gaps found while doing the same in
Springfield: the Django cache is cleared before each test, because
Wagtail keeps its site root paths there and a database rollback does
not invalidate them; the URL resolvers are reset, because Django caches
reversed URLs per language while our LocalePrefixPattern reads the
language through a function some tests mock, so a mocked test can
poison the cache for the real language; and the active language is
deactivated after each test.

AcceptedLocalesTest used to move the real data/www-l10n checkout aside
for the duration of the class, which under xdist can leave another
worker reading an empty Fluent repo. It now builds its stand-in locale
tree in a temporary directory and patches FLUENT_REPO_PATH to point at
it.

The integration test runner moves from pytest-parallel, which has had
no release since 2021 and hangs on macOS, to xdist as well.
PYTEST_PROCESSES maps straight onto -n, and tblib leaves the lock with
it.
Copilot AI balanced review requested due to automatic review settings November 12, 2026 14:35
@stevejalim
stevejalim marked this pull request as ready for review November 12, 2026 14:36
@stevejalim
stevejalim requested a review from a team as a code owner November 12, 2026 14:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Per-worker collection with live HTTP-generated download parametrization requires deployment-backed execution of the full integration matrix to verify consistent collections and report generation.

No open issues 🎉

What changed in this PR

This PR enables parallel Python testing with pytest-xdist and improves test isolation to prevent shared state and filesystem changes from interfering with workers.

Changes:

  • Replace pytest-parallel with pytest-xdist for unit and integration test runners, retaining configurable worker counts.
  • Add per-test cache, URL resolver, and language-state cleanup under bedrock/.
  • Move locale test data into a temporary directory instead of modifying the shared localization checkout.
File Description
uv.lock Lock pytest-xdist and execnet; remove pytest-parallel and tblib.
pyproject.toml Replace the parallel-testing dependency with pytest-xdist.
bin/​run-tests.sh Enable automatic worker selection with a serial-debugging override.
bin/​integration_tests/​run_integration_tests.sh Map the existing process setting to xdist’s -n option.
bedrock/​conftest.py Add fixtures that isolate cache, URL resolver, and language state.
bedrock/​base/​tests/​test_accepted_locales.py Use a temporary locale tree and patch its configured path.

Review effort: Balanced

@stevejalim

Copy link
Copy Markdown
Contributor Author

@copilot please fix the merge conflicts in this pull request.

# Conflicts:
#	bin/run-tests.sh

Co-authored-by: stevejalim <101457+stevejalim@users.noreply.github.com>

Copilot AI commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

@copilot please fix the merge conflicts in this pull request.

Resolved in merge commit 2c9b969. Bash syntax and pytest argument checks passed; the Python suite could not run because pytest is not installed.

@janriokrause

Copy link
Copy Markdown
Contributor

Thanks for this, the parallel run is a nice win on a big machine. 😊

I ran the suite several times. 3 runs each:

  • main, serial: 44s
  • PR, -n auto (18 cores): 33s. Faster, same results as main.
  • PR, -n 4: 42s. About the same as main.
  • PR, -n 2: 50s. Slower than main.
  • PR, -n 0: 71s. Much slower than main.

So the gain only shows up with many cores. With few cores or serial, the autouse fixtures in bedrock/conftest.py eat it. I tried removing each one (-n 0, 1 run each):

  • Without clear_url_caches(): about 18s saved.
  • Without cache.clear(): about 12s saved.

Do you know how many cores CI has? If it's 4 or fewer, this may not speed it up.

If you get a chance, could the fixtures be scoped? E.g. clear only after tests that mock get_language, or fix those tests directly.

Smaller question: run_integration_tests.sh defaults to PYTEST_PROCESSES=1. That now starts an xdist worker where --workers 1 ran in-process. Intended for cdn_tests.yml? I haven't tested the integration tests.

@stevejalim

Copy link
Copy Markdown
Contributor Author

@janriokrause the runners have 4 vcpu and based on what we saw on Springfield we do get a speed bump. Let's merge and see

@stevejalim
stevejalim merged commit c295e9f into main Nov 12, 2026
11 checks passed
@stevejalim
stevejalim deleted the ci-pytest-xdist branch November 12, 2026 19:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants