Skip to content

fix: restart local-eval polling thread after fork (#77)#203

Draft
khvn26 wants to merge 2 commits intomainfrom
fix/local-eval-fork-77
Draft

fix: restart local-eval polling thread after fork (#77)#203
khvn26 wants to merge 2 commits intomainfrom
fix/local-eval-fork-77

Conversation

@khvn26
Copy link
Copy Markdown
Member

@khvn26 khvn26 commented May 1, 2026

Closes #77.

The polling thread that refreshes the local evaluation environment document is created in the parent process. Threads do not survive os.fork(), so pre-fork servers (gunicorn, uwsgi, multiprocessing) end up with worker processes whose Python Thread object is intact but whose underlying OS thread is dead. is_alive() correctly reports False, no polls happen, and workers serve a frozen environment-document snapshot from the moment of fork for their entire lifetime.

EnvironmentDataPollingManager now composes a threading.Thread instead of inheriting from it, and registers an os.register_at_fork(after_in_child=...) hook so a fresh polling thread is started in each child. The hook also closes the parent's requests.Session because connection-pool sockets are inherited as shared FDs across fork; reusing them would interleave bytes between processes.

Notes

  • The same problem applies to the SSE event stream thread and the analytics processor thread. Perhaps we need a better approach than simply multiplying threads.

Test plan

  • New regression test test_polling_manager_keeps_polling_after_fork spawns a multiprocessing.get_context("fork").Process, asserts is_alive=True in the child, and asserts the child's ident differs from the parent's (i.e. it is a fresh thread, not the dead inherited one).
  • All existing tests pass (95/95).
  • mypy clean.

Notes

  • pytest-httpserver added as a dev dependency for the new test fixture.

The polling thread that refreshes the local evaluation environment
document is created in the parent process. Threads do not survive
os.fork(), so pre-fork servers (gunicorn, uwsgi, multiprocessing)
end up with worker processes whose Python Thread object is intact
but whose underlying OS thread is dead. Workers then serve a frozen
environment-document snapshot from the moment of fork for their
entire lifetime.

EnvironmentDataPollingManager now composes a threading.Thread instead
of inheriting from it, and registers an os.register_at_fork hook so a
fresh polling thread is started in each child. The hook also closes
the parent's requests.Session so connection-pool sockets are not
shared across processes.

Stream and analytics threads are not yet covered; tracking
separately.
@khvn26 khvn26 requested a review from a team as a code owner May 1, 2026 18:30
@khvn26 khvn26 requested review from emyller and removed request for a team May 1, 2026 18:30
@khvn26 khvn26 marked this pull request as draft May 1, 2026 18:30
The pytest-httpserver dev dependency requires python>=3.10. Move the
fork test (the only consumer) into its own file so the four
pre-existing polling-manager tests still run on 3.9, and skip the new
file there via importorskip. Add a mypy override so 3.9 type-checks
do not error on the missing module.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Local evaluation with multiprocessing environment

1 participant