[Bug]: Post-navigation page operations have no timeout, and page cleanup is skipped on task cancellation — pages can leak and crawls can hang forever

[SohamKukreti]

Summary

Found while investigating #2202 (Docker containers accumulating renderer processes for weeks). Two related defects in async_crawler_strategy.py make it possible for a crawl to hang indefinitely and for its page to never be closed.

1. Nothing after navigation has a timeout

page_timeout only bounds page.goto() (async_crawler_strategy.py:762-764). Every page interaction after navigation is an un-timed call into the page's JS engine:

• overlay/consent removal — remove_overlay_elements (:1550)
• body-visibility check, css_selector extraction (:1102), image-dimension updates
• page.content() (:1115)
• shadow-DOM flattening, iframe processing, user js_code

The adapter is a plain pass-through to page.evaluate (browser_adapter.py:61-65), which has no timeout in Playwright. The repo never calls set_default_timeout except inside if self.config.accept_downloads: (browser_manager.py:1215-1217) — and default timeouts would not cover evaluate anyway. Only scan_full_page is wrapped in asyncio.wait_for.

A page whose main thread goes busy after DOMContentLoaded (bad loop, broken ad script, hostile page) therefore hangs the crawl forever, holding its page (= one renderer process) open. Verified with a page that starts a busy loop right after DCL: navigation succeeds, then arun() never returns, far past page_timeout.

2. The page-closing finally doesn't survive cancellation

The cleanup block (async_crawler_strategy.py:1216-1231) guards with except Exception. asyncio.CancelledError is a BaseException, so a task cancelled during the first await in that block (release_page_with_context) propagates out and page.close() is never reached — the page and its context refcount leak. Reachable from dispatcher/stream teardown paths that cancel in-flight tasks.

Impact

In long-lived processes (the Docker server's browser pool, any SDK user reusing a crawler), leaked pages accumulate indefinitely. Combined with the launch flags that disable background throttling, each leaked page also burns CPU forever. This is the core mechanism behind the multi-week renderer accumulation in #2202.

Proposed fix

• Wrap the post-navigation phase (or at minimum every evaluate/content() call) in asyncio.wait_for with a budget derived from page_timeout, so a crawl always terminates.
• Make the cleanup finally cancellation-safe: catch BaseException (re-raising CancelledError after the page is closed) or shield the close.