Skip to content

Add opt-in crawl_timeout and shield page cleanup from cancellation - #2236

Open
SohamKukreti wants to merge 1 commit into
developfrom
fix/crawl-timeout-2205
Open

Add opt-in crawl_timeout and shield page cleanup from cancellation#2236
SohamKukreti wants to merge 1 commit into
developfrom
fix/crawl-timeout-2205

Conversation

@SohamKukreti

Copy link
Copy Markdown
Collaborator

Fixes #2205

A page whose JS thread goes busy after navigation hung arun() forever. page_timeout only covers goto; page.evaluate and page.content() have no timeout and are not covered by set_default_timeout, so every later step waited indefinitely and the page was never released. A task cancel landing inside the finally cleanup also skipped page.close(), leaking the page.

This PR:

  • Adds CrawlerRunConfig.crawl_timeout (ms, default None = no limit). It wraps the whole page visit, from navigation to final HTML including js_code and hooks, in asyncio.wait_for. On expiry the page is force-closed (a session is dropped, with a warning) and the crawl fails with Crawl exceeded crawl_timeout of N ms.
  • Runs the finally cleanup as a shielded task and re-awaits it on cancel, so page.close() always completes before the cancel propagates. The two console cleanup calls are bounded to 5 s so a hung page cannot block the cancel.
  • Docker: base_config sets crawl_timeout to 180 s. Untrusted requests are capped at 60 s, same as the other timeouts.

Known gaps, by design:

  • The Docker default reaches only the /crawl batch path; the other endpoints build CrawlerRunConfig() directly (follow-up).
  • Docker hooks that take more than 180 s now fail where they previously ran under the 300 s server deadline.

List of files changed and why

  • crawl4ai/async_configs.py - new crawl_timeout parameter (docstring, constructor, to_dict), added to the untrusted allowlist and the timeout cap.
  • crawl4ai/async_crawler_strategy.py - _crawl_web wraps the new _crawl_page in asyncio.wait_for when crawl_timeout is set; finally cleanup moved into a shielded _cleanup() task with bounded console calls; new _close_unresponsive_page to force-close the page or drop the session on timeout.
  • deploy/docker/config.yml - crawl_timeout: 180000 under crawler.base_config.
  • docs/md_v2/api/parameters.md - parameter table row.
  • docs/md_v2/core/browser-crawler-config.md - field list entry.
  • docs/md_v2/core/page-interaction.md - "Timing Control" entry.
  • tests/test_crawl_timeout.py - new tests (see below).

How Has This Been Tested?

tests/test_crawl_timeout.py (7 tests, all pass) uses a local HTTP server that serves a "trap" page whose JS thread goes busy after load:

  • trap page fails within crawl_timeout with the expected error, page closed, context refcount 0
  • trap page with remove_overlay_elements=True does not hang
  • trap page in a session drops the session and the same session_id works again
  • crawl_timeout=None keeps the old behaviour (crawl still running after 8 s), cancel leaves refcount 0
  • keep-last-page rule for headless/managed browsers unchanged
  • cancel landing inside the cleanup still closes the page (refcount 0)
  • an evaluate inside cleanup on a hung page does not block the timeout, refcount 0

Regression: tests/browser (excluding the docker dir) and the config tests give an identical pass/fail list on this branch and on clean develop; the failures there are pre-existing (missing asyncio markers, import errors).

Live crawls (headless, Python 3.13): 8 real sites with and without crawl_timeout=30000 succeed with the same HTML; js_code, wait_for, screenshot, console and network capture, scan_full_page, overlay removal and delay_before_return_html all work under the timer; trap page fails in 5.5 s and the next crawl works; session trap drops the session and the session id is reusable; arun_many with a trap mixed in finishes in about the timeout with only the trap failing; 5 mid-crawl cancels leave refcount 0. No orphan Chrome processes after the runs.

Also verified: headed managed Chrome survives a forced close of its last tab (a blank tab is opened first), and dump/load/clone/from_kwargs preserve crawl_timeout.

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have added/updated unit tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

…on (#2205)

crawl_timeout (ms, default None) bounds the whole page visit and force-closes a hung page
the finally cleanup is shielded so a cancel can no longer skip page.close()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant