Add opt-in crawl_timeout and shield page cleanup from cancellation - #2236
Open
SohamKukreti wants to merge 1 commit into
Open
Add opt-in crawl_timeout and shield page cleanup from cancellation#2236SohamKukreti wants to merge 1 commit into
SohamKukreti wants to merge 1 commit into
Conversation
…on (#2205) crawl_timeout (ms, default None) bounds the whole page visit and force-closes a hung page the finally cleanup is shielded so a cancel can no longer skip page.close()
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2205
A page whose JS thread goes busy after navigation hung
arun()forever.page_timeoutonly coversgoto;page.evaluateandpage.content()have no timeout and are not covered byset_default_timeout, so every later step waited indefinitely and the page was never released. A task cancel landing inside thefinallycleanup also skippedpage.close(), leaking the page.This PR:
CrawlerRunConfig.crawl_timeout(ms, defaultNone= no limit). It wraps the whole page visit, from navigation to final HTML includingjs_codeand hooks, inasyncio.wait_for. On expiry the page is force-closed (a session is dropped, with a warning) and the crawl fails withCrawl exceeded crawl_timeout of N ms.finallycleanup as a shielded task and re-awaits it on cancel, sopage.close()always completes before the cancel propagates. The two console cleanup calls are bounded to 5 s so a hung page cannot block the cancel.base_configsetscrawl_timeoutto 180 s. Untrusted requests are capped at 60 s, same as the other timeouts.Known gaps, by design:
/crawlbatch path; the other endpoints buildCrawlerRunConfig()directly (follow-up).List of files changed and why
crawl4ai/async_configs.py- newcrawl_timeoutparameter (docstring, constructor,to_dict), added to the untrusted allowlist and the timeout cap.crawl4ai/async_crawler_strategy.py-_crawl_webwraps the new_crawl_pageinasyncio.wait_forwhencrawl_timeoutis set;finallycleanup moved into a shielded_cleanup()task with bounded console calls; new_close_unresponsive_pageto force-close the page or drop the session on timeout.deploy/docker/config.yml-crawl_timeout: 180000undercrawler.base_config.docs/md_v2/api/parameters.md- parameter table row.docs/md_v2/core/browser-crawler-config.md- field list entry.docs/md_v2/core/page-interaction.md- "Timing Control" entry.tests/test_crawl_timeout.py- new tests (see below).How Has This Been Tested?
tests/test_crawl_timeout.py(7 tests, all pass) uses a local HTTP server that serves a "trap" page whose JS thread goes busy after load:crawl_timeoutwith the expected error, page closed, context refcount 0remove_overlay_elements=Truedoes not hangsession_idworks againcrawl_timeout=Nonekeeps the old behaviour (crawl still running after 8 s), cancel leaves refcount 0Regression:
tests/browser(excluding the docker dir) and the config tests give an identical pass/fail list on this branch and on cleandevelop; the failures there are pre-existing (missing asyncio markers, import errors).Live crawls (headless, Python 3.13): 8 real sites with and without
crawl_timeout=30000succeed with the same HTML;js_code,wait_for,screenshot, console and network capture,scan_full_page, overlay removal anddelay_before_return_htmlall work under the timer; trap page fails in 5.5 s and the next crawl works; session trap drops the session and the session id is reusable;arun_manywith a trap mixed in finishes in about the timeout with only the trap failing; 5 mid-crawl cancels leave refcount 0. No orphan Chrome processes after the runs.Also verified: headed managed Chrome survives a forced close of its last tab (a blank tab is opened first), and
dump/load/clone/from_kwargspreservecrawl_timeout.Checklist: