[Bug]: Browser pages and renderer processes accumulate for weeks on non-streaming /crawl (Docker 0.9.2)

[kesha1225]

crawl4ai version

0.9.2 (Docker image unclecode/crawl4ai:0.9.2, C4AI_VERSION=0.9.2)

Expected Behavior

After a non-streaming POST /crawl completes, the pages and browser contexts it opened are closed. A container that has served no traffic for two weeks sits at near-zero CPU and holds only the warm browser pool.

Current Behavior

The container had been up 29 days serving a low volume of non-streaming /crawl requests (a handful per day, max_depth=2, at most 5 pages). At inspection it held 46 chromium processes and had not served a request for 15 days:

 3  chrome-headless-shell (browser root)
 2  playwright/driver/node
35  chrome-headless-shell --type=renderer
 4  --type=zygote
 3  --type=utility
 2  --type=gpu

945 MB RSS across those processes, 957 MiB for the container, and a steady 31% of a CPU core with zero traffic — two consecutive docker stats samples read 30.69% and 30.86%.

Renderer start timestamps cluster on the days crawls actually ran: dozens on Jul 29 between 19:41 and 23:21, isolated ones later, the newest on Aug 11. A --type=renderer process lives exactly as long as its page, so those pages were never closed and the renderers outlived their requests by weeks.

There are also two independent driver → browser root branches, started Jul 28 14:25:55 and Jul 29 17:21:15. The older branch was never torn down when the newer one started.

Restarting the container dropped it to 7 chromium processes, 426 MiB and 0.4% CPU.

Is this reproducible?

Yes

Inputs Causing the Bug

Plain non-streaming request — no session id, no hooks:

{
  "urls": ["<single http(s) url>"],
  "browser_config": {"type": "BrowserConfig", "params": {"headless": true, "accept_downloads": false}},
  "crawler_config": {"type": "CrawlerRunConfig", "params": {
    "stream": false,
    "cache_mode": "bypass",
    "wait_until": "domcontentloaded",
    "page_timeout": 30000,
    "process_iframes": false,
    "remove_overlay_elements": true,
    "exclude_external_images": true
  }}
}

Steps to Reproduce

1. Run unclecode/crawl4ai:0.9.2 in Docker with no memory or CPU limit on a cgroup v2 host.
2. Send the request above a few times a day for a few days, then stop sending traffic.
3. After some days, inspect the process table inside the container.

OS

Linux, Docker, cgroup v2 host

Python version

3.12.13 (image default)

Browser

chromium_headless_shell-1228, bundled with the image

Error logs & Screenshots

Nothing is logged and the container reports healthy the whole time.

Possibly related: #943 (closed, fixed in 0.8.5), #2083, #2155, #2123. This container had no memory limit, so per #2123 the memory guard reads host RAM instead of container usage — that may be why nothing reclaims these pages.

[nightcityblade]

Hi, I'd like to work on this. I'll submit a PR shortly.

[nightcityblade]

I’m withdrawing my claim so I don’t block anyone. The report points to a long-lived browser lifecycle leak, but it does not have a compact, reliably reproducible fix suitable for the small scoped change I can validate here.

[SohamKukreti]

Thanks for the exceptionally detailed report — the process tree, the renderer timestamps, and the two driver branches made this diagnosable. I've reproduced the behavior against the 0.9.2 image and traced each symptom to a specific cause. Confirming everything you observed, and adding the root causes.

What I verified live

• The two driver → browser branches: reproduced. A fresh container has one tree; a single plain /crawl creates a second. The first tree is the pool's permanent browser, started at boot — but its config is fingerprinted before the egress proxy (random port each boot) is written into request configs (server.py:199 vs api.py:687-690, hashed in crawler_pool.py:46-49). No request signature can ever match it, so it serves nothing, and the janitor only sweeps HOT_POOL/COLD_POOL — never PERMANENT. Your Jul 28 branch is that browser; the Jul 29 branch is the pool browser doing the real work. Tracked in [Bug]: Docker pool's permanent browser never serves any request - signature is computed without the egress proxy, so no /crawl can ever match it #2204.

• Renderers outliving requests by weeks: reproduced. page_timeout only bounds navigation (async_crawler_strategy.py:762-764). Everything after it — overlay removal (your config had remove_overlay_elements: true), body-visibility checks, page.content() — is a page.evaluate with no timeout, and the repo never sets a default. A page that keeps its main thread busy after DOMContentLoaded hangs the crawl indefinitely: I verified navigation succeeds, then the request never returns, well past page_timeout. Each hang leaves one renderer alive and, because the handler's finally never runs, the pooled browser's active_requests counter never decrements — and the janitor explicitly skips any browser with active_requests > 0 (crawler_pool.py:180-181). That pins the browser forever, which is why your Jul 29 browser survived 15 idle days and why later crawls kept stacking renderers into the same tree (your Aug 11 one). Verified: a hung browser survived well past the idle TTL while a cleanly-released control browser was reaped on schedule. The core-library side of this is tracked in [Bug]: Post-navigation page operations have no timeout, and page cleanup is skipped on task cancellation — pages can leak and crawls can hang forever #2205.

• The constant ~31% CPU: every browser launches with --disable-background-timer-throttling / --disable-renderer-backgrounding (browser_manager.py:1100-1104), so leaked pages run their JS timers at full foreground rate forever. My hung test renderers each pinned a full core on an idle container.

• The memory guard: your hunch about [Bug]: Docker memory guard reads host RAM when no container limit is set — cgroup v2 "max" defeats get_container_memory_percent #2123 is exactly right. On cgroup v2 with no container memory limit, /sys/fs/cgroup/memory.max contains the literal max; int("max") raises, the bare except in utils.py:411-434 swallows it and returns the host's memory percent. So nothing ever tightened the janitor or refused work over those 29 days.

Also: the leaked-Playwright-driver issue (#2155) is fixed on develop (701b0b1) but not in 0.9.2 — a separate, additive source of stray node processes.

Plan

Splitting the fixes so each is reviewable on its own:

1. This issue ([Bug]: Browser pages and renderer processes accumulate for weeks on non-streaming /crawl (Docker 0.9.2) #2202) → make the pool un-pinnable: cancellation-safe active_requests release, a default per-crawl wall-clock deadline, and a janitor backstop that force-closes a browser stuck "busy" beyond any legal crawl duration.
2. [Bug]: Post-navigation page operations have no timeout, and page cleanup is skipped on task cancellation — pages can leak and crawls can hang forever #2205 → bound every post-navigation page call with a timeout derived from page_timeout, and make the page-closing finally survive CancelledError.
3. [Bug]: Docker pool's permanent browser never serves any request - signature is computed without the egress proxy, so no /crawl can ever match it #2204 → fix the permanent-browser fingerprint so it actually serves default-config requests.
4. [Bug]: Docker memory guard reads host RAM when no container limit is set — cgroup v2 "max" defeats get_container_memory_percent #2123 → the cgroup v2 memory read.