Summary
Found while investigating #2202. The Docker server's "permanent" warm browser is dead weight: it is started at boot, serves zero requests, is never cleaned up, and causes every container to run a second browser tree from the first request onward.
Root cause
The pool matches requests to browsers by hashing the full BrowserConfig (crawler_pool.py:46-49), and proxy_config is part of to_dict() (async_configs.py:968).
init_permanent() is called at startup with a config built straight from config.yml — without enforce_egress() (server.py:199-203), so its fingerprint has proxy_config: None.- Every
/crawl request's config goes through enforce_egress() first (api.py:687-690), which sets proxy_config to the egress pinning proxy — whose port is random each boot (egress_broker.py:198-201).
The two fingerprints can never be equal, so _is_default_config() (crawler_pool.py:60) never matches and PERMANENT is never returned. Note get_default_browser_config() (server.py:126-141) does apply enforce_egress, so even the server's own /html, /screenshot, /pdf, /execute_js endpoints miss it.
Additionally, the janitor only sweeps HOT_POOL and COLD_POOL (crawler_pool.py:177,198) — PERMANENT is never inspected, so the unused browser also can never be reclaimed.
Impact
Verified
Reproduced on unclecode/crawl4ai:0.9.2: fresh container = one browser tree; after a single plain /crawl = two independent trees, permanent one idle.
Proposed fix
Keep the permanent browser, fix the match: build its config through the same path requests use (get_default_browser_config(), i.e. after enforce_egress), and compute DEFAULT_CONFIG_SIG from that. Alternatively/additionally, exclude the server-injected proxy_config from the pool signature, since the server sets it identically on every request.
Summary
Found while investigating #2202. The Docker server's "permanent" warm browser is dead weight: it is started at boot, serves zero requests, is never cleaned up, and causes every container to run a second browser tree from the first request onward.
Root cause
The pool matches requests to browsers by hashing the full
BrowserConfig(crawler_pool.py:46-49), andproxy_configis part ofto_dict()(async_configs.py:968).init_permanent()is called at startup with a config built straight fromconfig.yml— withoutenforce_egress()(server.py:199-203), so its fingerprint hasproxy_config: None./crawlrequest's config goes throughenforce_egress()first (api.py:687-690), which setsproxy_configto the egress pinning proxy — whose port is random each boot (egress_broker.py:198-201).The two fingerprints can never be equal, so
_is_default_config()(crawler_pool.py:60) never matches andPERMANENTis never returned. Noteget_default_browser_config()(server.py:126-141) does applyenforce_egress, so even the server's own/html,/screenshot,/pdf,/execute_jsendpoints miss it.Additionally, the janitor only sweeps
HOT_POOLandCOLD_POOL(crawler_pool.py:177,198) —PERMANENTis never inspected, so the unused browser also can never be reclaimed.Impact
driver → browserprocess branches after the first request (observed in [Bug]: Browser pages and renderer processes accumulate for weeks on non-streaming /crawl (Docker 0.9.2) #2202's process listing: one branch from boot, one from the first crawl a day later).text_modeoptimization configured for the default browser never applies to real traffic.Verified
Reproduced on
unclecode/crawl4ai:0.9.2: fresh container = one browser tree; after a single plain/crawl= two independent trees, permanent one idle.Proposed fix
Keep the permanent browser, fix the match: build its config through the same path requests use (
get_default_browser_config(), i.e. afterenforce_egress), and computeDEFAULT_CONFIG_SIGfrom that. Alternatively/additionally, exclude the server-injectedproxy_configfrom the pool signature, since the server sets it identically on every request.