Build permanent browser with egress-hardened default config so its pool signature matches

[SohamKukreti]

Summary

Partially fixes #2204

The Docker server starts a "permanent" always-warm browser at boot so default-config requests can skip browser startup. On develop it never serves a single request: its pool signature (_sig() = SHA1 of BrowserConfig.to_dict()) is computed from a raw config that skips enforce_egress, so it has proxy_config: None. Every real request goes through enforce_egress, which sets proxy_config to the pinning proxy URL. The two signatures can never be equal, so get_crawler() never returns PERMANENT and every container carries an idle ~270 MB Chromium + Playwright driver tree that the janitor never reclaims. This is the mysterious second driver → browser tree in #2202.

This PR builds the permanent browser via get_default_browser_config() (the same helper /html, /screenshot, /pdf, /execute_js already use), so the permanent browser gets the same egress hardening and therefore the same signature as those endpoints. Side effect: the permanent browser now also routes through the pinning proxy and has TLS verification on, which it did not before.

The pinning proxy port is random per boot, so the signature changes on every restart. That is fine: DEFAULT_CONFIG_SIG lives only in memory and is recomputed on each boot; nothing persists signatures (monitor only persists endpoint stats to Redis).

What this does not fix (deliberately left for a follow-up)

/crawl, /crawl/stream and /crawl/job still miss the permanent browser. They build BrowserConfig from the request body only (BrowserConfig.load(body)), never from config.yml. The permanent browser is built from config.yml, which adds text_mode: true and four extra_args. So even an empty browser_config (or no browser_config at all) hashes differently and lands in the cold pool.

Making /crawl match requires a decision that changes user-visible behaviour: merge config.yml browser defaults into every /crawl body before hashing (and then decide whether text_mode: true — which disables JS — should stay in config.yml). Keeping that out of this PR so this one stays a one-line, behaviour-neutral fix.

Also not in this PR (known, from the same investigation, in POST /monitor/actions/restart_browser): the route holds LOCK and then calls init_permanent, which re-acquires the same non-reentrant lock (deadlock); it imports PERMANENT by value so the global is never reset after close; and it rebuilds the config without enforce_egress, which would re-introduce this exact mismatch after a manual restart.

List of files changed and why

• deploy/docker/server.py — in lifespan(), replace the raw BrowserConfig(extra_args=..., **kwargs) passed to init_permanent with get_default_browser_config(), so the permanent browser's config (and thus its pool signature) goes through enforce_egress like every other server-built config.

How Has This Been Tested?

Built the image locally from this branch (docker build -t crawl4ai:ticket4-a .) and ran it with -e CRAWL4AI_API_TOKEN=... -p 11299:11235.

Before (0.9.2 / develop):

• Fresh container: 1 driver → chrome-headless-shell tree, no --proxy-server flag on it.
• After one /crawl with a plain body: 2 trees. Log shows 🆕 Creating new browser in cold pool, never 🔥 Using permanent browser.

After (this branch):

• Fresh container: 1 tree; the permanent browser's Chromium command line now carries --proxy-server=http://127.0.0.1:<egress port> matching the egress pinning proxy listening on ... log line.
• POST /html {"url":"https://example.com"} → log shows 🔥 Using permanent browser, still 1 tree.
• POST /crawl {"urls":["https://example.com"]} (with and without browser_config) → still 🆕 Creating new browser in cold pool (sig=d32dbb0a), 2 trees. Expected — see "What this does not fix".

Commands used:

docker exec <ctr> ps -eo pid,args | grep chrome-headless-shell | grep -v -- --type= \
  | grep -o -E "^ *[0-9]+|--proxy-server=[^ ]+|--disable-javascript"
docker logs <ctr> 2>&1 | grep -E "permanent|cold pool|hot pool"

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas (one-line change, no comment needed)
• I have made corresponding changes to the documentation (no user-facing behaviour change)
• I have added/updated unit tests that prove my fix is effective or that my feature works (verified live in the container, see above)
• New and existing unit tests pass locally with my changes (not run — no test touches this path)