Gateway: make Timeout recoverable; bridge: drop the per-client RwLock

[ehsan6sha]

Backend half of the FxFiles web-hang investigation. The client-side work is already shipped; this is what it was routing around.

Not for release yet — no version bump, no tag. Opening for CI (cargo test --workspace, wasm clippy, wasm-pack test --headless --firefox) and for review of the lock removal.

1. f4d6635 — gateway: let a block Timeout reach the gc-recovery path

The gateway already has a full gc-recovery system (recovery_fallback.rs: resolve key→CID from the cluster pinset mirror and serve the block). Production could not reach it.

Recovery is invoked only from the miss branches gated on NotFound/Unavailable. A read that fails with Timeout isn't in that set, so:

• try_recover_block never runs — the cluster mirror that still holds the data is never consulted;
• Timeout has no arm in ApiError::error_code(), so it hits the _ catch-all → InternalError → HTTP 500;
• a 5xx trips the client SDK's health gate, marking the entire backend down — exactly what the deliberate Unavailable → 410 mapping exists to prevent.

Why Timeout survives instead of becoming Unavailable: cluster_fallback::classify_exhaustion only rewrites it when the local daemon's offline probe returns a clean miss. Read-only inspection of the live host found load ~12.8, fula-gateway at 12.4 GB RSS, and kubo answering /id in 45 ms but unable to answer repo/stat within 30 s — so that probe errors out, responsive is false, and the Timeout passes through. A bucket whose data is still pinned in the cluster therefore 500s.

Also fixes a second gap: open_bucket_for_user was called with a bare ?. It reads the prolly index root — exactly the block class ipfs repo gc orphans — so a root-block miss bypassed the recovery gate one match below (Unavailable → 410 with no recovery and no client by-CID fallback, which fires only on 404; NotFound → 500). Both now route through the same gate, factored into recover_or_nosuchkey() so the two call sites can't drift.

Trade-off: a genuinely transient infra timeout now attempts recovery and, failing that, surfaces as 404 rather than 500. Better for clients, but a real outage reads as "not found" on this path. Errors that mean the daemon is actually unreachable (Connection, IpfsApi, …) are deliberately excluded and still propagate as 5xx, so the health gate still trips on a true outage.

This makes a damaged bucket fail gracefully. It does not repair one, and it does not address why the host is resource-saturated.

2. 984d6ed — bridge: drop the per-client RwLock

One slow bucket froze the entire web app. EncryptedClientHandle.inner was Arc<RwLock<EncryptedClient>> — a single lock, not per-bucket. load_forest took the exclusive guard and held it across the whole network fetch. Captured live on a phone:

loadForest: ENTER tag-metadata            <- ~30s, holds the write guard
getFlat:    ENTER website-metadata-v8/..  <- never completes
loadForest: ENTER tag-metadata-v8         <- never completes (healthy, normally 0.1s)
WebsiteDetail: loadWebsites ENTER         <- never completes

A Dart-side .timeout() can't rescue this: FRB exposes no cancel handle for load_forest, so the Rust future keeps running and keeps the guard after the caller gives up.

The lock is deleted, not made per-bucket, because:

• EncryptedClient has zero &mut self methods — all ~180 take &self. The lock never protected mutability.
• The SDK already owns per-bucket locking: forest_cache (DashMap), migration_locks, bucket_write_mutex — the last documented as "the OUTERMOST per-bucket lock … per-bucket, so different buckets continue to flush in parallel." A bridge-level per-bucket map would duplicate that one layer up, and wouldn't even cover calls carrying no bucket (get_public_key, export_secret_key).
• The read/write split encoded no invariant: save_forest (read) and flush_forest (write) had byte-identical bodies; rewrap_object took write() for one object while rotate_bucket took read() for a whole bucket.
• No bridge fn composed two SDK calls under one guard.
• FulaClientHandle next door already uses a bare Arc — in-repo precedent.

57 acquisitions removed (forest 27, encrypted 18, sharing 5, rotation 4, chunked 3). Arc<EncryptedClient> derefs to &EncryptedClient, so every downstream guard.… call is untouched.

3. 5060ab2 — wasm timeout via reqwest's own mechanism

ClientBuilder::timeout is a no-op on wasm32, so Config::timeout was silently dropped in the browser — no request timeout at all; a stalled fetch could hang for the tab's lifetime holding SDK locks.

I first hand-rolled a futures::select against a gloo-timers timer, then verified the claim that dropping the loser aborts the fetch. It does (AbortGuard::drop → ctrl.abort()), but the same file showed the racing was unnecessary — reqwest's wasm backend honours a per-request timeout:

reqwest-0.12.24/src/wasm/request.rs:264  pub fn timeout(mut self, ..) -> RequestBuilder
reqwest-0.12.24/src/wasm/mod.rs:64       AbortGuard::timeout -> ctrl.abort_with_reason(..)
reqwest-0.12.24/src/wasm/client.rs:229   AbortGuard::new() -> abort.timeout(*timeout) -> init.signal(..)

Replaced by one cfg-gated line. That removed the helper, the new ClientError::Timeout variant and its FulaError arm — so the public ClientError enum is unchanged (no downstream exhaustive-match break), and a browser timeout surfaces as an ordinary reqwest::Error with is_timeout(), identical in shape to native.

Review focus

The lock removal is the part that needs eyes. Known residual, stated rather than hidden: rewrap_object / rotate_bucket are the only write-guard sites whose SDK method takes no per-bucket lock, so key rotation can now overlap with an upload on the same bucket. I deliberately did not add a lock inside rewrap_object_dek — rotate_bucket_inner calls it through buffer_unordered(MAX_CONCURRENT_REWRAPS), so that would serialise a deliberately-parallel operation. Rotation already mutates the forest concurrently within itself by design, and it has no call site in FxFiles today; this belongs in the SDK's own lock ordering rather than here.

Verified locally: cargo check clean on native and wasm32-unknown-unknown at each commit.