environmentd: bound HTTP and SUBSCRIBE output buffering to prevent OOM (SQL-428, SQL-423) - #37905
Open
mtabebe wants to merge 2 commits into
Open
environmentd: bound HTTP and SUBSCRIBE output buffering to prevent OOM (SQL-428, SQL-423)#37905mtabebe wants to merge 2 commits into
mtabebe wants to merge 2 commits into
Conversation
mtabebe
force-pushed
the
ma/sql-428-oom-buffered-output
branch
4 times, most recently
from
July 28, 2026 18:18
4cc5337 to
13ce959
Compare
…output (SQL-428) Problem: A single `SELECT` over the HTTP endpoints could make environmentd buffer the entire result in memory and OOM, even when the client reads as fast as possible. The peek path already streams results out of the replica in bounded batches, so the growth was entirely the HTTP layer re-buffering those batches as an inflated `serde_json::Value` tree (~10-15x the wire size). Solution: - WebSocket: stream rows straight through. `SELECT` now flows through a new `StatementResult::Rows` variant whose sender streams `Rows(desc)` + per-row `Row` messages, flushing between stash batches for backpressure and enforcing `max_result_size` incrementally. Only one batch is resident, so large results over WebSocket get the same bounded memory profile as pgwire `SELECT`. The row descriptor is sent lazily so a query that errors before producing rows still emits only an `Error`. - Plain JSON: the response is one buffered document, so it stays buffered. But its footprint is smaller and its cap is now accurate. Rows are accumulated as pre-serialized `Vec<Box<serde_json::value::RawValue>>` instead of a `Value` tree, and `max_result_size` is enforced on the real serialized byte count. A genuinely oversized result fails fast with `ResultSize` instead of OOMing. The MCP and prometheus consumers of `SqlResult::Rows` parse each `RawValue` on demand. A streamed `SELECT` that errors mid-peek now reports `has_rows: true` over WebSocket (it commits to being row-returning before streaming), matching the `SUBSCRIBE` arm and pgwire.
Problem:
A single slow or stopped SUBSCRIBE client could make environmentd buffer the
whole subscribe output in memory and OOM. The replica -> controller ->
coordinator response path is unbounded at every hop, and the producer runs on
the non-blockable coordinator main loop, so it cannot apply backpressure to a
slow client.
Solution:
Bound the terminal buffer and retire the sink.
`ActiveSubscribe` gains a shared byte counter and a per-sink budget.
Each `send()` adds the batch's byte size and carries it on the channel.
A thin receiver wrapper, installed where the subscribe `rx` is built, subtracts
it as the client writer drains each batch, so the counter tracks the currently
buffered depth.
After each `process_response`, the coordinator retires the subscribe if the
buffered depth exceeds the budget, surfacing a new
`AdapterError::SubscribeFellBehind` ("SUBSCRIBE fell behind ...") with a hint to
use a non-buffering client or `COPY (SUBSCRIBE ...) TO STDOUT`.
The budget is a new dyncfg `subscribe_max_buffered_bytes`, default 128 MiB.
mtabebe
force-pushed
the
ma/sql-428-oom-buffered-output
branch
from
July 28, 2026 20:24
13ce959 to
5854351
Compare
mtabebe
marked this pull request as ready for review
July 28, 2026 21:09
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem:
A single query can drive environmentd to OOM, even when the client read as
fast as it could, because two paths buffered unbounded output in the shared
process.
SELECT. The peek path already streams results out of the replica inbounded batches, but the HTTP SQL layer re-buffered every batch into a
serde_json::Valuetree roughly 10-15x the wire size. A largeSELECTover/api/wsor/api/sqlgrew memory without limit. The/api/sqlsize guardwas a stale
TODOand was never enforced.is unbounded at every hop, and the subscribe producer runs on the
non-blockable coordinator main loop, so it cannot apply backpressure to a slow
client. One slow or stopped SUBSCRIBE consumer could make the coordinator
buffer the entire subscribe output in memory.
Solution:
HTTP SQL layer (SQL-428).
/api/wsstreams rows straight through.SELECTnow flows through anew
StatementResult::Rowsvariant whose sender emitsRows(desc)plusper-row
Rowmessages, flushing between stash batches so a slow client appliesbackpressure and only one batch is resident.
max_result_sizeis enforcedincrementally. Large results over WebSocket now get the same bounded memory
profile as pgwire
SELECT./api/sqlis a single buffered document. Rows are accumulated aspre-serialized
RawValues instead of aValuetree, andmax_result_sizeisenforced on the actual serialized byte count. An oversized result fails fast
with
ResultSize.SqlResult::Rowsparse eachRawValueondemand.
SUBSCRIBE / COPY-TO buffer (SQL-423).
Bound the terminal buffer and retire the sink when it overflows.
ActiveSubscribecarries a shared byte counter and a per-sink budget. Each send adds the batch's
byte size and a receiver wrapper subtracts it as the client writer drains, so the
counter tracks the currently buffered depth. After each response the coordinator retires the subscribe if the depth exceeds the budget, surfacing
AdapterError::SubscribeFellBehindwith a hint to use a non-buffering client or
COPY (SUBSCRIBE ...) TO STDOUT.The budget is a new dyncfg
subscribe_max_buffered_bytes(default 128 MiB).COPY (SUBSCRIBE ...) TO STDOUTshares the channel and is covered too.Testing:
New integration tests in
src/environmentd/tests/server.rs:test_ws_select_streams_rows: aSELECTover/api/wsemits oneRowmessageper row before
CommandComplete, confirming the non-buffering path.test_http_sql_result_size_limit: withmax_result_sizelowered to 1 MB, anoversized
/api/sqlresult fails with a result-size error while a small one succeeds.test_subscribe_buffer_bound: withsubscribe_max_buffered_bytesset to 1 KiB,a COPY-wrapped SUBSCRIBE whose client never reads surfaces the "fell behind"
error instead of growing without bound.