feat(webapp): per-client database pool metrics that survive the driver adapter - #4541
Conversation
…r adapter Report Prisma pool and query metrics for every configured client (control-plane writer/replica, run-ops writer/replica, legacy writer/replica) instead of only the control-plane writer, tagged with db_client and db_driver attributes. Pool figures come from the authoritative source per driver: pg.Pool (totalCount/idleCount/waitingCount + connect/remove counters) for driver-adapter clients, and the Rust engine metrics for quaint clients. Query counters and duration histograms come from prisma metrics for both. Adds a db.pool.connections.waiting gauge. Stops exporting Prisma metrics from the Prometheus /metrics route; pool observability now lives entirely in the OTel pipeline. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KHjfL7qXHia5DTnxi1RePS
|
Observability mapAs of 18/100 over 413 measured of 429 entry points (base 18, no change) What this PR changed FIX FIRST
AUDIT 3 of 50 sensitive mutations record an actor. 47 without one. What the score is made ofThe score and findings here are report-only and never gate the merge. Separately, a required test suite keeps this tool's symbol and route lists in sync with the code they name, and can fail a pull request that renames or removes a symbol they reference, or that adds the first route with a segment they anticipate. Each failure names the list to edit. The rules and their reasons: internal-packages/observability-map/README.md. |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughThe change adds shared database metric registration and normalization. Standard Prisma and driver-adapter clients register pool, connection, query, and histogram sources. The tracer collects metrics for every registered client and records client, driver, pool, and waiting-connection observations. The metrics route now serves 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 2📝 Generate docstrings 💡
🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…tribute Label each pool with its full datasource role (control-plane-writer, control-plane-replica, run-ops-writer, run-ops-replica, legacy-run-ops-writer, legacy-run-ops-replica) instead of the generic writer/reader, matching the db.datasource span attribute so metrics and traces correlate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KHjfL7qXHia5DTnxi1RePS
Back the metrics-source registry with singleton() keyed by clientType, matching the app's other process-wide registries and deduping so a re-evaluated module or a repeated label registers once. Document the removal of prisma_* from the Prometheus /metrics endpoint in the server-changes note. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KHjfL7qXHia5DTnxi1RePS
… note Run oxfmt on databaseMetrics.server.ts (code-quality check). Reword the server-changes note to a single user-facing sentence with no infra names, per the release-note guidance. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KHjfL7qXHia5DTnxi1RePS
A failed $metrics.json() previously zero-filled every engine-derived value, so the batch observer reported spurious zeros — including for cumulative counters (db.client.queries.total, db.datasource.queries.total, db.pool.connections.opened/closed.total), which reads as a counter reset and corrupts rate math. Restore the old fail-quiet behavior per client: the collector now signals engineMetricsAvailable and leaves engine-derived fields (counters/gauges/histograms, and quaint pool figures) undefined on failure, and the tracer skips those instruments for that client. Adapter clients still emit live pg.Pool figures because those don't depend on $metrics. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KHjfL7qXHia5DTnxi1RePS
What
Follow-up to #4539. The driver-adapter work is inert until a client flips to the pg driver adapter, but the moment one does, our database observability degrades: the OTel metrics pipeline reads pool stats from Prisma's
$metrics, which is owned by the Rust engine'squaintpool. Under the adapter,pg.Poolowns the pool, so those gauges read zero. The pipeline also only ever scraped a single client (the control-plane writer singleton).This PR makes database metrics driver-agnostic and per-client:
db_clientanddb_driver(quaint|pg-adapter) attributes.db_clientuses our canonical datasource-role labels (control-plane-writer,control-plane-replica,run-ops-writer,run-ops-replica,legacy-run-ops-writer,legacy-run-ops-replica) — the same strings used for thedb.datasourcespan attribute, so a metric and a trace point at the same pool.pg.Pool(totalCount/idleCount/waitingCount, plus cumulative opened/closed fromconnect/removeevents).$metricspool gauges/counters, exactly as before.$metricsfor both drivers (the Rust engine executes queries in both cases).db.pool.connections.waitinggauge (pg.Pool exposes this; quaint reports 0)./metricsroute. Pool observability now lives entirely in the OTel pipeline, per driver, per client.Why
So we can flip any client (including the control-plane writer, the primary desync-fix target) to the driver adapter without losing pool visibility. Existing dashboards keyed on the same metric names keep working; they gain a per-client dimension.
Testing
Unit (
apps/webapp/app/utils/databaseMetrics.server.test.ts): the pure normalizer — quaint reads pool from$metrics; adapter reads pool frompg.Pooland keeps engine query metrics;busynever goes negative; graceful zeroing when$metricsis unavailable (adapter still reports live pool figures).Live smoke test against a prod-shaped local stack: three physically-distinct Postgres DBs (control-plane, run-ops, legacy) behind dual PgBouncers, split mode on, with a mix of adapter and quaint clients. Reading the actual emitted OTel metrics, every pool shows up as its own series:
Confirms: metrics are attributed per pool with the correct driver; adapter pools' figures come from
pg.Pool; and query counters/duration histograms keep incrementing under the pg adapter. Also verified/metrics(Prometheus) now returns zeroprisma_*series while still serving the app's own metrics.pnpm run typecheck --filter webapppasses.Notes
/metrics(Prometheus) no longer includesprisma_*series. Anything scraping that endpoint for Prisma metrics should read the equivalentdb.*metrics from the OTel exporter instead.?schema=gotcha (separate from this PR, worth flagging for rollout): since feat(webapp,database): opt-in per-client Prisma driver adapters #4539 parses?schema=from the DSN and passes{ schema }to the adapter, node-postgres sendssearch_pathas a startup parameter. A transaction-mode PgBouncer rejects that withFATAL: unsupported startup parameter: search_path. Our prod control-plane DSNs use the defaultpublicschema with no?schema=param, so this is latent, but any client we flip to the adapter must not carry?schema=in its DSN (or the pooler needsignore_startup_parameters = search_path).