Repository navigation
v2.3.7 benchmark ledger: every quotable claim, with n, range, condition and source #796
Description
Summary
One public page for every number v2.3.7 may quote. Each row holds the claim, its before and after, n and range, the condition and the source comment. Marketing quotes only rows marked Ready.
The rows came from the v2.3.6 ledger (#763) at the v2.3.6 release audit on 2026-11-14, with their children. No row is Ready yet.
How this page works
- Agent in the repo updates a row the day a child posts a
Measuredcomment. It copies the numbers and links the comment. It never computes a new number here. - Maintainer moves a row to Ready, or marks it "No claim".
- At release, the
release-productskill §Benchmarks copies the Ready rows into itsBENCHMARKS.md, audits them and promotes them. - Use follows that skill: Promote, Note, Internal or No claim.
- A row's number comes from one source only. Each M1 (deploy time) number comes from the Measure CI/CD time to live after the v2.3 work: 1090 s median before #569
stats.tsoutput.
Rules for every claim
From .agents/skills/release-product/SKILL.md §Benchmarks and the 2026-11-11 and 2026-11-12 milestone reviews:
- Keep the condition in every sentence that quotes the number ("on one laptop", "simulated mobile", "production").
- No production claim without a production before.
- Never call a pad timeline number "equal to Lighthouse". Never compare a pad timeline number with a Home Lighthouse number.
- Patch y-indexeddb so a closed offline copy cannot stop pad updates, and WebKit can compact it #751 and Make GlitchTip browser counts usable: one Safari IndexedDB event per page load, no service-worker register noise #752 count traces (distinct
trace_id), not page loads, unless a local check proves one trace per page load. - Rows from after the Move production to a new Debian 13 droplet #747 host move form a new sample. Never pool them with earlier rows.
- The 2026-11-11 London collab database move splits samples. So does the Upgrade Supabase Postgres from 15.8.1.073 to 17 #322 upgrade. Never pool rows across either for a metric that touches that database.
- n = 5 never comes from
gh run rerun. - Pad fill time comes from
fillcheck.ts. The Cut the editor stack out of Home's first load, and start the pad's editor chunks sooner #733 throttled fill (t4) is a different number. Never compare the two. - Every save-path claim needs its content-safety checklist passed (Commit one content-safety checklist and one Measured format to the release-product skill #762).
- Credit the release that shipped the change. A gain measured during v2.3.7 on code in an earlier tag says "since v2.3.", never "in v2.3.7" (audit of 2026-11-14).
- Quote exact source numbers. Never round 21.71 s to 21.7 s, or 3,869,403 B to 3.87 MB, in a ledger claim.
Measured, audit due
| Child | Claim (draft) | Before | After | n, range | Condition | Status |
|---|---|---|---|---|---|---|
| #737 | Ops: "On one laptop, with 4 large saves for one pad on one worker, peak RSS fell from 1,255.6 MiB to 882.8 MiB (n = 5). Store jobs at once per worker: 10 → 1." | 1,255.6 MiB (1,140.9-1,514.4), concurrency 10; 5 of 5 runs above 1 GiB | 882.8 MiB (855.9-922.8), concurrency 1; 0 of 5 above 1 GiB | n = 5 + 5 | one laptop (Apple M5 Max, Bun 1.4.4), macOS RSS from ps, 4 saves of 12,518,591 B onto a 12,518,535 B head. The before ran on c9cd2c98a, the after on 3d90636a9 plus #737. Cost: a small save behind the queue waits 1,554 ms (was 5 ms); quote it beside the gain |
Held at the 2026-11-14 audit: a save-path claim needs the #762 content-safety checklist. Production 7-day after due 2026-11-20. before, after |
Pending
<x> marks a number that the child has not posted yet. Status is Pending on every row.
CI/CD
| Child | Claim (draft) | Metric | Before | After |
|---|---|---|---|---|
| #554 | Uses the #569 M1 row on the v2.3.6 ledger (#763) only. | M1 | see #569 | see #569 |
| #557 | Internal: "Silent CI failure paths: 2 → 0." | failure paths with no alert | 2 | |
| #562 | "A README picture change now starts 0 extension test suites, down from 1 (2 for hyperlink media)." | Ext suites per README media change | 1 | , n = 5 |
| #563 | "A docs-only change now passes CI in s, down from 217 s. It runs 0 build or test jobs, down from 3." The basis waits on the #563 ruling (recommended: 278 s median, n = 7 non-deploy pushes, against n = 5 draft-PR runs). | M6 of a docs-only change | 217 s (n = 1); n = 7 median 278 s | 81 s (n = 1); n = 5 pending |
| #566 | On a drop: "Deploys now build the backend image in s, down from 82 s (n = 15 → n = 5)." | backend image build seconds | median 82 s, range 73-113 s, n = 15 | |
| #567 | Only on keep: "Running the two image builds at once cut the image build step from s to s (median of 5 production deploys each)." | image build step seconds | ||
| #649 | No claim. #660 publishes the Bun claims. #654 and #655 closed with an adopt ruling on 2026-11-14; their swaps are #794 and #795. | |||
| #794 | "Type-checking all 15 docs.plus workspaces fell from 5.53 s to s on one laptop (median of 5)." | type-check seconds | 5.53 s, range 5.27-6.74 s, n = 5 (tsc, from #654 Measured) |
|
| #795 | "The docs.plus format check fell from 10.61 s to s on one laptop (median of 5)." | format gate seconds | 10.61 s, range 10.03-11.41 s, n = 5 (Prettier, from #655 Measured) | |
| #660 | Copies each Bun child's claim word for word. | |||
| #756 | Ops: "The host keeps GB free after each deploy, up from a median of 40.5 GB, with the 15 GB build cache kept." | |||
| #759 | Internal: "A skill script edit runs 0 build or test jobs, down from 3." | |||
| #760 | Ops, after the fix: "All five services roll in a median of s, down from 82 s." | |||
| #762 | No claim. It owns the content-safety checklist and the Measured format. |
Observability
| Child | Claim (draft) | Metric | Before | After |
|---|---|---|---|---|
| #753 | "A broken alert file now fails the deploy in under s, before it can switch off alerting." | broken-tree check result and added seconds | Verify passes; Grafana loops | Pre-check passed in production in 11 s; live Verify read 60 rules, 0 in error health (#753 Measured). Broken-tree fail time |
| #712 | "We cut monitoring self-logs from to lines a day." | Loki self-log lines a day | 1,392,680 lines a day (median, range 1,371,977 to 1,413,338, 7 days to 2026-11-10, #712 Measured) | . Caps (7d1h, 1d1h) and 768M limit live; idle 99.7 MiB, was 448-485 MiB (#712 Measured). Drop stages live 2026-11-13 17:38Z. First read, 15 min: the 3 services 586.7 → 0.3 lines a minute (#712 deploy note). The 7-day after is due 2026-11-20 |
| #710 | "Our API error alert now reads only real API traffic. Health probes made 92.8 % of its input before and 0 % now." | probe share of the alert input | 92.8 % (441,735 of 476,161, 7 days to 2026-11-10, #710 Measured) | . Probe-free rules live, 60 rules, 0 in error health (#710 Measured) |
| #324 | No claim. Internal: "Compute size ruled on 7 days of measured host data." | |||
| #642 | No claim. Internal: "The eviction alert names every TTL key family that production holds." | |||
| #455 (v2.4.0) | Internal: "MCP server faults now count as failures, and one request id traces each one across REST and WS." | |||
| #607 | Only with the measured median: "A push-service outage now pages within about minutes." |
Collab and save path
| Child | Claim (draft) | Metric | Before | After |
|---|---|---|---|---|
| #750 | Tooling: "Store and load time now split by phase and pad size." | store-job phase series in production | 0 | store_job_phase_seconds (6 phases x 4 size classes), document_head_bytes and loaded room bytes live since 2026-11-13 13:44Z. Local guard: store job mean 0.1574 → 0.1583 s, n = 5 (after) |
| #726 | "An MCP edit on a 3.6 MB pad: 2.06 s → s held, 3.12 s → s cold (local, n = 5)." Production: " s → s per write against the London before-2." | MCP write seconds | local 2.06 s held, 3.12 s cold; London before-2: edit_blocks p50 3.81 s, p95 4.13 s, max 5.69 s, n = 91, every write cold (#726 Measured) |
|
| #738 | "Opening the largest public demo pad: MB → MB on the wire, time to fill s → s (fillcheck.ts, unthrottled, median, n = 10)"; "stored pad heads: 41.4 MB → MB." The claim waits for the n = 10 fill before. The 24.6 s read is context only: fillcheck.ts, unthrottled, median, n = 5 (#732 comment of 11:20Z). 41.4 MB is the 12:15Z re-read of 2026-11-11. | |||
| #703 | No claim. It is the before for #704 and #706. | |||
| #704 | "Orphan claim-check data per 92-write session: MB → 0 (local, n = 5, --gap-ms 14000, 70 ms delay)"; "failed store attempts per 100 MCP writes: → 0 (production, London)." The 290 MB figure is local n = 3, with no gap and no delay, so it is context. | |||
| #706 | "Version rows per co-edited pad: → at most 1.0 per minute." | |||
| #705 | Ops: "Redis publishes per sync round at 3 replicas: → (measured); truncated saves 0." The model values (9 → 7, from code) are context only. | |||
| #729 | Ops: "A stall over 1 s is now visible on a panel and a rule; paging alerts from stalls: 0." Reword after the #729 for ruling: with for: 5m, the rule sees repeated stalls, not one stall. |
|||
| #602 | Verification: "0 repeated frames in 15 runs per arm (5 per size) on amd64, Bun 1.4.4." | |||
| #739 | "Collab sync on a 4 MB pad: 5,882,274 B → 2,422,805 B; each keystroke 559 B → 30 B." Restate with the post-#738 table. | |||
| #659 | Security: "One connection can add at most B per 60 s; real-editor closes 0 in 7 days." | |||
| #736 | "'Synced' now means the server has the edit: false 'synced' on a slow link 5/5 → 0/5"; "store writes per 100 keystrokes: → ." Step 0 (from #735) has its own row once posted: React commits per keystroke. | |||
| #740 | "Reopening a large pad: download MB → MB, upload MB → KB." | |||
| #754 | Ops: "Peak REST memory with two large exports: MiB → MiB (local, n = 5)." Out-of-memory restarts of 0 are a guard, not a gain. | |||
| #701 | "New pads no longer gain position-migration maps: net growth in heads with an entry, 15 in 3 days (36 → 51, 2026-11-08 to 2026-11-11) → in the 3 days after this deploy (production, read before the #738 backfill)." | |||
| #758 | Internal: "1 save confirmation per save, down from 2 at 2 replicas." | |||
| #761 | No claim. |
Data and import
| Child | Claim (draft) | Metric | Before | After |
|---|---|---|---|---|
| #322 | "We moved the docs.plus Supabase database from Postgres 15.8 to 17 with minutes of planned downtime. 0 chat messages were lost." | |||
| #325 | If the window is clean: "Over 7 days, no docs.plus chat Realtime poll took 1 s or more. The slowest took ms, against 14.6 s before." | |||
| #577 | "A 306 KiB Word file that unpacks to 40 MiB used to add MiB (median of 5, local bench on ). Now the server refuses it at the zip walk, and no admitted file adds more than MiB." | added import RSS, n = 5 | until the #577 step 0 Measured comment. Context only: 316 MiB at 10 MiB unpacked, 818 MiB at 40 MiB (#724 bench, one laptop, n = 1) | |
| #677 | "One crafted table cell used to add 398 MB to a docs.plus Word export. Now it adds MB." |
Web speed
| Child | Claim (draft) | Metric | Before | After |
|---|---|---|---|---|
| #571 | Tracker. Its ledger rows link here. | |||
| #733 | "Home's JavaScript fell from 3.87 MB to MB across v2.3.6, and pad first sync stayed within % of before (median, n = 10, production /demo, simulated slow link)." |
|||
| #734 | "Opening the demo pad downloads MB less before you scroll (n = 5)." | |||
| #751 | "Safari's offline copy stays compact: rows to at most 2 after 5 reloads." | |||
| #752 | Internal: "Flood events per trace: 252 → 1." Ledger source: local console check 1 (2 flood rejections → 1 request). If the production 7-day read sees 0 flood traces, mark it "not observed" and quote nothing from it. | flood events per trace | 252 (1,765 events in 7 traces) | |
| #731 | v2.3.6 posts delivery counts only. Field p75: moved to #764 (v2.4.0). | |||
| #641 | No claim. Security fix, verification line: "/_next/image fetches 0 of 3 remote probe hosts (before: 2 of 2)." |
Security and host
| Child | Claim (draft) |
|---|---|
| #552 | "Only main deploys reach the production runner. Open settings 4 → 0, with s added per deploy." |
| #725 | "Dead stage pipeline removed: 1 workflow, 1 orphan GitHub environment and its secret, and stage lines in agent docs 17 → 0 and 18 → 0." |
| #730 | "Both browser suites are green on main for 10 runs in a row: chatroom s, editor s median." |
| #741 | "docs.plus sends Google no pad address or title: leaking page views per Home-to-pad move → 0, and Home loads KB less third-party script." is the Step 1 5-trial before. The n = 1 live test is not the before. |
| #742 | "World-readable production env files 14 → 0, and new copies stay owner-only." |
| #743 | "After a reboot, the deploy runner is back on its own in s, and jobs no longer see app secrets: keys → 0." |
| #744 | "Production SSH is keys-only. guessing attempts a day now end in a ban." |
| #746 (with #713 folded in) | "Edge on Traefik 3.7.18: matching advisories 6 → 0, with s of edge outage. Failed ACME requests for www.new.docs.plus: about 10 a day → 0. Webapp p95 stayed within % (7 days each side)." |
| #747 | "Production moved to Debian 13 with s of editing pause and 0 lost edits." |
| #748 | "Every production store is backed up daily or weekly, encrypted, to a second region. A full restore drill took minutes, with a data-loss window of hours." |
| #708 | "Production runs the newest installed kernel, with s of downtime, and every service came back with no manual start." |
| #757 | "docs.plus production costs % less a month ( to )." Percent only, never USD. Quote it per ruled-saving line, never as a pooled total. |
Close rule
This issue closes when v2.3.7 ships. By then each row is Ready, No claim, or moved with its child to the next milestone.
Related
#763 (the v2.3.6 ledger, where these rows came from), #660 (Bun report, one source row), #569 (M1 source), #554 and #649 (formats), #762 (content-safety checklist and Measured format), #794 and #795 (the bun swaps), and every child listed above.
Activity