tools/diskbench.c reports page-cache throughput on Linux, not disk - #22
tools/diskbench.c reports page-cache throughput on Linux, not disk#22fab2s wants to merge 1 commit into
Conversation
|
Confirmed, and thank you — the diagnosis is exact. For the record on how it got there: this is LEARNED §14 in the one place §14 did not reach. The engine's Linux bypass was written blind, found and fixed on 2026-07-28 ( I have pushed an extended version on 1. Probe and fallback. The flag alone is not sufficient, for the reason 2. Your write change was right, and I nearly talked myself out of it. I had drafted a comment asking you to leave the write buffered: row 1 stands for the download and the conversion landing, those write through the page cache like everything else, and a subsequent Verification and its limit: macOS is unchanged against On the hardcoded |
…rint The derived column read `-> %.2f tok/s at 12.5 GB/token cold` with 12.5 in the format string. That is K3's figure, so every run answered for K3 whatever container was being sized — ~8x off on a 48B model at 1.61 GB/token measured, and silent about the assumption, which is the part that makes it a trap rather than an approximation. It is now the fifth positional argument and has no default: without it the column is not printed at all. Defaulting to 12.5 would have kept the trap, and a tool cannot derive bytes-per-token from a scratch file — that number belongs to a container, and `waste bench` already reports it as "disk N GB total, M GB/token". The usage line says so. docs/GATES.md's Gate H table keeps its "tok/s @12.5 GB/token" header: it was a K3 decision, the figure is stated in the header rather than hidden in a format string, and the numbers under it were measured with it. Reported by fab2s alongside the O_DIRECT bug in #22. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Superseded by #25, merged as ffb7ebc — your O_DIRECT fix is its base commit, with you as co-author, plus the probe-and-fallback for filesystems that refuse the flag, per-row labelling of whether the bypass was obtained, and the GB/token column you flagged (now an argument with no default, so it cannot silently answer for K3 again). Thank you for the report. The numbers from real Linux hardware are the part this repo cannot produce on its own — the Linux path is still only stub-verified here, as LEARNED §14 records for the engine itself, so if you ever run the merged version on that 970 PRO the result is worth an issue. |
|
Both fixes are in v0.6.6, as The bypass. The flag alone was not enough, for the reason One thing your patch had that I nearly dropped, and was wrong to doubt: bypassing the write. The argument for leaving it buffered is that it models the conversion landing, and an Two more found while in there: a sub-page record silently rounded to zero and was divided by, and a failed sequential read just ended the loop and shortened the row rather than saying anything. The tok/s column. Fixed as you flagged — 12.5 GB/token was K3's figure sitting in a format string, so every run answered for K3 whatever was being sized, ~8x off on a 48B model at a measured 1.61 GB/token. It is now the fifth positional argument with no default: without it the column is simply not printed. Defaulting to anything would have kept the trap, and a tool cannot derive bytes-per-token from a scratch file — that number belongs to a container, and Scope, for anyone finding this later. No published figure moves: every Verified here on macOS as unchanged within noise (7.85 vs 8.01 GB/s sequential, 6.84 vs 6.74 random 1-thread). The Linux path compiles and runs here only against stubs, which covers probe-succeeds and probe-refused and confirms the write probe restores the file byte for byte — but not the kernel's actual Thank you — this is the best kind of bug report: the tool that measures the thing was the thing that was wrong. |
diskbench v0.6.6 on real Linux hardware: two drives verified, tmpfs no longer refusesRan the merged tool on this machine. The bypass is real on both drivesSamsung 9100 PRO 2TB, PCIe Gen5 x4 (link ceiling 15.75 GB/s): Samsung 970 PRO 512GB, PCIe Gen3 x4 (link ceiling 3.94 GB/s): Every row sits under its link ceiling, which is the check the old code could not The The fallback path could not be exercised, and tmpfs is why
So the probe succeeds and the run is labelled bypassed: Nothing else here refuses either — ext4, tmpfs at One gap this exposesThat tmpfs table is 45.57 GB/s labelled That is a narrower version of the original bug rather than a return of it: the What is still unverifiedThe probe-refused branch. If you know a filesystem that still refuses the flag on |
`diskbench` documented itself as measuring cache-bypassed reads (F_NOCACHE / O_DIRECT) and on Linux did neither: `nocache()` had an `#ifdef __APPLE__` body and nothing else in it, and O_DIRECT appeared nowhere in the file. Reported against a Samsung 970 PRO on Gen3 x4 — 44.67 GB/s sequential and 65.72 GB/s random against a 3.94 GB/s link, 11x and 17x the ceiling. With the flag, 3.15 and 3.33 GB/s, saturating at two threads, which is what that drive should do. This is LEARNED §14 in the one place §14 did not reach. The engine's bypass was written blind on 2026-07-28 and fixed there; the tool that exists to characterise the engine's I/O kept measuring RAM. §46's standing rule — "before claiming anything is disk-bound, run diskbench and divide" — therefore returned a fiction on Linux for that whole window. No number in the docs is affected: every diskbench figure in GATES.md, EFFICIENCY.md and LEARNED §44/§46 was measured on macOS, where F_NOCACHE did work. The flag alone is not enough, for the reason bank_open already knows: O_DIRECT is accepted at open and refused at transfer (tmpfs does this, so would a device wanting a bigger block than we align to), so a bare flag turns a refusing filesystem into "short read -1" and a table of zeroes with no cause given. So this follows bank_open: probe with one aligned transfer, fall back to a plain open plus POSIX_FADV_RANDOM, and label every row — a bench that quietly measures something else is worse than one that says it could not. The write is bypassed too, and that is not symmetry for its own sake. Leaving it buffered — on the argument that it models the conversion landing, and that an O_DIRECT read invalidates the range anyway — was measured wrong on macOS: F_NOCACHE stops new pages being cached but does not evict resident ones, so a buffered write leaves the file in the UBC and every read row below reports RAM. 1 GB file, M5 Pro: 8.07 GB/s sequential read with the write bypassed, 26.04 GB/s with it buffered. The original `nocache()` on the write fd was load-bearing. Also: a sub-page record silently rounded to zero and divided by it, and a failed sequential read just ended the loop and shortened the row. Verified on macOS — unchanged against main within noise (7.85 vs 8.01 GB/s sequential, 6.84 vs 6.74 random 1-thread). The Linux body compiles and runs here only against stubs for O_DIRECT and posix_fadvise, which covers both the probe-succeeds and probe-refused paths and confirms the write probe restores the file byte for byte, but not the kernel's actual O_DIRECT behaviour. The 3.15/3.33 GB/s figures above are the reporter's, on hardware this machine does not have. Reported and diagnosed by fab2s in sqliteai#22, with the O_DIRECT fix as written there. Co-Authored-By: fab2s <fabrice.de.stefanis@gmail.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
tools/diskbench.creports page-cache throughput on Linux, not diskdiskbenchdocuments itself as measuring "random record reads, cache-bypassed<- the number that sets tok/s", with the page cache bypassed via
"F_NOCACHE / O_DIRECT". On Linux it does neither:
O_DIRECTnever appears inthe file, and
nocache()is a no-op outside macOS.Reported numbers therefore exceed the drive's physical link limit.
Reproduction
Samsung 970 PRO, PCIe Gen3 x4 (link ceiling 3.94 GB/s), Ubuntu 26.04,
kernel 7.0, 16 GB file, 3 MB records:
44.67 and 65.72 GB/s are 11× and 17× the link ceiling.
Cause
and all three opens are unqualified:
The engine handles this correctly —
src/model.c:1355notes "macOS says so withfcntl. Linux needs O_DIRECT and Windows FILE_FLAG_NO_BUFFERING", guarded by
#if defined(_WIN32) || (defined(__linux__) && defined(O_DIRECT)). The tool isout of step with the code it exists to characterise.
Fix
Adding the flag is sufficient: the buffers are already
posix_memalign'd to4096 and the record size is a 4096 multiple, so
O_DIRECT's alignmentrequirements are already satisfied.
Verification
Same drive, same invocation, after the patch:
3.33 GB/s is 85% of the link ceiling and saturates at 2 threads, which is what a
Gen3 x4 drive should do.
Unrelated, same file
The derived column prints
-> X tok/s at 12.5 GB/token cold, where 12.5 GB/tokenis K3's figure hardcoded. On a 48B model (1.61 GB/token measured) that column is
off by ~8×. Taking the per-token figure from the container's
plan --json, ordropping the column, would avoid the trap.