diskbench: bypass the page cache on Linux, and say so when it cannot

[marcobambini]

Extends #22, with that fix as the base and its author as co-author.

tools/diskbench.c documented itself as measuring cache-bypassed reads and on Linux did neither: nocache() had an #ifdef __APPLE__ body and nothing else in it, and O_DIRECT appeared nowhere in the file. Reported by @fab2s on a Samsung 970 PRO, Gen3 x4 — 44.67 GB/s sequential and 65.72 GB/s random against a 3.94 GB/s link, 11x and 17x the ceiling; 3.15 and 3.33 GB/s with the flag, saturating at two threads.

This is LEARNED §14 in the one place §14 did not reach. The engine's bypass was written blind and fixed there on 2026-07-28; the tool that exists to characterise the engine's I/O kept measuring RAM, so §46's standing rule — before claiming anything is disk-bound, run diskbench and divide — returned a fiction on Linux for that whole window. No published number is affected: every diskbench figure in docs/GATES.md, docs/EFFICIENCY.md and LEARNED §44/§46 was measured on macOS, where F_NOCACHE did work.

On top of #22

The flag alone is not sufficient, for the reason bank_open already knows: O_DIRECT is accepted at open and refused at transfer — tmpfs does this, and so would a device wanting a bigger block than the tool aligns to. A bare flag turns such a filesystem into short read -1 and a table of zeroes with no cause given. So it now follows bank_open: probe with one aligned transfer, fall back to a plain open plus POSIX_FADV_RANDOM, and label every row.

seq write   :  10.45 GB/s  (PAGE CACHE, not the disk)
seq read    :  34.17 GB/s  (PAGE CACHE, not the disk)
rand  1 thr :  32.48 GB/s  (PAGE CACHE, not the disk)

the page cache could not be bypassed on this filesystem, so the rows
above are partly the kernel serving RAM back and are not the disk. The
engine falls back the same way and reports it as direct_io=0; move the
working file to the filesystem the container will live on.

Bypassing the write is load-bearing, not symmetry. I had drafted a review asking to leave it buffered — row 1 stands for the conversion landing, and an O_DIRECT read invalidates the range anyway. That holds on Linux and is wrong on macOS: F_NOCACHE stops new pages being cached but does not evict resident ones, so a buffered write leaves the file in the UBC and every read row below reports RAM. Same binary, 1 GB file, 4 MB records, differing only in the write fd's bypass:

	write bypassed	write buffered
seq read	7.9–8.1 GB/s	26.04 GB/s
random, 1 thread	6.8–7.0	24.34

The 12.5 GB/token column, also reported in #22, is fixed here rather than left for later. It was K3's figure hardcoded into the format string, so every run answered for K3 whatever container was being sized — ~8x off on a 48B model at 1.61 GB/token — and said so nowhere. It is now the fifth positional argument with no default: without it the column is not printed at all. Defaulting to 12.5 would have kept the trap, and the tool cannot derive bytes-per-token from a scratch file — that number belongs to a container, and waste bench already reports it as disk N GB total, M GB/token. The usage line points there.

rand  2 thr :   8.48 GB/s  (cache bypassed)
rand  2 thr :   8.48 GB/s  ->  0.68 tok/s at 12.5 GB/token cold  (cache bypassed)
rand  2 thr :   8.48 GB/s  ->  5.27 tok/s at 1.61 GB/token cold  (cache bypassed)

docs/GATES.md's Gate H table keeps its tok/s @12.5 GB/token header: that was a K3 decision, the assumption is stated in the header rather than buried in a format string, and the numbers under it were measured with it.

Also two silent-wrong-number paths in passing: a sub-page record rounded to zero and then divided by it, and a failed sequential read just ended the loop and shortened the row.

Verification, and its limit

macOS is unchanged against main within noise — 7.85 vs 8.01 GB/s sequential, 6.84 vs 6.74 random 1-thread, 1 GB file on the internal SSD.

The Linux body compiles and runs here only against stubs for O_DIRECT and posix_fadvise, which covers both the probe-succeeds and probe-refused paths and confirms the write probe restores the file byte for byte (1 GiB written, 1 GiB read back), but is not the kernel's real O_DIRECT behaviour. This is the same limitation §14 recorded for the engine, for the same reason: there is no Linux on this machine. @fab2s's 3.15 / 3.33 GB/s remain the only figures here from actual Linux hardware.

Not in this PR

CI's SPDX glob covers tools/*.py and tools/*.sh, not tools/*.c — which is why diskbench.c has no header while lutbw.c and lutmt.c do. Widening the glob is the actual fix; adding the header alone would drift again.

🤖 Generated with Claude Code