README.md

Benchmarks

A workload catalogue + harness for measuring cljw against other runtimes. Each benchmarks/NN_<name>/ directory holds a meta.yaml (name / category / expected_output / description) and one source file per language (bench.clj, bench.c, bench.zig, Bench.java, bench.py, bench.rb, bench.js, bench.go). expected_output doubles as a correctness oracle — a runtime that prints the wrong value is shown as a SKIP, never timed.

Every table and chart below is generated from a measurement YAML, never hand-maintained — a hand-curated table drifts from the data (the lesson from cw's predecessor). A run that is only printed to a terminal is scrollback, not data: it cannot be diffed against the next one, so each harness writes a --yaml datum first and the prose is rendered from it.

How this directory is layered

Three strata, and nothing reaches across one:

Stratum	Files	May do
boundary	bench/*.sh, bench/*.clj	spawn runtimes, read the clock, write the datum; a cljw-only measurement is a cljw program
datum	bench/*.yaml	nothing — inert, diffable, committed
calculation	bench/*.py	datum → Markdown, datum → SVG

bench_domain.py is the calculation stratum's model, and the only place that knows what a runtime is called, which order the columns go in, which workloads are cross-language-comparable, and that milliseconds are not microseconds. The renderers depend on the model rather than on a YAML shape, so gen_cross_table.py (Markdown) and gen_charts.py (SVG) can never disagree about what was measured, and a harness changing its output format touches one adapter.

A Suite — one machine, one toolchain, one date — is the unit of comparability. Numbers from two Suites never share a table or a chart; a run on a different box is a new Suite, not extra rows.

Running

# cljw-only: per-bench millisecond table (builds ReleaseSafe, uses hyperfine)
bash bench/run_bench.sh                 # all native workloads
bash bench/run_bench.sh --bench=sieve   # one workload
bash bench/run_bench.sh --no-wasm       # skip the -Dwasm FFI workloads

# Cross-language comparison → measurement YAML
bash bench/compare_langs.sh --yaml=bench/cross-lang-latest.yaml

# Wasm FFI: cljw's embedded zwasm against wasmtime, same module, same loop count
bash bench/wasm_bench.sh --yaml=bench/wasm-ffi-latest.yaml
bash bench/wasm_bench.sh --engine=interp   # the fallback tier (default: auto)
bash bench/wasm_jit_vs_interp.sh           # one module, both tiers, ratio
./zig-out/bin/cljw -cp bench -m wasm-percall --yaml=bench/wasm-percall-latest.yaml
                                        # one wasm/call crossing in ns, per engine and thread (a cljw program)

# Render from a datum — either harness's, the shape is detected
yq -o=json bench/cross-lang-latest.yaml | python3 bench/gen_cross_table.py
yq -o=json bench/wasm-ffi-latest.yaml   | python3 bench/gen_charts.py --out docs/assets/bench

# Occasional-use tools (on demand)
bash bench/build_bench.sh               # `cljw build` artifact: build time / size / startup
clojure -M bench/clj_warm_bench.clj bench/benchmarks/01_fib_recursive/bench.clj
                                        # warm-JVM comparison for one workload
bash bench/release_metrics.sh           # binary-size + startup metrics (RELEASE_METRICS.md)

The cross-language harness compiles each language on the fly and auto-skips any toolchain that is absent (command -v guard); run it from inside nix develop for the pinned, reproducible toolchains. cljw is built ReleaseSafe (the shipped build); timing is hyperfine, reported as cold-start wall-clock (process launch → exit, startup included).

Cross-language results

Cold-start wall-clock (process launch → exit), µs, lower is better. Columns: ClojureWasm, then Python / Ruby / Node.js / Babashka, then Java / Go / C. Only cold-start is published: it is the metric that compares uniformly across languages. A startup-subtracted "compute" table is intentionally omitted — for the fast languages the per-run compute sits below process-spawn noise (~3 ms ± 10%), so subtracting startup would report noise, not signal.

Conditions: MacBook Pro (Mac16,8), Apple M4 Pro, 12-core (8P+4E), 48 GB RAM, macOS 26.5.2 (25F84), hyperfine 5 warmup + 10 runs, 2026-07-16.

Cold-start = process launch → exit (startup included). Only cold-start is shown: it is the metric that compares uniformly across languages. A startup-subtracted compute number is omitted because, for the fast languages, compute sits below process-spawn noise.

Cold-start wall-clock (µs, lower is better)

Benchmark	ClojureWasm	Python	Ruby	Node.js	Babashka	Java	Go	C
fib_recursive	23592	24084	40152	51683	25530	28891	5132	2306
fib_loop	7296	19860	34568	48962	14062	28046	3589	2288
tak	11508	21862	34328	50946	16398	26646	2498	1308
arith_loop	38648	59600	56904	57668	52308	29097	7687	1540
map_filter_reduce	14597	11830	34047	53874	17153	31237	1525	2319
vector_ops	6246	17910	34781	50970	17331	31080	3932	3342
map_ops	8317	19489	35475	48525	13764	30530	5878	0
list_build	6771	25464	35397	48022	14947	25389	7103	2862
sieve	21020	20145	34407	50930	9766	32725	1693	2424
nqueens	20509	24901	36540	50522	23139	28398	598	2789
atom_swap	9846	19230	39096	54700	18411	30278	1015	1166
gc_stress	26898	30161	41916	52238	37935	41715	5858	5944
lazy_chain	14140	19144	37334	52050	18639	32501	713	666
transduce	13061	26357	33400	47270	9386	31750	0	1893
keyword_lookup	14288	25774	36582	48963	20833	32758	8510	576
protocol_dispatch	9651	18004	31117	49750	-	28951	7577	331
nested_update	14634	18474	34204	57072	18991	30037	2022	1990
string_ops	27776	33485	40155	51049	19171	33796	2040	6024
multimethod_dispatch	9896	21865	34332	50649	17072	36672	1740	2992
real_workload	20756	21397	38026	49682	17760	34616	3390	1149
gc_alloc_rate	42149	-	-	55576	38063	-	-	-
gc_large_heap	39475	-	-	54312	30295	-	-	-
bigint_factorial	22604	25381	43982	78638	22139	37042	2149	-
ratio_sum	25328	-	-	-	35542	-	-	-
stm_refs	38278	-	-	-	45086	-	-	-
regex_count	15613	25307	40024	51574	16090	36368	5440	12225
sort	6472	19568	35478	49777	13070	31522	2133	2182
destructure	53008	-	-	-	41037	-	-	-
edn_roundtrip	27514	-	-	-	103409	-	-	-
json_parse	35479	31780	44924	55282	-	-	-	-

Warm, startup subtracted (µs, lower is better)

Benchmark	ClojureWasm	Python	Ruby	Node.js	Babashka	Java	Go	C
fib_recursive	13619	5259	6404	5316	11947	915	1916	0
fib_loop	0	1034	819	2595	479	69	373	0
tak	1535	3036	579	4580	2815	0	0	0
arith_loop	28674	40774	23156	11301	38724	1120	4471	0
map_filter_reduce	4624	0	299	7507	3569	3260	0	0
vector_ops	0	0	1032	4604	3748	3104	716	92
map_ops	0	664	1726	2158	180	2553	2662	0
list_build	0	6639	1648	1656	1364	0	3887	0
sieve	11047	1320	658	4564	0	4749	0	0
nqueens	10536	6076	2792	4156	9556	421	0	0
atom_swap	0	405	5347	8333	4827	2301	0	0
gc_stress	16924	11336	8167	5871	24351	13738	2642	2694
lazy_chain	4167	318	3586	5684	5055	4524	0	0
transduce	3088	7531	0	904	0	3774	0	0
keyword_lookup	4315	6949	2834	2597	7249	4781	5294	0
protocol_dispatch	0	0	0	3384	-	974	4361	0
nested_update	4660	0	455	10705	5408	2060	0	0
string_ops	17803	14660	6407	4683	5588	5820	0	2774
multimethod_dispatch	0	3040	584	4282	3489	8696	0	0
real_workload	10782	2572	4278	3315	4176	6639	174	0
gc_alloc_rate	32176	-	-	9210	24480	-	-	-
gc_large_heap	29501	-	-	7946	16711	-	-	-
bigint_factorial	12630	6556	10233	32272	8556	9066	0	-
ratio_sum	15354	-	-	-	21958	-	-	-
stm_refs	28305	-	-	-	31502	-	-	-
regex_count	5639	6481	6275	5208	2506	8391	2223	8975
sort	0	743	1729	3410	0	3545	0	0
destructure	43035	-	-	-	27454	-	-	-
edn_roundtrip	17540	-	-	-	89826	-	-	-
json_parse	25506	12955	11175	8915	-	-	-	-

Startup, process spawn + runtime init (µs, lower is better)

Startup (µs)	ClojureWasm	Python	Ruby	Node.js	Babashka	Java	Go	C
process spawn + init	9973	18825	33749	46366	13584	27976	3216	3249

Warm = cold − startup; digits below startup-measurement noise are indicative.

About these numbers

• A few ClojureWasm facts (no comparison intended): cold-start folds in loading the embedded AOT-compiled clojure.core (ADR-0056), so the table is the real time-to-first-eval — for cljw (a tree-walking + bytecode-VM interpreter) that is mostly genuine work; for the compiled languages a small-workload run is mostly process spawn. cljw is built ReleaseSafe (optimised, all safety checks on).
• Reproducibility. Toolchains are pinned in flake.nix; run from inside nix develop for the full, reproducible column set. The harness auto-skips a language whose toolchain is absent.

Wasm FFI — cljw's embedded zwasm against wasmtime

This is the measurement behind the north star (gap II, Wasm-edge-native): when a Clojure program reaches through wasm/call into a module, what does that execution cost against a dedicated wasm runtime?

Both sides run the same TinyGo-compiled .wasm export with the same in-module loop count, so the iteration happens inside wasm on both sides and neither host's own interpreter is in the loop. That makes it wasm-execution against wasm-execution rather than language against language.

(wasm/load path opts) defaults to zwasm's .auto engine — JIT-first with interpreter fallback (D-488) — so the default column is JIT against JIT. --engine=interp measures the fallback tier instead.

Conditions: Linux 7.0.0-30-generic x86_64, Intel(R) Core(TM) Ultra 9 185H, ClojureWasm v1.14.0 (ReleaseSafe, wasm), wasmtime 48.0.0 (f1412a598 2026-08-20), :engine auto, hyperfine 2 warmup + 5 runs, 2026-09-04.

Both runtimes execute the SAME .wasm export with the same in-module loop count, so this is wasm-execution against wasm-execution, not language against language. warm subtracts each runtime's own module-load + startup; at the sub-10 ms end that subtraction is larger than the signal, so read only the workloads that run for hundreds of ms.

Total wall-clock, module load + execution (ms, lower is better)

Benchmark	ClojureWasm	wasmtime
fib	1333.6	501.5
tak	4301.0	1619.4
arith	46.3	10.7
sieve	95.1	26.1
fib_loop	69.4	18.1
gcd	185.4	110.8

Execution, load and startup subtracted (ms, lower is better)

Benchmark	ClojureWasm	wasmtime
fib	1286.4	488.5
tak	4253.8	1606.4
arith	0.1	0.1
sieve	47.9	13.1
fib_loop	22.2	5.1
gcd	138.2	97.8

Startup, process spawn + runtime init (ms, lower is better)

Startup (ms)	ClojureWasm	wasmtime
process spawn + init	47.2	13.0

Warm = cold − startup; digits below startup-measurement noise are indicative.

Per-crossing cost, one wasm/call in nanoseconds

The FFI tables above cannot see a per-call cost: every workload loops inside the module and crosses the boundary once per process. This is the other axis, measured by bench/wasm_percall.clj, a cljw program (the loop on the cljw side, only the calls timed, median of trials with the dispersion recorded). The 2026-09-05 finding that the default engine cost ~16.5 us per call on Linux (zwasm/D-584, the initial-thread stack query) is the kind of number this table exists to show; ADR-0195 removed it and ADR-0196 documents what remains.

Conditions: Linux amd64, ClojureWasm v1.14.1 (wasm), bench/wasm_percall.clj 1 warmup + 7 runs, 2026-09-06.

The loop is on the cljw side and only the calls are timed: one module instantiated once per engine, then trials of N calls to a trivial add export. This is the axis the FFI table above cannot see (its workloads loop inside the module). fn-call is a plain Clojure function call in the same process, the platform constant a crossing is measured against. -worker rows run on a future thread. Engines have opposite preferences: compute-bound bodies favour the JIT (~15x, see bench/wasm_jit_vs_interp.sh), crossing-heavy calls favour the interpreter.

Measurement conditions: median run-to-run dispersion 24%, so ratios under 1.68x are not distinguishable from noise.

One wasm/call crossing, median of trials (ns, lower is better)

Benchmark	ClojureWasm
auto-main	1222
auto-worker	999
jit-main	1193
jit-worker	937
interp-main	355
interp-worker	480
fn-call	140

Reading the FFI numbers

• The gap is 1.4-4.4×, not an order of magnitude. wasmtime is a dedicated, mature wasm JIT; zwasm is a from-scratch engine embedded in a Clojure runtime whose binary is ~9 MB in total. Landing within a small constant of wasmtime on compute-bound modules is the result that matters for gap II — it says a program can reach into wasm from Clojure without paying an execution tax that changes what is feasible.
• arith is below the floor and carries no signal. Its total (46.3 ms) is under cljw's own startup (47.2 ms), so the subtraction clamps to 0.1 and the "1.0×" is arithmetic, not a measurement. The chart shows it for completeness; read fib, tak, sieve, fib_loop and gcd.
• The engine tier is worth an order of magnitude, and it is on by default. bench/wasm_jit_vs_interp.sh runs one 20M-iteration i32 loop through both tiers of the same embedding: 0.157 s on :jit against 2.430 s on :interp — 15.5×, end-to-end including startup, on this box. That ratio is why the .interp → .auto default flip (D-488) was the load-bearing change and not a tuning knob.
• Startup is cljw's, not wasm's. The 47 ms cljw pays before the first wasm/call is a Clojure runtime booting its embedded AOT clojure.core (ADR-0056), against wasmtime's 13 ms of loading only the module. A long-lived process pays it once; the warm table is the honest per-call view.