[joseph-isaacs]

Summary

Adds vortex.approx_distinct_count, a footer-only file statistic in the preview2026.08.0 edition. DataFusion and DuckDB both receive it as a column distinct count for planning.

The partial has two modes:

• Exact: the set of distinct non-null values seen. Two exact partials merge by union.
• Inexact: only a count. If either side of a merge is inexact, the two counts are added, so an inexact count is an upper bound.

An exact set becomes inexact once it holds more than 64Ki values, which caps memory per column. The serialized partial is only the count: the set is dropped on write, so a reader always gets an inexact count.

Changes

• vortex-array: adds the ApproxDistinctCount aggregate and registers it.
  • Supports bool, primitive, decimal (precision ≤ 38), UTF-8, binary, and extension types over those.
  • Fixed-width values are keyed as u128. Variable-length values go in an arena-backed byte set, so inserting a value allocates nothing per value.
• vortex-layout:
  • Adds it to the default file statistics, but not to zone maps.
  • When nested statistics are not written, no longer computes default aggregates that have no slot in the legacy statistics layout. That work was thrown away before.
• vortex-edition: adds it to preview2026.08.0. The edition record is regenerated and the editions doc updated.
• vortex-datafusion: the VortexFormat and v2 sources fall back to it as an inexact distinct_count.
• vortex-duckdb: passes it as distinct_count. For numeric columns it takes the smaller of this value and the min/max span estimate. Strings now get distinct counts too.
• vortex-bench: enables the preview edition and writes nested file statistics, so benchmark files carry the stat.

The stat is only written with include_nested_statistics(), because the legacy statistics layout has no slot for it.

Inputs that arrive as chunked arrays come out inexact. The generic chunked aggregate kernel passes partials through their scalar form, which keeps only the count.

Tests:

• Unit tests for counting, merging, scalar round trips and the inexact switchover.
• File-stats tests for exact merging across chunks and for skipping the stat in legacy-only footers.
• A vortex-file round trip that covers the edition gating.
• A DataFusion test for the distinct-count fallback.

Local measurements (SF=1, 4 cores)

The arm without the stat was produced by a temporary local toggle that is not part of this PR.

Ingest (parquet → vortex including compression, median of 3 runs): TPC-H takes 5.86 s → 6.69 s (+14%), and TPC-DS 9.45 s → 10.73 s (+13%).

Queries (vortex format, 5 iterations × 2 alternating rounds; ratios are with-stat over without, below 1 is faster):

	total	geomean ratio
TPC-H DataFusion	−0.6%	0.952
TPC-H DuckDB	−5.0%	0.952
TPC-DS DataFusion	+1.9%	1.018
TPC-DS DuckDB	+3.6%	1.026

Between rounds of the same arm, the geomean varies by up to ±5%. Plans change in both engines. TPC-DS q19 on DuckDB is about 2.7× slower in every run: DuckDB's row estimate drops sharply and it picks a worse join order, even though the stored counts are mostly exact.

---

Generated by Claude Code