feat(format): define hidden row lineage columns

[BubbleCal]

First of a stack for §5.3 of the Stable Row ID GA design (#8931, tracked as #9250): let a fragment's row lineage sequences -- its row ids and its created-at and last-updated-at versions -- leave the manifest and live as hidden uint64 columns of a Lance data file. This PR defines the format; the stack continues with readers and the spill primitives, then compaction, the commit read-ahead, the update path, and compaction writing the columns into its own output files.

Builds on the prototype in #8953 (Will is co-author).

Stack

1. this PR feat(format): define hidden row lineage columns
2. feat(dataset): read and write spilled row lineage columns #9336 feat(dataset): read and write spilled row lineage columns
3. feat(dataset): spill row lineage at compaction #9337 feat(dataset): spill row lineage at compaction
4. feat(table): load spilled row lineage ahead of a commit #9338 feat(table): load spilled row lineage ahead of a commit
5. feat(dataset): spill row lineage when updating rows #9339 feat(dataset): spill row lineage when updating rows
6. feat(dataset): write compaction's spilled lineage into the fragment's data file #9347 feat(dataset): write compaction's spilled lineage into the fragment's data file
7. test(bench): measure row lineage placement inline versus spilled #9340 test(bench): row lineage spill benchmark

Problem

Each sequence is stored inline in the fragment's manifest entry. An appended fragment's sequences are single runs and cost a few dozen bytes, but once compaction merges fragments whose rows came from many places the row id sequence degrades to 4-8 bytes per row and the version sequences to a run per row. The manifest then grows with the table's row count and every commit rewrites all of it (#8621).

Format

• Every negative field id is reserved for system use and never names a schema field; readers skip any negative id in a data file's fields. -1 and -2 keep their meaning; -3, -4, -5 are the hidden _rowid, _row_created_at_version and _row_last_updated_at_version columns.
• DataFragment gains an empty RowLineageColumn marker arm on each lineage oneof: column_row_ids = 12, column_last_updated_at_versions = 13, column_created_at_versions = 14. The marker carries no file reference: the column lives in one of the fragment's files, the single entry whose fields carry the reserved id, and its column_indices locates it like a user column. Zero or several carriers is corruption. The three sequences may share one file with each other or with the fragment's user data.
• The columns have an executable schema: three non-nullable uint64 fields, each holding exactly physical_rows values in physical row order. A null or a length mismatch is corruption and is rejected, never defaulted.
• The marker is valid only in a fragment whose data files are Lance v2 files: a legacy v1 DataFile has no column_indices to locate the column, and a fragment cannot mix v1 and v2 files. A writer on a v1 dataset leaves every sequence inline.
• New feature flag FLAG_UNSTABLE_SPILLED_ROW_LINEAGE = 1 << 11 (bit 9 is reserved, bit 10 is the tagged FRI flag from feat(format): define a unified tagged fragment reuse history #9136), above FLAG_UNKNOWN the way that flag is. Every released build (v11.0.0 boundary 256, v12/v13 pre-releases 512) already refuses such a dataset. Like data overlay files, release builds understand the bit only with LANCE_ENABLE_UNSTABLE_SPILLED_ROW_LINEAGE=1; debug builds always do.
• The external_* arms are untouched.

Placement rule, which the writers in later PRs follow: a value the commit assigns -- an appended fragment's row ids, an inserted row's created-at, every row's last-updated-at -- can change when a commit conflict is retried, so it stays inline where the retry can rewrite it. A value carried over from existing rows is fixed before the commit and may go to a data file.

Code

• RowIdMeta::Column and RowDatasetVersionMeta::Column unit variants, with proto and JSON round trips and manifest interning. Fragment::row_lineage_file(field_id) finds the carrier among files and reports more than one as corruption. Because the carrier is an ordinary entry of files, cleanup, shallow-clone base_id rewriting, file listing and validation already cover it; validation skips the negative ids in a file's fields.
• apply_feature_flags sets the flag when any fragment uses a column arm.
• Nothing writes the columns yet. Reading one -- in the row id loader, the fragment reader, and the commit-time paths that resolve an update's lineage from existing fragments -- returns NotSupported instead of falling back to defaults. RowDatasetVersionMeta::load_sequence does the same for the external arm instead of todo!().
• assign_row_ids treats a spilled sequence as covering every physical row, as it always does.

Validation

• cargo test -p lance-table
• cargo test -p lance --lib -- rowid row_version stable_row optimize::tests dataset_transactions fragment::tests feature_flag update::tests merge_insert::tests dataset_io
• cargo clippy --all --tests --benches -- -D warnings, cargo fmt --all
• cargo check --manifest-path python/Cargo.toml, cargo check --manifest-path java/lance-jni/Cargo.toml

Refs #8931, #9250

🤖 Generated with Claude Code

[github-actions]

Important

Format specification vote

This PR modifies the Lance format specification, so it requires 3 binding +1 votes from PMC members (excluding the proposer) and a minimum 72-hour voting period, weekends excluded, before it can merge. Vote by approving this PR (+1) or requesting changes (−1, a veto). See the voting process.

Status: ❌ Blocked — 0 of 3 required approvals

Approvals (this commit)	none (0/3)
Vetoes	none
Voting period	ends Tue 2026-09-22 08:56 UTC (01:56 PDT)

_{Updated automatically by the format-spec vote gate, which re-checks every 15 minutes — just voted? Re-check now (press Run workflow; leave the input blank to re-check every open format PR). A PMC member may apply the format-waived label to waive the vote for a trivial edit (typo, wording, formatting).}

[BubbleCal]

Addressed the three gatekeeper points, here and in the stack:

• Single owner for the carrier file: the column arms are now empty RowLineageColumn markers, the file is one of DataFragment.files found by its reserved field id, and zero or several carriers is corruption.
• Executable schema: the three columns are defined with pa.schema as non-nullable uint64, exactly physical_rows values each; nulls or a length mismatch are rejected, never defaulted.
• V1: the marker is valid only next to v2 data files, and a v1 dataset never spills.

The rest of the stack (#9336 through #9340, plus the new #9347, which writes compaction's spilled columns into the fragment's own data file) is rebased on this representation.

[lance-gatekeeper]

✅ Gate recommendation: approve.

3 fixed. The follow-up is borne out by the current head: DataFragment.files is the single carrier authority, the hidden columns have an executable non-nullable uint64 schema with exact row-count and corruption semantics, and the marker is V2-only while V1 writers remain inline. The feature flag fences unsupported readers and writers.