refactor: compose exact file versions explicitly

[Xuanwo]

Important

This draft is an end-to-end layout proposal implemented in code. It is not intended to be merged as one PR. Its purpose is to make the final ownership boundaries, composition model, and migration impact concrete enough to review. If the direction is accepted, we should split this branch into a sequence of smaller, independently validated PRs and land them incrementally.

Online version of this layout: https://www.tldraw.com/p/kT0w5pdO2-miVZL6YWHln?d=v-255.-286.3044.2080.page

Why

Lance file versions are externally parallel persisted formats, but the implementation has historically treated them as an ordered capability progression. Format policy consequently leaked into shared encoding, file, table, and dataset code through comparisons, defaults, compatibility branches, and time-relative concepts such as previous.

This makes changes risky: extending a newer version can accidentally alter an older stable format, while adding a version requires finding implicit policy throughout the repository.

Layout shown by this draft

lance-file::versions
├── v1
├── v2_0
├── v2_1
├── v2_2
└── v2_3

Each exact version is a complete composition root for the grammar it writes and accepts. Shared code below those roots is limited to version-free executable mechanisms. Release selectors such as Stable and Next resolve once at the boundary and are not persisted as identities.

At the other layers:

• lance-encoding contains reusable encoding and compression mechanisms without file-version policy.
• lance-file owns exact reader/writer grammar and runtime dispatch between exact roots.
• lance::dataset::versions owns explicit operation-level dataset differences without capability bags.
• lance-table retains only persisted identity codecs, recovery, and homogeneity validation.

The branch also includes exact-version fixtures and an automated boundary checker so the intended dependency direction is reviewable and enforceable.

Review focus

The useful question for this draft is whether this is the right final layout: whether the ownership boundaries are correct, whether the shared mechanisms are at the right granularity, and whether exact versions are isolated without introducing redundant wrappers or capability profiles.

Please do not review this as a request to land the entire diff directly. Once the layout is agreed, the implementation can be separated into ordered PRs covering exact identity and fixtures, V1 ownership, current-version reader/writer roots, version-free encoding mechanisms, and dataset/table dispatch cleanup.

Validation caveat

The relevant boundary checks, compatibility fixtures, crate tests, and Python/Java lint paths pass. The repository-wide cargo clippy --all --tests --benches -- -D warnings command is currently blocked by the pre-existing CoalesceBatchesExec deprecation in rust/lance/src/dataset/scanner.rs, which is also present on origin/main; targeted strict clippy for the changed architecture passes.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: QUIET

Plan: Pro Plus

Run ID: bba3b681-2381-4a08-ba55-01fb6ce5cbcb

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch xuanwo/exact-file-version-composition-full

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}