Skip to content

feat(namespace): idempotent create_table_version with strict CAS and dotfile-safe reserved marker - #7940

Open
XuQianJin-Stars wants to merge 3 commits into
lance-format:mainfrom
XuQianJin-Stars:fix/namespace-idempotent-create-table-version
Open

feat(namespace): idempotent create_table_version with strict CAS and dotfile-safe reserved marker#7940
XuQianJin-Stars wants to merge 3 commits into
lance-format:mainfrom
XuQianJin-Stars:fix/namespace-idempotent-create-table-version

Conversation

@XuQianJin-Stars

Copy link
Copy Markdown
Contributor

Summary

Make create_table_version in the directory namespace (lance-namespace-impls)
idempotent, strictly enforce version CAS, and make the .lance-reserved marker
creation safe across object stores.

Problem

In the namespace directory provider, create_table_version is currently unsafe
against the retries and races that happen with real object-store deployments:

  1. Non-idempotent retries. If a version is successfully created but the
    client retries (network blip, or the coordinating node renamed staging →
    final while the caller lost the response), the second attempt sees the
    version already published and fails with ConcurrentModification. Callers
    cannot distinguish a genuine conflict from a benign retry.

  2. Loose version CAS. There is no enforced "must be latest + 1" check,
    so a stale or duplicate write can fork the version chain.

  3. Reserved-marker staging breaks on some object stores. The table is
    reserved by writing a .lance-reserved marker. Some backends implement
    PutMode::Create as a temp-file + rename, producing temporary names
    containing ... Object stores and unified file systems that reject ..
    path components then fail the marker write and the table can never be
    created.

Changes

  • Strict CAS: create_table_version only accepts version == latest + 1
    (version == 1 on an empty chain); any other value maps to
    NamespaceError::ConcurrentModification.
  • Idempotent retry on published versions: if the requested version already
    exists with identical manifest content (e_tag / size / bytes), return
    success instead of ConcurrentModification.
  • Content-conflict detection: if a version with the same number exists but
    content differs, fail with ConcurrentModification.
  • Dotfile-safe reserved marker: stage via a non-dot sibling file then
    rename_if_not_exists (conditional no-replace) onto .lance-reserved; fall
    back to PutMode::Create when rename is unsupported.

Test plan

  • Added tests for idempotent retry, content conflict, and CAS gap rejection,
    plus table-not-found and on-branch variants.
  • cargo test -p lance-namespace-impls --lib create_table_version passes.

Scope

Targets the dir namespace provider (create_table_version / reserved-marker
lifecycle). The retry/conflict semantics apply to any object-store backend,
not a specific one.

Closes #7939

@github-actions github-actions Bot added the A-namespace Namespace impls label Jul 23, 2026
@github-actions

Copy link
Copy Markdown
Contributor

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

@coderabbitai

coderabbitai Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

The directory namespace now publishes reserved markers through staging and conditional creation, and creates table versions with strict sequential CAS, idempotent identical-content retries, conflict detection, and create-only publication semantics.

Changes

Namespace publication

Layer / File(s) Summary
Dotfile-safe marker publication
rust/lance-namespace-impls/src/dir.rs, rust/lance-namespace-impls/src/dir/manifest.rs
Reserved markers are staged under non-dot sibling paths and published with conditional rename or PutMode::Create, with best-effort staging cleanup.
Idempotent table-version publication and validation
rust/lance-namespace-impls/src/dir.rs
Table-version creation validates staging existence, enforces latest + 1, resolves identical-content retries, rejects differing content, and publishes manifests without overwriting. Tests cover retries, conflicts, and version gaps.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant create_table_version
  participant ObjectStore
  participant ManifestResolver
  Client->>create_table_version: submit version and staging manifest
  create_table_version->>ObjectStore: head final manifest
  ObjectStore-->>create_table_version: existing or missing manifest
  create_table_version->>ManifestResolver: compare staged and final bytes
  ManifestResolver-->>create_table_version: success or ConcurrentModification
  create_table_version->>ObjectStore: create-only publish
  ObjectStore-->>Client: published manifest response
Loading

Suggested reviewers: jackye1995, xuanwo, hamersaw

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately summarizes the main change: idempotent create_table_version with strict CAS and dotfile-safe reserved marker handling.
Description check ✅ Passed The description matches the implemented namespace changes and test coverage.
Linked Issues check ✅ Passed The changes address the linked issue requirements for idempotent retries, strict CAS, and dotfile-safe marker creation.
Out of Scope Changes check ✅ Passed No clear out-of-scope code changes are evident beyond the namespace versioning and marker robustness work.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@XuQianJin-Stars XuQianJin-Stars changed the title feat(namespace-dir): idempotent create_table_version with strict CAS and dotfile-safe reserved marker feat(namespace-dir): idempotent create_table_version with strict CAS and dotfile-safe reserved marker Jul 23, 2026
@github-actions github-actions Bot added the enhancement New feature or request label Jul 23, 2026
@XuQianJin-Stars XuQianJin-Stars changed the title feat(namespace-dir): idempotent create_table_version with strict CAS and dotfile-safe reserved marker feat(namespace): idempotent create_table_version with strict CAS and dotfile-safe reserved marker Jul 23, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (2)
rust/lance-namespace-impls/src/dir.rs (1)

11308-11312: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Assert on the error variant, not just the message.

test_create_table_version_conflict and test_create_table_version_cas_rejects_gap only check the error string. This file already has a namespace_code(&err) helper; asserting ErrorCode::ConcurrentModification in addition to the message makes these tests robust against message wording changes.

As per coding guidelines: "Assert on both the error variant and the message content in tests; do not check only is_err()."

Also applies to: 11413-11417

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@rust/lance-namespace-impls/src/dir.rs` around lines 11308 - 11312, Update
test_create_table_version_conflict and test_create_table_version_cas_rejects_gap
to assert that namespace_code(&err) equals ErrorCode::ConcurrentModification,
while retaining the existing message-content checks for “already exists” or
“ConcurrentModification”.

Source: Coding guidelines

rust/lance-namespace-impls/src/dir/manifest.rs (1)

3493-3576: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Deduplicate the dotfile-safe marker publication logic. Both sites implement the identical stage-non-dot-sibling → rename_if_not_existsPutMode::Create fallback → best-effort cleanup flow; keeping two copies risks divergent fixes to this security/correctness-sensitive path.

  • rust/lance-namespace-impls/src/dir/manifest.rs#L3493-L3576: replace the inline staging/publish block in declare_table with a call to a shared helper (e.g. a free function put_marker_file_atomic(object_store, path, description) in the crate).
  • rust/lance-namespace-impls/src/dir.rs#L2685-L2751: extract the body of DirectoryNamespace::put_marker_file_atomic into that shared helper so both callers stay in sync.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@rust/lance-namespace-impls/src/dir/manifest.rs` around lines 3493 - 3576,
Deduplicate marker publication by extracting the existing
DirectoryNamespace::put_marker_file_atomic implementation into a shared
crate-level helper, preserving staging, conditional rename, PutMode::Create
fallback, and best-effort cleanup. In
rust/lance-namespace-impls/src/dir.rs#2685-2751, move the method body into the
helper and update the method to use it if needed; in
rust/lance-namespace-impls/src/dir/manifest.rs#3493-3576, replace the inline
declare_table flow with the shared helper, passing the object store, marker
path, and description while preserving current error context.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@rust/lance-namespace-impls/src/dir/manifest.rs`:
- Around line 3531-3546: Move the inline object_store import from the
declare_table fallback arm to the file-level imports, adding PutMode and
PutOptions alongside the existing ObjectStoreError and path::Path imports.
Remove the function-body use declaration while leaving the put_opts logic
unchanged.

---

Nitpick comments:
In `@rust/lance-namespace-impls/src/dir.rs`:
- Around line 11308-11312: Update test_create_table_version_conflict and
test_create_table_version_cas_rejects_gap to assert that namespace_code(&err)
equals ErrorCode::ConcurrentModification, while retaining the existing
message-content checks for “already exists” or “ConcurrentModification”.

In `@rust/lance-namespace-impls/src/dir/manifest.rs`:
- Around line 3493-3576: Deduplicate marker publication by extracting the
existing DirectoryNamespace::put_marker_file_atomic implementation into a shared
crate-level helper, preserving staging, conditional rename, PutMode::Create
fallback, and best-effort cleanup. In
rust/lance-namespace-impls/src/dir.rs#2685-2751, move the method body into the
helper and update the method to use it if needed; in
rust/lance-namespace-impls/src/dir/manifest.rs#3493-3576, replace the inline
declare_table flow with the shared helper, passing the object store, marker
path, and description while preserving current error context.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: QUIET

Plan: Pro Plus

Run ID: 5a061bb4-d017-42be-8f71-3666f88028b9

📥 Commits

Reviewing files that changed from the base of the PR and between 8f51a21 and 485cdfa.

📒 Files selected for processing (2)
  • rust/lance-namespace-impls/src/dir.rs
  • rust/lance-namespace-impls/src/dir/manifest.rs

Comment on lines +3531 to +3546
Err(ObjectStoreError::NotImplemented { .. })
| Err(ObjectStoreError::NotSupported { .. }) => {
use object_store::{PutMode, PutOptions};
let result = self
.object_store
.inner
.put_opts(
&reserved_file_path,
Bytes::from_static(b"reserved").into(),
PutOptions {
mode: PutMode::Create,
..Default::default()
},
)
.await
.map(|_| ());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win

Move use object_store::{PutMode, PutOptions}; to the top of the file.

This use is declared inside the declare_table fallback arm. The top-level import (Line 63) already brings in ObjectStoreError and path::Path; add PutMode/PutOptions there.

As per coding guidelines: "Place use imports at the top of the file, not inline within function bodies."

♻️ Proposed change
-use object_store::{Error as ObjectStoreError, ObjectStoreExt, path::Path};
+use object_store::{Error as ObjectStoreError, ObjectStoreExt, PutMode, PutOptions, path::Path};
             Err(ObjectStoreError::NotImplemented { .. })
             | Err(ObjectStoreError::NotSupported { .. }) => {
-                use object_store::{PutMode, PutOptions};
                 let result = self
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
Err(ObjectStoreError::NotImplemented { .. })
| Err(ObjectStoreError::NotSupported { .. }) => {
use object_store::{PutMode, PutOptions};
let result = self
.object_store
.inner
.put_opts(
&reserved_file_path,
Bytes::from_static(b"reserved").into(),
PutOptions {
mode: PutMode::Create,
..Default::default()
},
)
.await
.map(|_| ());
Err(ObjectStoreError::NotImplemented { .. })
| Err(ObjectStoreError::NotSupported { .. }) => {
let result = self
.object_store
.inner
.put_opts(
&reserved_file_path,
Bytes::from_static(b"reserved").into(),
PutOptions {
mode: PutMode::Create,
..Default::default()
},
)
.await
.map(|_| ());
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@rust/lance-namespace-impls/src/dir/manifest.rs` around lines 3531 - 3546,
Move the inline object_store import from the declare_table fallback arm to the
file-level imports, adding PutMode and PutOptions alongside the existing
ObjectStoreError and path::Path imports. Remove the function-body use
declaration while leaving the put_opts logic unchanged.

Source: Coding guidelines

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)
rust/lance-namespace-impls/src/dir.rs (2)

11304-11313: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Assert the namespace error code as well as message text. These tests can pass if the implementation returns the wrong error variant with a similar message.

  • rust/lance-namespace-impls/src/dir.rs#L11304-L11313: assert ErrorCode::ConcurrentModification before checking the message.
  • rust/lance-namespace-impls/src/dir.rs#L11412-L11420: assert ErrorCode::ConcurrentModification before checking the message.

As per coding guidelines, “Assert on both the error variant and the message content in tests; do not check only is_err().”

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@rust/lance-namespace-impls/src/dir.rs` around lines 11304 - 11313, Update
both error assertions in rust/lance-namespace-impls/src/dir.rs at lines
11304-11313 and 11412-11420 to verify the returned error has
ErrorCode::ConcurrentModification before checking its message content. Replace
the broad is_err/message-only checks while preserving the existing message
validation.

Source: Coding guidelines


1650-1654: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Do not accept a caller-supplied ETag as manifest identity.

The REST request forwards e_tag directly, but this shortcut never verifies it against the staging object. A request can supply the published manifest’s ETag with different staging bytes, receive a false idempotent success, and have its staging manifest deleted. Always compare content (or a verified digest of both objects); retain size only as a rejection fast path.

Proposed fix
-        if let (Some(req_tag), Some(exist_tag)) = (request_e_tag, final_meta.e_tag.as_deref())
-            && req_tag == exist_tag
-        {
-            return Ok(true);
-        }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@rust/lance-namespace-impls/src/dir.rs` around lines 1650 - 1654, Remove the
request-Etag equality shortcut in the manifest identity check near
final_meta.e_tag. Determine idempotent identity by comparing the staging and
published manifest contents, or verified digests of both objects; retain size
comparison only as an early rejection fast path, and do not delete staging or
return success unless content matches.
🧹 Nitpick comments (1)
rust/lance-namespace-impls/src/dir.rs (1)

11069-11071: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Move new test imports to the module import block.

  • rust/lance-namespace-impls/src/dir.rs#L11069-L11071: move these imports to the test module imports.
  • rust/lance-namespace-impls/src/dir.rs#L11184-L11186: reuse the module-level imports.
  • rust/lance-namespace-impls/src/dir.rs#L11333-L11335: reuse the module-level imports.

As per coding guidelines, “Place use imports at the top of the file, not inline within function bodies.”

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@rust/lance-namespace-impls/src/dir.rs` around lines 11069 - 11071, Move the
inline imports for TryStreamExt, DatasetBuilder, and CreateTableVersionRequest
into the test module’s top-level import block in
rust/lance-namespace-impls/src/dir.rs at lines 11069-11071, then remove the
duplicate local imports at lines 11184-11186 and 11333-11335 so those tests
reuse the module-level imports.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@rust/lance-namespace-impls/src/dir.rs`:
- Around line 1748-1764: Update enforce_create_table_version_cas so an empty
chain accepts a non-1 initial version only for the branch-bootstrap path, while
the main path requires version 1. Replace saturating_add(1) with checked_add(1),
returning a contextual error when the latest version overflows.

---

Outside diff comments:
In `@rust/lance-namespace-impls/src/dir.rs`:
- Around line 11304-11313: Update both error assertions in
rust/lance-namespace-impls/src/dir.rs at lines 11304-11313 and 11412-11420 to
verify the returned error has ErrorCode::ConcurrentModification before checking
its message content. Replace the broad is_err/message-only checks while
preserving the existing message validation.
- Around line 1650-1654: Remove the request-Etag equality shortcut in the
manifest identity check near final_meta.e_tag. Determine idempotent identity by
comparing the staging and published manifest contents, or verified digests of
both objects; retain size comparison only as an early rejection fast path, and
do not delete staging or return success unless content matches.

---

Nitpick comments:
In `@rust/lance-namespace-impls/src/dir.rs`:
- Around line 11069-11071: Move the inline imports for TryStreamExt,
DatasetBuilder, and CreateTableVersionRequest into the test module’s top-level
import block in rust/lance-namespace-impls/src/dir.rs at lines 11069-11071, then
remove the duplicate local imports at lines 11184-11186 and 11333-11335 so those
tests reuse the module-level imports.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: QUIET

Plan: Pro Plus

Run ID: dbad1e3b-0fc0-4049-bc07-62a7f8143400

📥 Commits

Reviewing files that changed from the base of the PR and between 485cdfa and 28fee9e.

📒 Files selected for processing (1)
  • rust/lance-namespace-impls/src/dir.rs

Comment thread rust/lance-namespace-impls/src/dir.rs Outdated
@XuQianJin-Stars
XuQianJin-Stars force-pushed the fix/namespace-idempotent-create-table-version branch from 28fee9e to 6a2fd05 Compare July 24, 2026 02:55

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
rust/lance-namespace-impls/src/dir.rs (1)

1650-1654: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Do not treat a request ETag as proof of staging-content equality.

A caller can reuse the published final manifest’s ETag in request.e_tag while supplying different staging bytes. This returns idempotent success without reading staging content, violating the required differing-content conflict behavior. Remove this success shortcut and always compare bytes (or a server-verified content digest); add a regression test with a matching request ETag and different staging bytes.

Proposed fix
-        if let (Some(req_tag), Some(exist_tag)) = (request_e_tag, final_meta.e_tag.as_deref())
-            && req_tag == exist_tag
-        {
-            return Ok(true);
-        }
+        // Request ETags are not a verified digest of `staging_path`; use them
+        // only as metadata hints, never as proof of content equality.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@rust/lance-namespace-impls/src/dir.rs` around lines 1650 - 1654, Remove the
request_e_tag/final_meta.e_tag equality early return in the surrounding manifest
validation flow so a matching request ETag cannot establish staging-content
equality. Always compare the staging bytes or a server-verified content digest
and return the differing-content conflict when they do not match; add a
regression test covering a matching request ETag with different staging bytes.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@rust/lance-namespace-impls/src/dir.rs`:
- Around line 1650-1654: Remove the request_e_tag/final_meta.e_tag equality
early return in the surrounding manifest validation flow so a matching request
ETag cannot establish staging-content equality. Always compare the staging bytes
or a server-verified content digest and return the differing-content conflict
when they do not match; add a regression test covering a matching request ETag
with different staging bytes.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: QUIET

Plan: Pro Plus

Run ID: 3f5dec13-b059-4c6a-be2d-8e105504b8ba

📥 Commits

Reviewing files that changed from the base of the PR and between 28fee9e and 6a2fd05.

📒 Files selected for processing (1)
  • rust/lance-namespace-impls/src/dir.rs

@XuQianJin-Stars
XuQianJin-Stars force-pushed the fix/namespace-idempotent-create-table-version branch from 6a2fd05 to 0e421e5 Compare July 24, 2026 03:06
@codecov

codecov Bot commented Jul 24, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 70.87379% with 120 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
rust/lance-namespace-impls/src/dir.rs 72.45% 98 Missing and 13 partials ⚠️
rust/lance-namespace-impls/src/dir/manifest.rs 0.00% 9 Missing ⚠️

📢 Thoughts on this report? Let us know!

…dotfile-safe reserved marker

- Enforce strict version CAS (latest+1) in create_table_version and treat
  an already-published identical-content version as idempotent success
  instead of ConcurrentModification.
- Resolve create conflicts by comparing staging vs final manifest content
  (e_tag/size/byte equality) so retries succeed after a conditional rename
  wins the publish race.
- Stage .lance-reserved via a non-dot sibling and rename_if_not_exists to
  avoid object stores whose Create uses temp+rename producing '..' temp
  names rejected by the backing store; fall back to PutMode::Create when
  rename is unsupported.
- Add tests for idempotent retry, content conflict, and CAS gap rejection.
@XuQianJin-Stars
XuQianJin-Stars force-pushed the fix/namespace-idempotent-create-table-version branch from 0e421e5 to f184470 Compare July 24, 2026 04:49

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.

🟡 Other comments (1)
rust/lance-namespace-impls/src/dir/manifest.rs-3568-3575 (1)

3568-3575: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Map atomic marker publish failures to TableAlreadyExists.

declare_table keeps the manifest-lookup guard but then writes the .lance-reserved marker directly. If two concurrent calls both miss the manifest check, the loser loses on rename_if_not_exists or the PutMode::Create fallback. Map AlreadyExists and Precondition failures from this publish to NamespaceError::TableAlreadyExists { table_name }; all other publish failures may remain Internal.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@rust/lance-namespace-impls/src/dir/manifest.rs` around lines 3568 - 3575,
Update the error mapping around the atomic marker publish in declare_table so
AlreadyExists and Precondition failures from rename_if_not_exists or the
PutMode::Create fallback become NamespaceError::TableAlreadyExists { table_name
}. Preserve the existing Internal mapping for all other publish failures and
keep the manifest-lookup guard unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Other comments:
In `@rust/lance-namespace-impls/src/dir/manifest.rs`:
- Around line 3568-3575: Update the error mapping around the atomic marker
publish in declare_table so AlreadyExists and Precondition failures from
rename_if_not_exists or the PutMode::Create fallback become
NamespaceError::TableAlreadyExists { table_name }. Preserve the existing
Internal mapping for all other publish failures and keep the manifest-lookup
guard unchanged.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: QUIET

Plan: Pro Plus

Run ID: e90838e0-347c-46c8-8d8e-726db082f285

📥 Commits

Reviewing files that changed from the base of the PR and between 0e421e5 and f184470.

📒 Files selected for processing (2)
  • rust/lance-namespace-impls/src/dir.rs
  • rust/lance-namespace-impls/src/dir/manifest.rs

A branch's first commit carries its fork version, which can be greater than
1 (e.g. forking main at v2). enforce_create_table_version_cas previously
hardcoded the expected version to 1 for an empty chain, so creating a branch
forked from a non-v1 version failed with DatasetAlreadyExists.

Relax the CAS so an empty branch chain accepts the requested fork version as
its bootstrap, while an empty main chain still must start at v1.
Replace saturating_add(1) with checked_add(1) when computing the expected
next version in enforce_create_table_version_cas, so a u64 version overflow
returns a descriptive ConcurrentModification error instead of silently
wrapping around.

@yanghua yanghua left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some comments.

Comment on lines +1773 to +1781
// Empty chain: main starts at v1, but a branch bootstraps at its fork
// version which can be > 1.
None => {
if is_branch {
version
} else {
1
}
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

enforce_create_table_version_cas only retrieves is_branch: bool, failing to obtain
BranchContents.parent_version. The documentation comment states that "branch bootstraps at its fork version", yet the actual implementation permits any arbitrary version (such as 1) to be set as the first
manifest of a branch forked from v5, breaking the semantics of "branch inherits main up to fork version".
resolve_branch_for_commit has already fetched BranchContents (the Ok(_) branch discards parent_version), and passing it downstream to perform the version == parent_version + 1 validation incurs negligible overhead.

Comment on lines +3568 to +3575
publish.map_err(|e| {
lance_core::Error::from(NamespaceError::Internal {
message: format!(
"Failed to create .lance-reserved file for table {}: {}",
table_name, e
),
})
})?;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When .lance-reserved already exists, the AlreadyExists / Precondition returned by rename_if_not_exists or the fallback PutMode::Create is indiscriminately wrapped as NamespaceError::Internal. In contrast, DirectoryNamespace::declare_table within the same crate explicitly maps such errors to NamespaceError::TableAlreadyExists via put_marker_file_atomic. Trigger scenarios: concurrent declare_table calls, retries after a previous incomplete declare operation. Callers cannot programmatically distinguish between "IO failures" and "table already declared". A directly applicable patch is attached.

Comment on lines +3493 to +3505
// Create the .lance-reserved file to mark the table as existing.
// Avoid `object_store.create(.lance-reserved)` / PutMode::Create on the
// final dotfile: some object stores implement Create via temp+rename
// using the final basename, which yields `..lance-reserved` temps that
// the underlying store rejects. Stage under a non-dot sibling, then
// rename_if_not_exists.
let reserved_file_path = table_path.clone().join(".lance-reserved");
let staging_name = format!("lance-marker.staging.{}", Uuid::new_v4().simple());
let staging_path = table_path.clone().join(staging_name.as_str());

// Some object stores write via temp+rename and may not flush empty
// objects through to the underlying filesystem, so the conditional
// rename can fail with NotFound. Use a tiny non-empty payload.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this duplicated with dir.rs::put_marker_file_atomic?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-namespace Namespace impls enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: Make namespace dir create_table_version idempotent with strict CAS and dotfile-safe reserved marker

2 participants