Skip to content

[fix](be) Propagate CCR delete bitmap calculation errors - #67143

Open
Ryan19929 wants to merge 2 commits into
apache:masterfrom
Ryan19929:fix/ccr-delete-bitmap-status
Open

[fix](be) Propagate CCR delete bitmap calculation errors#67143
Ryan19929 wants to merge 2 commits into
apache:masterfrom
Ryan19929:fix/ccr-delete-bitmap-status

Conversation

@Ryan19929

@Ryan19929 Ryan19929 commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: close #67120

Related PR: None

Problem Summary:

In Shared-Nothing CCR incremental replication, IngestBinlog discarded both the status returned by BaseTablet::commit_phase_update_delete_bitmap() and the asynchronous delete bitmap token's wait() status.

For UNIQUE KEY merge-on-write tablets, a delete bitmap calculation failure could therefore be logged but not propagated. The downloaded rowset could still be committed and published with an incomplete delete bitmap, exposing historical rows and producing duplicate unique keys.

The failure cleanup also captured download_success_files by value before downloads occurred, so downloaded segment and index files were not removed. Additionally, the result of batch_delete() was discarded and cleanup could be logged as successful even when deletion failed.

This PR:

  • preserves and propagates synchronous task-submission failures;
  • waits for already-submitted delete bitmap tasks and propagates asynchronous worker failures;
  • returns before commit_txn() so the prepared tablet transaction is aborted;
  • captures downloaded files by reference for failure cleanup;
  • reports batch_delete() failures without replacing the original ingest error;
  • adds a CCR MOW fault-injection regression case that verifies asynchronous worker failures keep the failed version invisible and allow the same binlog to be retried safely.

Release note

Fix CCR replay for MOW tables so delete bitmap calculation failures abort IngestBinlog instead of publishing a rowset with an incomplete delete bitmap.

Check List (For Author)

  • Test
    • Regression test
      • test_mow_ingest_binlog, including the new asynchronous delete bitmap worker failure case, passed.
      • The test injects BaseTablet::calc_segment_delete_bitmap.inject_err on the target BE.
      • It verifies that IngestBinlog returns an error from the delete bitmap failure branch before reaching the rowset commit step, the failed version remains invisible, no duplicate key becomes visible, and retrying the same binlog succeeds.
      • CCR P0 cases covering MOW and non-MOW _ingest_binlog paths passed.
    • Unit Test
    • Manual test
      • Full FE and BE ASAN build passed with clang 20.1.7.
      • A Docker two-cluster CCR environment was used, with separate source and target clusters and binlog enabled.
      • build-support/clang-format.sh and build-support/check-format.sh passed with clang-format 16.
      • Changed-lines clang-tidy completed with zero warnings.
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

Known test coverage limitations:

  • The regression test covers an asynchronous delete bitmap worker failure, but does not inject a synchronous task-submission failure from commit_phase_update_delete_bitmap().
  • It does not directly inspect the BE transaction manager to confirm the local transaction state; the result is supported by the executed error path, version invisibility, absence of duplicate keys, and successful retry.
  • It does not inspect the tablet directory before and after failure to verify deletion of downloaded segment or index files.
  • A filesystem batch_delete() failure was not injected, so the cleanup failure logging branch is not covered by a runtime test.

Known environment-only failures unrelated to this change:

  • Two test_create_table_with_binlog_config cases conflict with the Docker image's forced force_enable_feature_binlog setting.

  • test_is_being_synced requires an unavailable external S3 bucket.

  • clang-tidy 20 reports a pre-existing orphan NOLINTEND at types.h:576; no warning was reported on the lines changed by this PR.

  • Behavior changed:

    • No.
    • Yes. CCR IngestBinlog now returns delete bitmap calculation errors and aborts before committing the rowset. Cleanup failures are also reported accurately.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: Shared-Nothing IngestBinlog ignored the status returned by commit-phase delete bitmap task submission and by the asynchronous task token. A failed MOW delete bitmap calculation could therefore continue to commit the rowset without the required bitmap and expose duplicate unique keys. Propagate both statuses, wait for already-submitted tasks before aborting, and return the failure before commit. Capture downloaded files by reference so the existing failure cleanup removes them. Add a CCR MOW fault-injection regression case that verifies the failed version remains invisible and can be retried safely.

### Release note

Fix CCR replay for MOW tables so delete bitmap calculation failures abort ingest instead of committing an incomplete delete bitmap.

### Check List (For Author)

- Test: Regression test added; Groovy syntax parsing passed. The modified BE object compiled with toolchain 0.25, and clang-tidy passed. The full two-cluster CCR regression case was not run because no patched master source/target cluster was available.
- Behavior changed: Yes. IngestBinlog now returns delete bitmap calculation failures and aborts the local tablet transaction.
- Does this need documentation: No
### What problem does this PR solve?

Issue Number: close apache#67120

Related PR: None

Problem Summary: CCR IngestBinlog failure cleanup discarded the status returned by batch_delete and logged cleanup as completed even when file deletion failed. Check the cleanup status and report the secondary cleanup failure without replacing the original ingest error.

### Release note

None

### Check List (For Author)

- Test: Manual test
    - build-support/check-format.sh
    - Compiled backend_service.cpp with toolchain 0.25
    - Changed-lines clang-tidy passed
- Behavior changed: Yes. Failed cleanup is now reported instead of being logged as successful.
- Does this need documentation: No
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@wenzhenghu

Copy link
Copy Markdown
Contributor

Reviewed the current head aecf7779e87354bc7f8f1f704b614c92827854f7 against master@272596237e9cfe2504289bfe3ce539d8027cc528.

No blocking issue was found. LGTM from the static review perspective.

Key conclusions:

  • The statuses returned by BaseTablet::commit_phase_update_delete_bitmap() and CalcDeleteBitmapToken::wait() are both propagated correctly.
  • Calling wait() even after a synchronous submission failure is necessary: tasks already submitted to the token must finish before the rowset, segments, and downloaded files can be released or removed.
  • A delete bitmap failure returns before commit_txn(). The existing deferred handler then aborts the local tablet transaction, so an incomplete MOW delete bitmap cannot be attached to a committed rowset through this path.
  • Capturing download_success_files by reference is lifetime-safe here and allows the failure handler to remove the files actually downloaded during the ingest. A secondary batch_delete() failure is logged without replacing the original ingest error.
  • The ordinary BaseRowsetBuilder path already propagates both submission and wait failures, so limiting this change to the Shared-Nothing CCR IngestBinlog path is appropriately scoped.
  • No new lock ordering, storage-format, Thrift/protocol, configuration, or rolling-upgrade compatibility change is introduced. The token is drained before failure cleanup, and no new concurrency or object-lifetime issue was found.
  • The regression case exercises the asynchronous delete bitmap failure, verifies that ingest reports failure and the target version is not advanced, and verifies that retrying the same binlog succeeds without exposing duplicate unique keys.

Validation performed in this review:

  • Confirmed the exact PR head/base and the two-file PR diff (+61/-6).
  • Traced the Shared-Nothing ingest path, CalcDeleteBitmapToken, BaseTablet::commit_phase_update_delete_bitmap(), TxnManager, and the ordinary rowset-builder path.
  • git diff --check passed for the PR diff.
  • This was a static review; I did not independently run a BE build, BE unit tests, or the two-cluster CCR regression suite.

@wenzhenghu wenzhenghu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] CCR IngestBinlog may commit MOW rowsets after delete bitmap calculation fails

3 participants