[fix](streaming-job) sync sourceProperties and targetProperties in replayOnUpdated#62553
Open
JNSimba wants to merge 2 commits intoapache:masterfrom
Open
[fix](streaming-job) sync sourceProperties and targetProperties in replayOnUpdated#62553JNSimba wants to merge 2 commits intoapache:masterfrom
JNSimba wants to merge 2 commits intoapache:masterfrom
Conversation
…sistence after FE restart Regression test to verify that sourceProperties and targetProperties altered via ALTER JOB survive FE restart. Expected to fail until StreamingInsertJob.replayOnUpdated() is fixed to sync both property maps. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…playOnUpdated After ALTER JOB changes source/target properties, replayOnUpdated() did not propagate sprops/tprops into the replaying job. The checkpoint thread replays journals on its own Env, so the generated image also lost the ALTERed values — any subsequent FE restart (from image or EditLog) would revert to the original CREATE-time properties. For CDC jobs this can cause follower failover to use stale JDBC credentials or miss updated load options such as max_filter_ratio. Also tighten the regression test's ALTER JOB to only specify the changed properties. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Member
Author
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
FE Regression Coverage ReportIncrement line coverage |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Problem Summary:
After
ALTER JOBchanges a streaming job's source/target properties,StreamingInsertJob.replayOnUpdated()did not propagatesourcePropertiesand
targetPropertiesfrom the replay job to the in-memory job.The checkpoint thread replays journals on its own
Env, so the generatedimage also lost the ALTERed values — any subsequent FE restart (from image
or EditLog) would revert to the original CREATE-time properties. For CDC
jobs this can cause follower failover to use stale JDBC credentials or
miss updated load options such as
max_filter_ratio.Fix: sync both maps in
replayOnUpdated()with null-guards, matching thepattern used for
offsetProviderPersist(#62449).Release note
Fix streaming job ALTER JOB source/target properties being lost after FE checkpoint/restart.
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)