Skip to content

[flink] Batch Log Table scan fix #3972 - #4031

Open
binary-signal wants to merge 1 commit into
apache:mainfrom
binary-signal:fix-batch-log
Open

[flink] Batch Log Table scan fix #3972#4031
binary-signal wants to merge 1 commit into
apache:mainfrom
binary-signal:fix-batch-log

Conversation

@binary-signal

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: close #3972

Batch Flink SQL reads of Fluss log tables can fail with
IllegalStateException: Have records for a split that was not registered when
buckets finish while writers keep appending, and can hang when the table is
empty because EARLIEST_OFFSET + stoppingOffset=0 is not treated as an empty
split.

Brief change log

  • In FlinkSourceSplitReader.forLogRecords, unsubscribe finished bounded log
    splits (subscribedBuckets / stoppingOffsets + logScanner.unsubscribe)
    so later appends are not reported for a split that SourceReaderBase has
    already unregistered.
  • Treat stoppingOffset == 0 (and non-negative startingOffset >= stoppingOffset)
    as an empty log split so batch scans of empty buckets finish immediately
    instead of subscribing with the EARLIEST_OFFSET sentinel.

Tests

  • FlinkSourceSplitReaderTest#testSubscribeEmptySplitWithEarliestStartingOffset
    — empty bucket with startingOffset=EARLIEST_OFFSET, stoppingOffset=0
    finishes on the first fetch.
  • FlinkSourceSplitReaderTest#testFinishedLogSplitIsNotFetchedAgain
    — after a bounded log split finishes, further appends to the same bucket are
    not returned under that split id.

API and Format

No. Internal Flink source reader behavior only; no public API or storage format
change.

Documentation

No new user-facing feature. Behavior aligns batch log scans with Flink's
SourceReaderBase finished-split contract.

…apache#3972

Signed-off-by: Evan <binary-signal@users.noreply.github.com>
@binary-signal binary-signal changed the title fix for [flink] Batch Log Table scan fails when some buckets are empt… fix for [flink] Batch Log Table scan fails when #3972 Aug 18, 2026
@binary-signal binary-signal changed the title fix for [flink] Batch Log Table scan fails when #3972 [flink] Batch Log Table scan fix #3972 Aug 18, 2026
@leekeiabstraction

Copy link
Copy Markdown
Contributor

@binary-signal Thank you for your PR, however this seems like a duplicate, can you check that this has been resolved by this merged PR? #4007

@leekeiabstraction leekeiabstraction added the duplicate This issue or pull request already exists label Aug 28, 2026
@leekeiabstraction leekeiabstraction added this to the v1.0 milestone Aug 28, 2026
@binary-signal

Copy link
Copy Markdown
Contributor Author

@leekeiabstraction it has been resolved,

i don't know if it's worth cherry picking from this PR the following

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

duplicate This issue or pull request already exists

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[flink] Batch Log Table scan fails when some buckets are empty

2 participants