Repository navigation
[python] Support incremental-between with snapshot ids and tags - #10857
Merged
JingsongLi merged 1 commit intoOct 29, 2026
Merged
JingsongLi merged 1 commit into
JingsongLi merged 1 commit into
Conversation
As in Java AbstractDataTableScan, batch reads with incremental-between read the changes between two snapshot ids or two tags, start exclusive and end inclusive, with Java's validation of the range. PyPaimon only recognized incremental-between-timestamp, so these reads fell back to a full read of the latest snapshot. incremental-between-scan-mode is now honored for both options: delta and changelog read the delta or changelog files, and auto picks one by changelog-producer, as in Java. The diff mode, the default diff between two tags (without incremental-between-tag-to-snapshot) and incremental-to-auto-tag are not supported and raise NotImplementedError instead of being ignored. Code that pins a snapshot or rejects incremental scans now treats incremental-between and incremental-to-auto-tag like incremental-between-timestamp.
plusplusjiajia
marked this pull request as ready for review
October 28, 2026 16:35
Contributor
|
+1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
PyPaimon only recognized
incremental-between-timestamp, so batch reads withincremental-betweenset to two snapshot ids or two tags fell back to a full read of the latest snapshot, andincremental-between-scan-modeandincremental-to-auto-tagwere ignored.As in Java
AbstractDataTableScan:incremental-betweenreads the changes between two snapshot ids or two tags, start exclusive and end inclusive, with Java's validation of the range. Tags are resolved before snapshot ids, and branch tables use their own history.incremental-between-scan-modeis honored for bothincremental-betweenandincremental-between-timestamp:deltaandchangelogread the delta or changelog files, andautopicks one bychangelog-producer.The
diffmode, the default diff between two tags (withoutincremental-between-tag-to-snapshot) andincremental-to-auto-tagare not supported and now raiseNotImplementedErrorinstead of being ignored.Code that pins a snapshot or rejects incremental scans (snapshot copies, temporal alignment, window datasets and Ray joins) now treats
incremental-betweenandincremental-to-auto-taglikeincremental-between-timestamp, and native planning leaves these reads to the Python scanner.Tests
native_plan_incremental_test.py: snapshot id and tag ranges on append and primary-key tables, range validation, expired and empty histories, tag resolution order and branch history, the scan modes, and the unsupported options.table_scan_mode_test.py,file_store_table_test.py,native_plan_test.py,multimodal_temporal_test.pyandtable_write_test.py: option validation, snapshot pinning, native fallback, temporal alignment and writes through an incremental copy.