[core] Preserve timestamp values on changelog retraction records - #9691
[core] Preserve timestamp values on changelog retraction records#9691junmuz wants to merge 2 commits into
Conversation
56365ee to
ba28382
Compare
|
@JingsongLi Can I get a review on the PR. |
JingsongLi
left a comment
There was a problem hiding this comment.
The Cassandra event-timestamp use case has end-to-end value, but rewriting ordinary before-image fields needs a different contract boundary before merging. The Flink source still advertises a normal all-kinds changelog, whose -U/-D rows must retract the previously emitted values.
For example, after +I(id=1, event_ts=50), an update to event_ts=100 now emits -U(id=1, event_ts=100), +U(id=1, event_ts=100). A downstream WHERE event_ts < 75 never sees the retraction and keeps the old row. GROUP BY event_ts also retains the old group because the retraction addresses the new group. I reproduced the emitted records using the exact merge/wrapper source and checked them with filter/group consumer probes; this was not a full Flink job.
Please expose the incoming event timestamp as separate metadata/a CDC envelope for the external sink, preserving the stored before-image, or define and enforce a separate restricted consumption mode instead of advertising these rows as a normal relational changelog. Add an end-to-end consumer test covering both a filter and an aggregation before accepting the new option. Also clarify that the current helper reads the merged result, which need not equal the incoming event for aggregation merge engines.
Purpose
Tests
Manual verification