SS-454 - design: one ceiling and one batch for a pinned snapshot by martykulma · Pull Request #38932 · MaterializeInc/materialize

martykulma · 2026-09-19T00:30:16Z

Updates the design doc to a simpler approach. Instead of having the minter create descriptions that group data into multiple batches, the minter commits to a ceiling timestamp.

The minter commits a ceiling a "lookahead" (configurable) past the newest data and broadcasts it to the writers. Writers put everything below the ceiling into one builder, whatever the timestamp, so snapshot rows and replication rows share a batch.

Net effect is the snapshot plus its catch-up is one description, one append. The shard upper waits for the frontier to reach the ceiling, about one lookahead no matter how long the snapshot ran.

Rows that outrun the ceiling write their own batch, same as everything does with
the flag off.

🤖 Generated with Claude Code
👨 Revised heavily by Human (@martykulma)

linear-code · 2026-09-21T13:13:18Z

SS-454

ublubu · 2026-09-23T12:33:16Z

+only once `desired_frontier` reaches the upper.
+
+A ceiling has to reach the writers before the rows it covers, since a builder only takes rows at
+times it was opened for and cannot be finished at an upper below a row it holds. The first is


Suggested change

times it was opened for and cannot be finished at an upper below a row it holds. The first is

times it was opened for and cannot be finished at an upper below a row it holds. The first ceiling is

ublubu · 2026-09-23T12:36:02Z

+source's `timestamp_interval`.
+
+The minter has two rules.
+- Emit `[current_upper, desired_frontier)` whenever the frontier is ahead and no outstanding


What are current_upper and desired_frontier?

Is current_upper the end of the previous batch description?
desired_frontier is that lookahead past the largest timestamp any worker has seen?

current_upper is the upper the mint operator tracks and desired_frontier, is the frontier of the data stream (desired_input).

When the data frontier has moved past the operator's frontier (current_upper < desired_frontier), the current_upper becomes the lower for the description that gets minted. This is the normal mode of operation that exists today - both are driven by frontier advancements.

The lookahead is the next rule, and it doesn't affect either current_upper or desired_frontier.

Makes sense. I think this segment should explain those terms. The current phrasing of the rule is more abstract than I'd like.

"commits to nothing" doesn't have one obvious meaning to me either.

ublubu · 2026-09-23T12:38:09Z

+- Commit `max_seen + lookahead` whenever that is past the current ceiling, while the export's
+  snapshot is in progress.
+
+Committing costs the minter the coupling between the upper it emits and the frontier it observes.


costs the minter the coupling

Unclear whether the cost is losing the coupling or being saddled with the coupling.

I'll update.

It breaks the coupling, but only for the brief period where the minter committed to a ceiling and the desired_frontier jumped forward to some time before the ceiling (e.g. current_upper < desired_frontier < ceiling).

During normal operation, as soon as current_upper < desired_frontier, the minter would emit a description. During the snapshot though, it won't until the desired_frontier reaches the ceiling (these are both uppers, so both exclusive).

ublubu · 2026-09-23T12:43:40Z

-When the snapshot ends, `F` jumps to wherever the source has reached. Every window below it appends
-in one pass. The window it lands in binds, and the distance from `F` to `C` is the tail:
+When the snapshot ends, `F` jumps to wherever the source has reached and then climbs to `C`. The
+distance it still has to cover is the tail, about one lookahead whatever the snapshot's length,


whatever the snapshot's length

It's not clear to me what incorrect assumptions this parenthetical is supposed to save the reader from.

It's also not clear to me what "snapshot's length" means.

Would duration make more sense? This is trying to capture the independence of lookahead and the size/duration of the snapshot.

"size/duration" works better for me. Or "size" or "duration".

Man, it's weird that "length" means both those things (and both meanings are close enough), so it should be perfect, but instead it just makes me wonder which one it is.

ublubu · 2026-09-23T12:45:49Z

+MZ time  ----c--------------------------+---+----+---→   F jumps to where the source reached
+                                        ┼────tail─┼      shard upper waits at c until F = C


Suggested change

MZ time ----c--------------------------+---+----+---→ F jumps to where the source reached

┼────tail─┼ shard upper waits at c until F = C

MZ time ----c--------------------------+---+----+---→ F jumps to where the source reached.

┼────tail─┼ shard upper waits at c until F = C

Added a period because I read this as:

F jumps to where the source reached shard upper...

ublubu · 2026-09-23T12:48:09Z

-there and a committed upper the data never reaches would hold the shard upper forever.
+An export snapshots when its resume upper is at or below the `as_of`. The controller uses the same
+test to hand the connector a minimum from-time resume upper, so the sink and the connector agree on
+which tables snapshot. A restart mid-snapshot still reads as one, since the shard upper has moved


reads as one

Reads as one of what?

still reads as a snapshot, will update.

ublubu · 2026-09-23T12:55:13Z

+snapshot. The sink gets the `as_of` for such an export. That is `T:c`, the time its snapshot lands
+at.
+
+Only the OLTP sources take part, the ones that snapshot alongside their CDC stream. They are what


Only the OLTP sources take part

Interesting framing of agency here.

Maybe personal preference, but it could be more obvious what we mean by:

keeps the upsert envelope out

CDCv2 envelope stays out for a reason of its own

It'd be easier for me to follow something like:

Where we won't do this:

Kafka and Load generators: Those don't snapshot.

Upsert envelope: <explain the deadlock>

CDCv2: MZ times come from the data, not reclocking, so wall-clock lookahead means nothing...

I'll try to enhance without adding too much detail.

ublubu · 2026-09-23T12:56:21Z

-
-Leaving the window at zero mints descriptions from the frontier alone, so the sink writes one batch
-per timestamp exactly as it does today.
+The minter permits the second rule only while `desired_frontier` equals `T:c`, and it is told


I cannot, for the life of me, remember what the rules are and what order they exist in 😿

lookahead rule - updated

ublubu

Send it 🏂

ublubu · 2026-09-23T21:36:15Z

+source's `timestamp_interval`.
+
+The minter has two rules.
+- Emit `[current_upper, desired_frontier)` whenever the frontier is ahead and no outstanding


Makes sense. I think this segment should explain those terms. The current phrasing of the rule is more abstract than I'd like.

"commits to nothing" doesn't have one obvious meaning to me either.

ublubu · 2026-09-23T21:39:46Z

-When the snapshot ends, `F` jumps to wherever the source has reached. Every window below it appends
-in one pass. The window it lands in binds, and the distance from `F` to `C` is the tail:
+When the snapshot ends, `F` jumps to wherever the source has reached and then climbs to `C`. The
+distance it still has to cover is the tail, about one lookahead whatever the snapshot's length,


"size/duration" works better for me. Or "size" or "duration".

Man, it's weird that "length" means both those things (and both meanings are close enough), so it should be perfect, but instead it just makes me wonder which one it is.

Step 4 is rewritten for a bound the minter commits and broadcasts to the writers rather than mints as a description. The writers group every update below it into one builder, and the minter mints nothing below an outstanding ceiling, so the whole snapshot and the catch-up behind it become one description appended once, whatever the snapshot's length. The lookahead is fixed rather than a width that doubles, which puts the tail at about one lookahead instead of growing with the snapshot's duration. That closes the tail open question, and it makes restart case 2 unreachable, since the shard upper now sits at T:c until the one description goes in. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The persist sink is handed the as_of of an export that snapshots in this incarnation and commits a ceiling only while the frontier sits there. A fresh source's snapshot lands at the minimum, where no progress statement ever arrives, so inferring the pin from the first non-minimum frontier never opened the gate for it. Co-Authored-By: Claude <noreply@anthropic.com>

@martykulma

) Updates the design doc to a simpler approach. Instead of having the minter create descriptions that group data into multiple batches, the minter commits to a ceiling timestamp. The minter commits a ceiling a "lookahead" (configurable) past the newest data and broadcasts it to the writers. Writers put everything below the ceiling into one builder, whatever the timestamp, so snapshot rows and replication rows share a batch. Net effect is the snapshot plus its catch-up is one description, one append. The shard upper waits for the frontier to reach the ceiling, about one lookahead no matter how long the snapshot ran. Rows that outrun the ceiling write their own batch, same as everything does with the flag off. 🤖 Generated with [Claude Code](https://claude.com/claude-code) 👨 Revised heavily by Human (@martykulma) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

martykulma added this pull request to stack #38937 September 19, 2026 00:30

martykulma changed the title ~~maz sink ceiling design~~ design: one ceiling and one batch for a pinned snapshot Sep 19, 2026

martykulma changed the title ~~design: one ceiling and one batch for a pinned snapshot~~ SS-454 - design: one ceiling and one batch for a pinned snapshot Sep 21, 2026

martykulma force-pushed the maz-sink-ceiling-design branch from 166658a to cdc2922 Compare September 21, 2026 13:15

martykulma requested review from patrickwwbutler, peterdukelarsen and ublubu September 21, 2026 13:44

martykulma marked this pull request as ready for review September 21, 2026 13:52

martykulma force-pushed the maz-sink-ceiling-design branch 4 times, most recently from 6788ddb to 9e1a923 Compare September 23, 2026 11:05

ublubu reviewed Sep 23, 2026

View reviewed changes

martykulma force-pushed the maz-sink-ceiling-design branch 2 times, most recently from c88d535 to e8a0278 Compare September 23, 2026 14:41

ublubu approved these changes Sep 23, 2026

View reviewed changes

martykulma force-pushed the maz-sink-ceiling-design branch from e8a0278 to 396b44f Compare September 24, 2026 11:54

martykulma and others added 2 commits September 24, 2026 08:36

martykulma force-pushed the maz-sink-ceiling-design branch from 396b44f to 5950f6e Compare September 24, 2026 12:36

martykulma merged commit c4e13f2 into main Sep 24, 2026
10 checks passed

martykulma deleted the maz-sink-ceiling-design branch September 24, 2026 14:05

	times it was opened for and cannot be finished at an upper below a row it holds. The first is
	times it was opened for and cannot be finished at an upper below a row it holds. The first ceiling is

		MZ time ----c--------------------------+---+----+---→ F jumps to where the source reached
		┼────tail─┼ shard upper waits at c until F = C

Conversation

martykulma commented Sep 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

linear-code Bot commented Sep 21, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ublubu Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ublubu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

martykulma commented Sep 19, 2026 •

edited

Loading

ublubu Sep 23, 2026 •

edited

Loading