sql-499: Retain relations - #38922
Draft
SangJunBak wants to merge 9 commits into
Draft
SangJunBak wants to merge 9 commits into
SangJunBak wants to merge 9 commits into
Conversation
SangJunBak
force-pushed
the
jun/source-references-mv
branch
from
September 21, 2026 17:37
652d3fa to
f146a89
Compare
SangJunBak
force-pushed
the
jun/sql-499-convert-item-metadata-tables-to-catalog-views
branch
from
September 21, 2026 21:13
a064ee0 to
49fd4b7
Compare
SangJunBak
force-pushed
the
jun/source-references-mv
branch
2 times, most recently
from
September 22, 2026 18:04
c7c7db9 to
76e56f5
Compare
We derive global ID mappings from the catalog directly instead of from the in-memory catalog. These include introspection source indexes, builtin objects, and user objects. The only caveat is in `items`, these include runtime alterable builtins. Thus we explictly exclude those in the builtin CTE such that each CTE is distinct and we can UNION ALL rather than UNION.
- We already have tests for checking that the catalog ID -> global ID mapping is consistent in src/adapter/src/catalog/consistency.rs. But we add a new mz_object_global_ids.slt to assert that the SQL query reflects the correct information.
SangJunBak
force-pushed
the
jun/source-references-mv
branch
from
September 23, 2026 13:44
76e56f5 to
2b7fa10
Compare
Add test/sqllogictest/mz_source_references.slt, which pins the schema (column order, types, nullability), the empty relation on a fresh environment, one row per recorded reference with the upstream column order preserved, a reference whose columns read as NULL rather than an empty array, a non-epoch `updated_at`, the FK-like invariant against mz_sources, and retraction on DROP SOURCE. Load generator sources cover the shape without an upstream system; test/testdrive/source-tables.td covers PostgreSQL, MySQL and Kafka along with ALTER SOURCE ... REFRESH REFERENCES. Refresh the goldens that the relation kind change moves: catalog_server_explain.slt gains the materialized view plan and shifts its constant row counts, and information_schema_tables.slt and catalog.td reclassify the relation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Purification already replaces mz_now() in a REFRESH AT time or a REFRESH EVERY ... ALIGNED TO alignment with the literal the statement was planned at, and leaves any other expression as written. Fold the whole expression instead, storing <millis>::mz_timestamp. The stored create_sql is the only durable record of a refresh schedule, and the catalog views over mz_catalog_raw read it back without a planner, so the times have to be literals there. Planning a literal yields the same time, so the fold is idempotent, and an expression that does not fold is left for planning to reject with its usual error. Factor the planning of a refresh time into plan_refresh_time so the purifier and the planner share it. A fold can drop any reference the expression made (the target type of a cast, a function) besides adding mz_timestamp, so the purifier re-resolves the folded statement to recompute resolved_ids, which is what loading it from the catalog does. Add ast_rewrite_fold_refresh_times, a catalog migration that folds the times of materialized views stored before this change. Only name resolution needs the catalog: the expressions reference functions and types, never relations. SHOW CREATE MATERIALIZED VIEW now shows the folded time for a REFRESH AT built from an expression around mz_now(), which materialized_views.slt pins. The RefreshVariants platform check asserts that the schedule of a view created with literal times reads back exactly across upgrades. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ate_sql parse_catalog_create_sql gains the three fields the item-metadata catalog views need. retain_history_millis is the planned compaction window of a table, table from source, source, subsource, index or materialized view, mirroring plan_retain_history_option, with a disabled window reported as u64::MAX. replacement_target is the target of a REPLACEMENT FOR materialized view. refresh holds one entry per REFRESH option, with the EVERY interval rendered through the same duration round trip as planning and the times read from their mz_timestamp literals; a time still stored as an expression reads as NULL rather than failing every catalog view, which a 0dt read-only environment needs while it sees the leader's unmigrated catalog. Move the interval literal conversion from mz_sql::plan::literal into Interval::from_literal in mz_repr, which mz_expr can reach. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
mz_history_retention_strategies, mz_materialized_view_refresh_strategies and mz_replacements become materialized views over mz_catalog_raw, reading the RETAIN HISTORY, REFRESH and REPLACEMENT FOR clauses that parse_catalog_create_sql reports. Their packers go away and each gets a replacement migration step. Builtins have no create_sql. A builtin table, source or index flagged as a retained-metrics object follows the metrics_retention system parameter, read from mz_overridden_system_parameters with the compiled-in default inlined, and every other builtin has the default window. To make the flag visible to SQL, mz_builtin_tables and mz_builtin_sources gain an is_retained_metrics_object column and a generated mz_builtin_indexes view joins them. Builtin materialized views stay out of the retained set: CatalogItem::is_retained_metrics_object is false for materialized views, so the flag is inert for them and the table always reported the default window, which the view keeps. An override of metrics_retention set through --system-parameter-default is not durable and so invisible here; the view's comment records that. Closes: SQL-499 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Add sqllogictests for mz_replacements, mz_history_retention_strategies and mz_materialized_view_refresh_strategies: the schema, one row per RETAIN HISTORY spelling and per REFRESH option, builtin and introspection rows, a metrics_retention override, the folded create_sql, and retraction. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
SangJunBak
force-pushed
the
jun/source-references-mv
branch
from
September 23, 2026 14:00
2b7fa10 to
e84907a
Compare
SangJunBak
changed the title
SangJunBak
force-pushed
the
jun/sql-499-convert-item-metadata-tables-to-catalog-views
branch
from
September 23, 2026 14:10
49fd4b7 to
42cc0ab
Compare
SangJunBak
force-pushed
the
jun/source-references-mv
branch
2 times, most recently
from
September 24, 2026 15:03
1334ed3 to
6bc2133
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Derive the CatalogItemId -> GlobalId mapping from
mz_catalog_rawrather than packing it from the in-memory catalog. Four durable
sources make up the mapping:
Itemrows, whoseglobal_idis the root version's GlobalId. Thesecarry user items, temporary items, and the runtime-alterable
builtins, which are durably recorded in the items collection.
extra_versionson those rows, one GlobalId per later version of atable or materialized view. This is the only case where one
CatalogItemId maps to several GlobalIds.
GidMappingrows for the remaining builtins. Runtime-alterable onesare excluded by their sentinel fingerprint: their
Itemrow alreadycarries the same pair, so keeping both would double-count.
ClusterIntrospectionSourceIndexrows.UNION ALL combines the branches. A GlobalId is allocated once and
belongs to exactly one item, so the branches are disjoint. The desc
declares no key even though
global_idis unique, because a UNION ALLover jsonb extractions does not prove one and
verify_builtin_descscompares the declared desc against what the optimizer infers. The new
sqllogictest file asserts the uniqueness invariant instead.
Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.comRemove these sections if your commit already has a good description!
Motivation
Why does this change exist? Link to a GitHub issue, design doc, Slack
thread, or explain the problem in a sentence or two. A reviewer who has
no context should understand why after reading this section.
If this implements or addresses an existing issue, it's enough to link to that:
Closes
Fixes
etc.
Description
What does this PR actually do? Focus on the approach and any non-obvious
decisions. The diff shows the code --- use this space to explain what the
diff can't tell a reviewer.
Verification
How do you know this change is correct? Describe new or existing automated
tests, or manual steps you took.
Table-to-materialized-view conversions of builtin relations have to produce
byte-identical contents, and nothing checked that. The harness dumps the
configured relations from two fresh environments, one on a baseline image and
one on the locally built code, applies the same corpus of user objects to
both, and diffs the results.
System ids are not stable across builds, so dumps are canonicalized before
diffing: any cell that looks like a catalog id is rewritten to the qualified
name of the object it denotes on that side, resolved through a per-namespace
mapping. Rows that name an object the other side does not have at all are
tolerated; everything else is reported.
The relation set covers every conversion recorded in the MIGRATIONS list of
builtin_schema_migration.rs. Relations the corpus cannot populate without
external systems are empty on both sides, and their entries only validate the
dump machinery against the relation's schema.
Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com