diff --git a/doc/developer/generated/adapter/coord/sequencer.md b/doc/developer/generated/adapter/coord/sequencer.md index 6607f7e85eb7c..857fa7b30b4c0 100644 --- a/doc/developer/generated/adapter/coord/sequencer.md +++ b/doc/developer/generated/adapter/coord/sequencer.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/sequencer.rs -revision: 6eeaca032b +revision: d0e1746c64 --- # adapter::coord::sequencer diff --git a/doc/developer/generated/adapter/coord/sequencer/_module.md b/doc/developer/generated/adapter/coord/sequencer/_module.md index d46d727bc6220..b91193062b126 100644 --- a/doc/developer/generated/adapter/coord/sequencer/_module.md +++ b/doc/developer/generated/adapter/coord/sequencer/_module.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/sequencer.rs -revision: 6eeaca032b +revision: d0e1746c64 --- # adapter::coord::sequencer diff --git a/doc/developer/generated/adapter/coord/sequencer/inner/cluster.md b/doc/developer/generated/adapter/coord/sequencer/inner/cluster.md index ee3bd489afd7b..c9c7d19949378 100644 --- a/doc/developer/generated/adapter/coord/sequencer/inner/cluster.md +++ b/doc/developer/generated/adapter/coord/sequencer/inner/cluster.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/sequencer/inner/cluster.rs -revision: 6eeaca032b +revision: ac496d5de8 --- # adapter::coord::sequencer::inner::cluster @@ -12,7 +12,7 @@ Cluster and replica creation and deletion are applied through the catalog implic For `CREATE CLUSTER` (both managed and unmanaged variants), `scoped_overrides_create_op` is called with `ClusterEvalContext` and `ReplicaEvalContext` values built from plan data and pre-allocated replica IDs, and the resulting `Op::UpdateScopedSystemParameters` is folded into the same catalog transaction that creates the cluster. This makes the committed diff drive the replica-scoped controller push before `create_replica`, so the new cluster's optimizer and each new replica's render-frozen flags take effect before any dataflow renders, rather than falling back to environment-wide values until the next sync tick. `CREATE CLUSTER` validates the `HYDRATION SIZE` of an `AUTO SCALING STRATEGY` against the allowed replica sizes for the session role, just as `SIZE` itself is validated. This prevents a typo from failing silently at burst-arm time and prevents a size-restricted role from bursting at a size it may not `CREATE` with. When `ENABLE_CLUSTER_CONTROLLER` is on and the cluster is user-owned, a shape-changing managed-to-managed `ALTER CLUSTER` (one that changes `SIZE`, `AVAILABILITY ZONES`, or `INTROSPECTION`) reshapes into a durable `reconfiguration` record via `reshape_alter_cluster_managed` instead of going through the legacy 3-stage machine. The record carries the full target config shape, a deadline, and an `on_timeout` action (`ROLLBACK` by default). Non-shape fields (`workload_class`, `schedule`, `auto_scaling_strategy`, etc.) are applied to the realized config immediately. The controller converges the replica set onto the target and cuts the realized config over at hydration. `ClusterStage::AwaitReconfiguration` (carrying `AlterClusterAwaitReconfiguration`) is a foreground wait-shim that polls the durable record until it reaches a terminal status, then reports success or failure depending on whether the realized config reached the target; it is used when `ENABLE_BACKGROUND_ALTER_CLUSTER` is off. The legacy foreground `AlterClusterWaitForHydrated` readiness check additionally requires every pending replica to be online (per `mz_cluster_replica_statuses`) before declaring the reconfiguration ready to cut over, so a cluster hosting only single-replica sources (which are never placed on a pending replica until cut-over) does not declare readiness vacuously the moment the pending replicas are created. -When a reconfiguration is in flight, a subsequent shape-changing `ALTER` folds onto the in-flight target via `fold_reconfiguration_target`: dimensions the `ALTER` set explicitly take the new value, while dimensions left `Unchanged` keep the in-flight target's value (not the realized config's value, which would silently revert the in-flight transition). An `ALTER` back to the realized shape cancels the record. Changing the replication factor while a reconfiguration is in flight is refused with `AdapterError::AlterClusterReplicationFactorWhileReconfiguring`. +When a reconfiguration is in flight, a subsequent shape-changing `ALTER` folds onto the in-flight target via `fold_reconfiguration_target`: dimensions the `ALTER` set explicitly take the new value, while dimensions left `Unchanged` keep the in-flight target's value (not the realized config's value, which would silently revert the in-flight transition). The `deadline` and `on_timeout` are handled separately: when the folding `ALTER` carries an explicit `WAIT` clause, the clause's deadline and action overwrite the in-flight record's (allowing a later `ALTER` to steer the deadline without discarding hydration progress already made on the target); when the folding `ALTER` carries no `WAIT` clause, the in-flight record's `deadline` and `on_timeout` are preserved so that an unrelated shape-changing `ALTER` does not silently reset the user-set deadline. When there is no reconfiguration in flight and no `WAIT` clause, the system-default timeout and implicit `ROLLBACK` action apply. An `ALTER` back to the realized shape cancels the record. Changing the replication factor while a reconfiguration is in flight is refused with `AdapterError::AlterClusterReplicationFactorWhileReconfiguring`. `ALTER CLUSTER` validates a newly set or changed `AUTO SCALING STRATEGY`'s `HYDRATION SIZE` like `SIZE` itself: it must name a real replica size the session role may use. Only a changed strategy is checked, so an existing policy does not block unrelated `ALTER`s if the size allow-list later shrinks. Additionally, if a reconfiguration is in flight, `HYDRATION SIZE` must differ from the in-flight reconfiguration's target `SIZE`: equality would end the reshape with a no-op burst shape and a stored statement that fails its own re-plan. Unmanaged clusters reject `AUTO SCALING STRATEGY` changes with an error, mirroring how other managed-only options are handled. Converting a cluster to unmanaged while a hydration burst is in flight is refused with `AdapterError::AlterClusterUnmanagedWhileBursting`. The unmanaged variant carries no burst field, so converting would drop the record without a `Finished` audit event and strand the burst replica as an ordinary unmanaged replica nothing ever tears down. `validate_reconfiguration_resource_limits` checks that the transient peak credit rate (realized plus target shapes running simultaneously) and per-cluster replica count fit the environment-wide budget before the record is written. `validate_reconfiguration_credit_peak` computes the peak by excluding the reconfiguring cluster's own replicas from the base consumption rate to avoid double-counting an already-materialized overlap. diff --git a/doc/developer/generated/adapter/flags.md b/doc/developer/generated/adapter/flags.md index 5d3f28c327753..e2f4689bc2be8 100644 --- a/doc/developer/generated/adapter/flags.md +++ b/doc/developer/generated/adapter/flags.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/flags.rs -revision: 6c2b81feaf +revision: 1d07b38a9c --- # adapter::flags diff --git a/doc/developer/generated/environmentd/http/_module.md b/doc/developer/generated/environmentd/http/_module.md index cb4aa606ddb6b..ab7ecd41a7b2a 100644 --- a/doc/developer/generated/environmentd/http/_module.md +++ b/doc/developer/generated/environmentd/http/_module.md @@ -1,6 +1,6 @@ --- source: src/environmentd/src/http.rs -revision: 59b1f165b2 +revision: 53e55ae9df --- # environmentd::http @@ -9,6 +9,7 @@ Implements `environmentd`'s embedded HTTP server using Axum. Supports multiple authentication modes (`None`, `Password`, `Frontegg`, `Oidc`) via a layered middleware stack, manages session-based and per-request authentication (including WebSocket credential exchange), and routes requests across a rich set of endpoints: SQL execution (REST and WebSocket), Prometheus metrics scraping, memory/heap profiling, catalog and coordinator introspection, webhook ingestion, MCP (AI agent interface), audit event injection, console proxy, cluster replica proxying, and internal deployment management. Key types include `HttpServer`, `HttpConfig`, `AuthedClient`, `Metrics`, and `InternalRouteConfig`. `HttpConfig` carries `allowed_origin` (the `AllowOrigin` predicate for the CORS layer), `allowed_origin_list` (the raw `Vec` injected as an Axum `Extension` into the MCP router for server-side origin validation against DNS rebinding attacks), `frontegg_oauth_issuer_url` (the optional Frontegg workspace URL advertised as the authorization server in MCP OAuth discovery), `dyncfgs` (the process-wide `Arc` threaded into `WebhookState` for per-request dyncfg reads), `mcp_metrics` (a `McpMetrics` handle injected as an Axum `Extension` into the MCP router for Prometheus metrics tracking), `oauth_metadata_metrics` (an `OauthMetadataMetrics` handle for Prometheus metrics on OAuth metadata endpoints), `routes_enabled` (the `HttpRoutesEnabled` value that governs which route groups are active and which `AllowedRoles` each group enforces). Role policy lives per route group via `RouteGroup` values in `HttpRoutesEnabled`, not at the listener level. The MCP CORS layer restricts allowed methods to POST and allowed headers to `Authorization` and `Content-Type`. +When the listener authenticator is `None`, the MCP router applies `x-materialize-user` header authentication, matching the behavior of `base_router` and `ws_router` on unauthenticated listeners. `WebhookState` carries `dyncfgs: Arc` so the webhook handler can read the `WEBHOOK_MAX_REQUEST_SIZE_BYTES` limit from the live config without a coordinator round-trip. `AuthError` enumerates authentication failure modes; the `OidcFailed(String)` variant carries the sanitized `OidcError` display string, which is forwarded to the client so the console can surface OIDC sign-in failures on the login page. `group_claim_for` (which resolves the group claim via a coordinator `Command::GetSystemVars` round-trip) is called only when the authenticator is `Frontegg`; for `None`, `Password`, and `OIDC` paths the call is skipped so liveness and readiness probes are not coupled to coordinator health. diff --git a/doc/developer/generated/expr/scalar/func/_module.md b/doc/developer/generated/expr/scalar/func/_module.md index c68b62f0afe13..655cbcaff3b4b 100644 --- a/doc/developer/generated/expr/scalar/func/_module.md +++ b/doc/developer/generated/expr/scalar/func/_module.md @@ -1,6 +1,6 @@ --- source: src/expr/src/scalar/func.rs -revision: d08f8f74a0 +revision: e5dab9cc27 --- # mz-expr::scalar::func diff --git a/doc/developer/generated/mysql-util/_crate.md b/doc/developer/generated/mysql-util/_crate.md index ae7e2b151bca3..f1fe46f7e8de2 100644 --- a/doc/developer/generated/mysql-util/_crate.md +++ b/doc/developer/generated/mysql-util/_crate.md @@ -1,6 +1,6 @@ --- source: src/mysql-util/src/lib.rs -revision: e757b4d11b +revision: 1d07b38a9c --- # mysql-util diff --git a/doc/developer/generated/mysql-util/tunnel.md b/doc/developer/generated/mysql-util/tunnel.md index 692f006e0a5a0..e604902831819 100644 --- a/doc/developer/generated/mysql-util/tunnel.md +++ b/doc/developer/generated/mysql-util/tunnel.md @@ -1,10 +1,10 @@ --- source: src/mysql-util/src/tunnel.rs -revision: 2a6ac3ab4c +revision: 1d07b38a9c --- # mysql-util::tunnel Implements MySQL connection establishment with support for three transport modes via `TunnelConfig`: direct TCP (with optional pre-resolved IPs), SSH tunnel (via `SshTunnelManager`), and AWS PrivateLink. -Defines `Config` (wraps `mysql_async::Opts`, tunnel, SSH and timeout config, and optional AWS config), `MySqlConn` (a `Conn` wrapper that keeps the SSH tunnel handle alive), and `TimeoutConfig` (snapshot lock/execution timeouts, TCP keepalive, and connect timeout). +Defines `Config` (wraps `mysql_async::Opts`, tunnel, SSH and timeout config, and optional AWS config), `MySqlConn` (a `Conn` wrapper that keeps the SSH tunnel handle alive), and `TimeoutConfig` (snapshot lock/execution/wait timeouts, TCP keepalive, and connect timeout). The `Config::connect` method handles RDS IAM token injection, TLS hostname overriding through tunnels, and a configurable connection timeout. diff --git a/doc/developer/generated/mz-debug/internal_http_dumper.md b/doc/developer/generated/mz-debug/internal_http_dumper.md index 3345a4c25dcbf..cfd9bd6cea0a9 100644 --- a/doc/developer/generated/mz-debug/internal_http_dumper.md +++ b/doc/developer/generated/mz-debug/internal_http_dumper.md @@ -1,12 +1,12 @@ --- source: src/mz-debug/src/internal_http_dumper.rs -revision: dc4dcf22d7 +revision: 8b438b66ce --- # mz-debug::internal_http_dumper Implements `HttpDumpClient` and top-level orchestration functions `dump_emulator_http_resources` and `dump_self_managed_http_resources` for collecting heap profiles, CPU profiles, and Prometheus metrics from running Materialize processes. `HttpDumpClient` streams HTTP responses directly to disk and handles both HTTPS and HTTP with a fallback, as well as optional Basic authentication. -For self-managed (Kubernetes) environments, the module sets up per-service `kubectl` port-forwards to reach internal and external HTTP endpoints for each `environmentd` and `clusterd` instance; for emulator environments it connects directly to the container IP. +For self-managed (Kubernetes) environments, `dump_self_managed_http_resources` expands each cluster service into its backing pods via `find_service_pods`, then scrapes heap profiles and Prometheus metrics per pod using individual `kubectl port-forward` connections to each pod (not the service). This ensures that every process in a scaled multi-pod replica is reached. A failure on one pod is logged and skipped rather than aborting the dump. CPU profiles are captured per-pod in parallel afterwards. For emulator environments, `dump_emulator_http_resources` connects directly to the container IP. Port selection logic is governed by `AuthMode`: unauthenticated deployments use the internal HTTP port (6878) for all data, while password-authenticated `environmentd` uses the external port (6877) for heap and CPU profiles. CPU profiles are captured via a POST to each service's CPU profile endpoint after heap profiling completes; the server temporarily disables memory profiling during capture and restores it afterwards. The helper `dump_cpu_profile_and_verify_memory` probes the profiling mode endpoint to check CPU profiling support and verify memory profiling is active again after the capture. diff --git a/doc/developer/generated/mz-debug/kubectl_port_forwarder.md b/doc/developer/generated/mz-debug/kubectl_port_forwarder.md index 2157f2cb93dfe..7dd5673fca2a5 100644 --- a/doc/developer/generated/mz-debug/kubectl_port_forwarder.md +++ b/doc/developer/generated/mz-debug/kubectl_port_forwarder.md @@ -1,10 +1,11 @@ --- source: src/mz-debug/src/kubectl_port_forwarder.rs -revision: 047f99dfe6 +revision: 8b438b66ce --- # mz-debug::kubectl_port_forwarder -Provides `KubectlPortForwarder` and the resulting `PortForwardConnection` token for establishing `kubectl port-forward` sessions to Kubernetes services. +Provides `KubectlPortForwarder` and the resulting `PortForwardConnection` token for establishing `kubectl port-forward` sessions to Kubernetes resources. +`KubectlPortForwarder` targets either a `PortForwardTarget::Service` or a `PortForwardTarget::Pod`. Forwarding a service lets Kubernetes route to any backing pod (suitable when any replica will do); forwarding a pod reaches a specific process (required for profiling scaled multi-pod replicas). `spawn_port_forward` launches a `kubectl` subprocess, parses its stdout to detect the assigned local address and port, and returns a `PortForwardConnection` whose drop kills the child process. -Also exposes `find_environmentd_service` and `find_cluster_services`, which query the Kubernetes API to locate the relevant services, and `create_pg_wire_port_forwarder`, which combines these into a ready-to-use forwarder targeting the `sql` port. +Also exposes `find_environmentd_service` and `find_cluster_services`, which query the Kubernetes API to locate the relevant services; `find_service_pods`, which lists the pod names backing a service by its label selector (returns an empty vec for an empty selector, and sorts pod names for deterministic output); and `create_pg_wire_port_forwarder`, which combines these into a ready-to-use forwarder targeting the `sql` port. diff --git a/doc/developer/generated/mz-deploy/project/compiler/typecheck/catalog.md b/doc/developer/generated/mz-deploy/project/compiler/typecheck/catalog.md index 81bc864355311..a24e1de73dec4 100644 --- a/doc/developer/generated/mz-deploy/project/compiler/typecheck/catalog.md +++ b/doc/developer/generated/mz-deploy/project/compiler/typecheck/catalog.md @@ -1,8 +1,12 @@ --- source: src/mz-deploy/src/project/compiler/typecheck/catalog.rs -revision: 74f18a3354 +revision: a9e7b191b3 --- # mz-deploy::project::compiler::typecheck::catalog Catalog-backed runtime typechecking. + +Implements `CatalogRuntime`, an in-memory `SessionCatalog` built from `mz-sql` builtins, used to validate objects without a running Materialize container. Each typecheck run creates a fresh `CatalogRuntime`, populates it with the object's dependencies, then discards it, so state does not leak between validations. + +When seeding builtin types during `CatalogRuntime` construction, each array type back-patches its element type's `array_id` field to point at itself. This mirrors the adapter's bootstrap logic and is required so that name resolution can follow `element_type.array_id` when resolving spelled array types such as `text[]`. The ordering of `BUILTINS::iter()` — element types before the array types that reference them — ensures the element entry is present before the back-patch runs. diff --git a/doc/developer/generated/ore/pool/_module.md b/doc/developer/generated/ore/pool/_module.md index cb2948a302703..09846d808f9a2 100644 --- a/doc/developer/generated/ore/pool/_module.md +++ b/doc/developer/generated/ore/pool/_module.md @@ -1,6 +1,6 @@ --- source: src/ore/src/pool.rs -revision: f2082d0163 +revision: 4d6c73f6fa --- # mz-ore::pool diff --git a/doc/developer/generated/pgrepr/value/_module.md b/doc/developer/generated/pgrepr/value/_module.md index 4bee3b3a5f2be..e7d3d23b26f41 100644 --- a/doc/developer/generated/pgrepr/value/_module.md +++ b/doc/developer/generated/pgrepr/value/_module.md @@ -1,6 +1,6 @@ --- source: src/pgrepr/src/value.rs -revision: f147b1341a +revision: aef7da9d85 --- # mz-pgrepr::value @@ -12,6 +12,7 @@ The main entry points are `Value::from_datum` (converting an `mz_repr::Datum` to The helper function `values_from_row` converts a full `mz_repr::RowRef` into a `Vec>` for use by the pgwire protocol layer. Both text-format and binary-format decoding reject NUL (0x00) characters, matching PostgreSQL behavior. In the text path, `reject_nul(s)` is called once on the raw UTF-8 string before dispatching to any type-specific parser. In the binary path, `Text`, `BpChar`, and `VarChar` values go through `decode_binary_string`, which decodes the binary string and then calls `reject_nul`; `Name` values call `reject_nul` directly after decoding. Both helpers return `NulCharacterError` (from the `error` module) on failure. +Binary-format decoding of `Type::Time` reads the wire value as microseconds since midnight (an `i64`) and rejects any value outside `[0, 86_400_000_000)` with `"time out of range"`, rather than delegating to `NaiveTime::from_sql`, which silently wraps out-of-range values around midnight. The child modules supply `ToSql`/`FromSql` implementations for types that require non-trivial encoding logic: diff --git a/doc/developer/generated/pgtz/timezone.md b/doc/developer/generated/pgtz/timezone.md index 98e46d88e0e3d..62fb184f319a6 100644 --- a/doc/developer/generated/pgtz/timezone.md +++ b/doc/developer/generated/pgtz/timezone.md @@ -1,6 +1,6 @@ --- source: src/pgtz/src/timezone.rs -revision: 4267863081 +revision: e5dab9cc27 --- # mz-pgtz::timezone diff --git a/doc/developer/generated/proto/chrono.md b/doc/developer/generated/proto/chrono.md index 79c6da34d1d19..e213b2409828b 100644 --- a/doc/developer/generated/proto/chrono.md +++ b/doc/developer/generated/proto/chrono.md @@ -1,6 +1,6 @@ --- source: src/proto/src/chrono.rs -revision: a6d6c38fee +revision: e5dab9cc27 --- # mz-proto::chrono diff --git a/doc/developer/generated/sql/plan/hir.md b/doc/developer/generated/sql/plan/hir.md index 875bd7c61c2c3..42b50a621d6ac 100644 --- a/doc/developer/generated/sql/plan/hir.md +++ b/doc/developer/generated/sql/plan/hir.md @@ -1,6 +1,6 @@ --- source: src/sql/src/plan/hir.rs -revision: fc2aaf02e7 +revision: 0b5864836a --- # mz-sql::plan::hir @@ -9,5 +9,6 @@ Defines HIR (High-level Intermediate Representation), the plan IR produced by th `HirRelationExpr` and `HirScalarExpr` mirror MIR but additionally allow correlated subqueries, lateral references, and column references with nonzero level (outer-relation columns). Key types include `ColumnRef` (a leveled column reference), `JoinKind` (inner/left/right/full), `AggregateExpr`, `WindowExpr`, and `CoercibleScalarExpr` (a scalar that may still need type coercion). `could_run_expensive_function` detects potentially expensive expressions by checking for unary, binary, variadic, and windowing function calls, `CallTable`/`Reduce` operators, and conservatively returns `true` on `RecursionLimitError`. +`relation_node_count` counts the number of `HirRelationExpr` nodes in an expression tree (including those reachable through scalar subqueries) as a structural size metric for comparing two expressions against each other; scalar nodes themselves are not counted. `HirScalarExpr` implements `VisitChildren` to expose its immediate subquery bodies (`Exists`/`Select` arms) without descending further; this is the asymmetric counterpart of `VisitChildren` on `HirRelationExpr`, which traverses scalars into subqueries at any depth. HIR is converted to MIR via `HirRelationExpr::lower()`, which is defined in `plan::lowering` as an `impl HirRelationExpr` method. diff --git a/doc/developer/generated/sql/plan/plan_utils.md b/doc/developer/generated/sql/plan/plan_utils.md index 570dd88d6f9a4..e4741d8108bd9 100644 --- a/doc/developer/generated/sql/plan/plan_utils.md +++ b/doc/developer/generated/sql/plan/plan_utils.md @@ -1,8 +1,8 @@ --- source: src/sql/src/plan/plan_utils.rs -revision: 844ad57e4b +revision: 42a4392d36 --- # mz-sql::plan::plan_utils -Miscellaneous planning helpers shared across the planner: `maybe_rename_columns` applies user-supplied column aliases to a `RelationDesc`; `JoinSide` and `GroupSizeHints` are small utility types used in join planning and query hints respectively. +Miscellaneous planning helpers shared across the planner: `maybe_rename_columns` applies user-supplied column aliases to a `RelationDesc`, rejecting a list longer than the relation's arity; `maybe_rename_columns_exact` wraps it with an additional check that requires the list length to match exactly when the list is non-empty (unless `unsafe_enable_incomplete_view_column_lists` is set, which is force-enabled during item re-planning to preserve backward compatibility for bootstrap); the private `column_count_mismatch` helper builds the shared error message for both. `JoinSide` and `GroupSizeHints` are small utility types used in join planning and query hints respectively. diff --git a/doc/developer/generated/sql/plan/query.md b/doc/developer/generated/sql/plan/query.md index 2ae3b01a26530..239a7596be610 100644 --- a/doc/developer/generated/sql/plan/query.md +++ b/doc/developer/generated/sql/plan/query.md @@ -1,6 +1,6 @@ --- source: src/sql/src/plan/query.rs -revision: efc8864da8 +revision: 0b5864836a --- # mz-sql::plan::query @@ -14,4 +14,5 @@ The `repeat_row` function (identified via the `is_repeat_row` helper, which matc The private `humanize_or_debug` helper renders a `ResolvedItemName` for use in error messages, falling back to a debug dump if humanization fails. When a `JOIN ... USING (col) AS alias` is planned via `plan_using_constraint`, the expression bound to the aliased column is chosen based on join kind: for `INNER` and `LEFT OUTER` joins it uses the LHS column value; for `RIGHT OUTER` joins it uses the RHS column value; and for `FULL OUTER` joins it uses `COALESCE(lhs, rhs)`, matching the semantics of the unqualified join output column. `plan_nested_query` treats a nested query as an unordered relation: its `ORDER BY` is dropped rather than materialized into a `TopK` unless combined with a `LIMIT`/`OFFSET` clause. This diverges from PostgreSQL, where order-sensitive aggregates observe a sorted subquery's output as an executor artifact. Callers that need a specific aggregation order must use the in-aggregate `agg(value ORDER BY ...)` form. +`INTERSECT` planning calls `HirRelationExpr::relation_node_count` on both inputs and places the smaller one on the left. Because the planner duplicates the left input, a left-deep INTERSECT chain without this swap would produce exponential plan size; with it, plan size is bounded at O(n^log2(3)). `MAX_TYPE_NESTING_DEPTH` (128) and `MAX_TYPE_RESOLUTION_NODES` (100,000) are constants bounding custom type resolution. `scalar_type_from_catalog` delegates to `scalar_type_from_catalog_inner` via `TypeResolutionBudget`, rejecting custom types that are nested deeper than `MAX_TYPE_NESTING_DEPTH` or that require more than `MAX_TYPE_RESOLUTION_NODES` sub-type resolutions, preventing stack overflow and memory exhaustion from pathological types. diff --git a/doc/developer/generated/sql/plan/statement/ddl.md b/doc/developer/generated/sql/plan/statement/ddl.md index 0f2ad623964e3..a9e4d94696bc4 100644 --- a/doc/developer/generated/sql/plan/statement/ddl.md +++ b/doc/developer/generated/sql/plan/statement/ddl.md @@ -1,6 +1,6 @@ --- source: src/sql/src/plan/statement/ddl.rs -revision: 6eeaca032b +revision: 42a4392d36 --- # mz-sql::plan::statement::ddl @@ -8,6 +8,7 @@ revision: 6eeaca032b The largest statement-planning file: implements `describe_*` and `plan_*` for all DDL statements — `CREATE`/`ALTER`/`DROP` for sources, sinks, views, materialized views, tables, indexes, connections, secrets, clusters, roles, schemas, databases, types, network policies, and more. Delegates connection-specific logic to the `connection` submodule. Exports `PgConfigOptionExtracted` and `PlannedRoleAttributes` for use by other modules. +`plan_view` and `plan_create_materialized_view` call `plan_utils::maybe_rename_columns_exact` instead of `maybe_rename_columns`, so a column-name list shorter than the query's arity is rejected unless `unsafe_enable_incomplete_view_column_lists` is active (force-enabled during bootstrap). `plan_create_connection` dispatches on `CreateConnectionType::GlueSchemaRegistry` to plan `CREATE CONNECTION ... FOR AWS GLUE SCHEMA REGISTRY`, guarded by the `ENABLE_GLUE_SCHEMA_REGISTRY` feature flag. `plan_alter_connection` maps `Connection::Gcp(_)` to `CreateConnectionType::Gcp`. Iceberg sink planning validates that key columns are primitive, non-floating-point types using an allow-list; columns outside the list produce a `PlanError::IcebergSinkUnsupportedKeyType`. Iceberg type overrides are applied via `iceberg_type_overrides` from `mz-storage-types`. Range types are permitted as Iceberg sink key columns. diff --git a/doc/developer/generated/sql/plan/statement/ddl/_module.md b/doc/developer/generated/sql/plan/statement/ddl/_module.md index afc146d90506f..74182b2b98dad 100644 --- a/doc/developer/generated/sql/plan/statement/ddl/_module.md +++ b/doc/developer/generated/sql/plan/statement/ddl/_module.md @@ -1,6 +1,6 @@ --- source: src/sql/src/plan/statement/ddl.rs -revision: 6eeaca032b +revision: 42a4392d36 --- # mz-sql::plan::statement::ddl @@ -11,6 +11,7 @@ The `iceberg_sink_builder` function accepts an optional `storage_connection: Opt `REFRESH EVERY` intervals are validated to be at least 1 ms; intervals smaller than 1 ms produce a `PlanError`. `TOPIC METADATA REFRESH INTERVAL` for Kafka sources and sinks is validated to be between 1 second and 1 hour (inclusive); intervals outside this range produce a planning error. `SourceExportStatementDetails::Postgres` carries a `cast_oid_full_range: bool` field; `plan_create_subsource` passes it through to `generate_column_casts` to control whether OID-based casts cover the full range. +`plan_view` and `plan_create_materialized_view` call `plan_utils::maybe_rename_columns_exact` instead of `maybe_rename_columns`, so a column-name list shorter than the query's arity is rejected unless `unsafe_enable_incomplete_view_column_lists` is active (force-enabled during bootstrap). `plan_create_connection` dispatches on `CreateConnectionType::GlueSchemaRegistry` to plan `CREATE CONNECTION ... FOR AWS GLUE SCHEMA REGISTRY`, guarded by the `ENABLE_GLUE_SCHEMA_REGISTRY` feature flag. `plan_alter_connection` maps `Connection::Gcp(_)` to `CreateConnectionType::Gcp`. `AvroSchema::Glue { connection, seed, .. }` in a source format is fully planned: the connection is resolved (must be a `Connection::GlueSchemaRegistry` item) and the seed (populated by purification) is required. In a sink format (`kafka_sink_builder`), `AvroSchema::Glue` is handled separately: it resolves the connection, uses per-side `KEY`/`VALUE` schema name and compatibility-level options (rejecting the source-only `SCHEMA NAME`), generates the Avro schema via `AvroSchemaGenerator`, and produces `WireFormat::Glue { registry: Some(glue_connection_id) }`. diff --git a/doc/developer/generated/sql/session/vars.md b/doc/developer/generated/sql/session/vars.md index 634085e0c07de..07536aba62028 100644 --- a/doc/developer/generated/sql/session/vars.md +++ b/doc/developer/generated/sql/session/vars.md @@ -1,6 +1,6 @@ --- source: src/sql/src/session/vars.rs -revision: e4df9977da +revision: d0e1746c64 --- # mz-sql::session::vars diff --git a/doc/developer/generated/sql/session/vars/_module.md b/doc/developer/generated/sql/session/vars/_module.md index 6538f30dc86ea..82efe0e824ff8 100644 --- a/doc/developer/generated/sql/session/vars/_module.md +++ b/doc/developer/generated/sql/session/vars/_module.md @@ -1,6 +1,6 @@ --- source: src/sql/src/session/vars.rs -revision: e4df9977da +revision: d0e1746c64 --- # mz-sql::session::vars diff --git a/doc/developer/generated/sql/session/vars/definitions.md b/doc/developer/generated/sql/session/vars/definitions.md index 47e449ecb28d6..e1f43c9797919 100644 --- a/doc/developer/generated/sql/session/vars/definitions.md +++ b/doc/developer/generated/sql/session/vars/definitions.md @@ -1,6 +1,6 @@ --- source: src/sql/src/session/vars/definitions.rs -revision: b6b85cfcef +revision: 1d07b38a9c --- # mz-sql::session::vars::definitions @@ -11,5 +11,5 @@ The `feature_flags!` macro accepts an optional `scope: ,` field per flag e The `lazy_value!` and `value!` macros (from `polyfill`) are used extensively to express default values that cannot be computed at compile time. This file is the authoritative source of truth for which variables exist and their defaults. `RESTRICT_TO_USER_OBJECTS` is a read-only `bool` session variable (default `false`) that restricts queries from accessing system catalog objects; it is designed to be set only via `ALTER ROLE ... SET` by superusers. -Feature flags include `enable_repeat_row_non_negative` (guards the `repeat_row_non_negative` table function), `enable_storage_introspection_logs` (guards forwarding storage timely logging events into the compute introspection dataflow), `enable_kafka_broker_matching_rules` (guards `MATCHING` broker rules in `BROKERS` for Kafka PrivateLink connections), `enable_bounded_staleness_isolation` (guards the `bounded staleness ` transaction isolation level; defaults to `true`), `enable_fixed_correlated_cte_lowering` (gates the corrected HIR-to-MIR lowering path that uses CTE-aware branch keys when decorrelating CTEs referenced from nested correlated scopes; defaults to `true`), `enable_statement_arrival_logging` (when enabled, SQL frontends log incoming statements at info level as they arrive, before processing, with literals redacted; intended as an emergency diagnostic for incidents where statements crash the process before reaching the statement log; defaults to `false`), `unsafe_enable_unbounded_custom_type_resolution` (default `false`, `enable_for_item_parsing: true`; when enabled, custom type resolution proceeds without the nesting-depth and total-node limits that ordinarily bound resolution work), and `enable_auto_scaling_strategy` (guards the `AUTO SCALING STRATEGY` cluster option; default `true`, `enable_for_item_parsing: true`; `RESET (AUTO SCALING STRATEGY)` is intentionally not gated by this flag so clusters can shed an autoscaling policy after a flag rollback). +Feature flags include `enable_repeat_row_non_negative` (guards the `repeat_row_non_negative` table function), `enable_storage_introspection_logs` (guards forwarding storage timely logging events into the compute introspection dataflow), `enable_kafka_broker_matching_rules` (guards `MATCHING` broker rules in `BROKERS` for Kafka PrivateLink connections), `enable_bounded_staleness_isolation` (guards the `bounded staleness ` transaction isolation level; defaults to `true`), `enable_fixed_correlated_cte_lowering` (gates the corrected HIR-to-MIR lowering path that uses CTE-aware branch keys when decorrelating CTEs referenced from nested correlated scopes; defaults to `true`), `enable_statement_arrival_logging` (when enabled, SQL frontends log incoming statements at info level as they arrive, before processing, with literals redacted; intended as an emergency diagnostic for incidents where statements crash the process before reaching the statement log; defaults to `false`), `unsafe_enable_unbounded_custom_type_resolution` (default `false`, `enable_for_item_parsing: true`; when enabled, custom type resolution proceeds without the nesting-depth and total-node limits that ordinarily bound resolution work), and `enable_auto_scaling_strategy` (guards the `AUTO SCALING STRATEGY` cluster option; default `true`, `enable_for_item_parsing: true`; `RESET (AUTO SCALING STRATEGY)` is intentionally not gated by this flag so clusters can shed an autoscaling policy after a flag rollback), and `unsafe_enable_incomplete_view_column_lists` (default `false`, `enable_for_item_parsing: true`; when enabled, `CREATE VIEW` and `CREATE MATERIALIZED VIEW` accept a column-name list shorter than the query's arity, applying the names as a prefix rename; this flag is force-enabled during bootstrap so views that an earlier version accepted with incomplete column lists continue to re-plan without crashing environmentd). `DEFAULT_TIMESTAMP_INTERVAL`, `CLUSTER_CHECK_SCHEDULING_POLICIES_INTERVAL`, `SSH_CHECK_INTERVAL`, and `STORAGE_STATISTICS_INTERVAL` each carry a `NON_ZERO_DURATION` constraint, so setting any of them to zero raises an `InvalidParameterValue` error. diff --git a/doc/developer/generated/storage-types/dyncfgs.md b/doc/developer/generated/storage-types/dyncfgs.md index f9003596bde55..abd0d74265ad0 100644 --- a/doc/developer/generated/storage-types/dyncfgs.md +++ b/doc/developer/generated/storage-types/dyncfgs.md @@ -1,6 +1,6 @@ --- source: src/storage-types/src/dyncfgs.rs -revision: fe04b48e6f +revision: 56d7ec66b5 --- # storage-types::dyncfgs @@ -14,7 +14,7 @@ Covers: * **AWS**: `AWS_PREFETCH_STS_CONNECT_TIMEOUT` — connect timeout for the AssumeRole credentials prefetcher's STS calls, defaulting to the SDK's 3.1 second default. Read once when a connection is set up; running sinks are unaffected until their dataflow restarts. * **Networking**: `ENFORCE_EXTERNAL_ADDRESSES` — whether to reject private/local addresses when resolving external connections (defaults `false`). * **Kafka**: `KAFKA_CLIENT_ID_ENRICHMENT_RULES` (JSON array of pattern/payload rules for enriching `client.id`), `KAFKA_POLL_MAX_WAIT`, `KAFKA_LOW_WATERMARK_CHECK` (gates whether Kafka sources check the partition low watermark and error when the start offset/resume upper has been compacted away), `KAFKA_DEFAULT_AWS_PRIVATELINK_ENDPOINT_IDENTIFICATION_ALGORITHM` (defaults to `"none"`), `KAFKA_BUFFERED_EVENT_RESIZE_THRESHOLD_ELEMENTS`, `KAFKA_RETRY_BACKOFF`, `KAFKA_RETRY_BACKOFF_MAX`, `KAFKA_RECONNECT_BACKOFF`, `KAFKA_RECONNECT_BACKOFF_MAX` (defaults 30 s to reduce reconnection churn), `KAFKA_SINK_MESSAGE_MAX_BYTES`, `KAFKA_SINK_BATCH_SIZE`, and `KAFKA_SINK_BATCH_NUM_MESSAGES`. -* **MySQL**: `MYSQL_REPLICATION_HEARTBEAT_INTERVAL` and `MYSQL_SOURCE_SNAPSHOT_PARALLELISM` (gates PK-range splitting of snapshot reads across workers, defaults `true`). +* **MySQL**: `MYSQL_REPLICATION_HEARTBEAT_INTERVAL` and `MYSQL_SOURCE_SNAPSHOT_PARALLELISM` (gates PK-range splitting of snapshot reads across workers, defaults `false`). * **PostgreSQL**: `PG_FETCH_SLOT_RESUME_LSN_INTERVAL`, `PG_SCHEMA_VALIDATION_INTERVAL`, and `PG_SOURCE_VALIDATE_TIMELINE` (whether a timeline switch is treated as a definite error, defaults `true`). * **SQL Server**: `SQL_SERVER_SOURCE_VALIDATE_RESTORE_HISTORY` (whether a restore-history change is treated as a definite error, defaults `true`), plus additional SQL Server configs registered from `crate::sources::sql_server` (`CDC_CLEANUP_CHANGE_TABLE`, `CDC_CLEANUP_CHANGE_TABLE_MAX_DELETES`, `MAX_LSN_WAIT`, `SNAPSHOT_PROGRESS_REPORT_INTERVAL`). * **Upsert**: `STORAGE_UPSERT_PREVENT_SNAPSHOT_BUFFERING` (defaults `true`), `STORAGE_ROCKSDB_USE_MERGE_OPERATOR` (defaults `true`), `STORAGE_UPSERT_MAX_SNAPSHOT_BATCH_BUFFERING`, `ENABLE_UPSERT_PAGED_SPILL` (allows the upsert-v2 stash to spill chunks to the shared buffer pool, defaults `false`), `STORAGE_USE_CONTINUAL_FEEDBACK_UPSERT` (defaults `true`), and `ENABLE_UPSERT_V2` (defaults `false`). diff --git a/doc/developer/generated/storage/source/mysql/snapshot.md b/doc/developer/generated/storage/source/mysql/snapshot.md index bfbe9513f9ff3..738f0d03914ee 100644 --- a/doc/developer/generated/storage/source/mysql/snapshot.md +++ b/doc/developer/generated/storage/source/mysql/snapshot.md @@ -1,6 +1,6 @@ --- source: src/storage/src/source/mysql/snapshot.rs -revision: 8d40d19a29 +revision: 1d07b38a9c --- # mz-storage::source::mysql::snapshot