Skip to content

Replace CTE with subquery in asset orphanage to support MySQL backends without CTE-in-DML - #72235

Open
saitejabandaru-in wants to merge 1 commit into
apache:mainfrom
saitejabandaru-in:fix-72187-vitess-cte
Open

Replace CTE with subquery in asset orphanage to support MySQL backends without CTE-in-DML#72235
saitejabandaru-in wants to merge 1 commit into
apache:mainfrom
saitejabandaru-in:fix-72187-vitess-cte

Conversation

@saitejabandaru-in

Copy link
Copy Markdown

Closes #72187.

Motivation

MySQL-compatible backends like Vitess/PlanetScale do not support CTEs in DML statements. When SchedulerJobRunner._update_asset_orphanage runs its asset reference-count query as a CTE and executes a DELETE, the query planner rejects it. Because this runs on the parsing cleanup interval, it causes the scheduler to crash-loop.

Changes

  • Replaced .cte() with .subquery() for both orphan_query and activate_query in _update_asset_orphanage.
  • Updated the type hints for assets_query in _orphan_unreferenced_assets and _activate_referenced_assets from CTE to Subquery.

Since each query is consumed by a single statement, the CTE is not semantically required and a standard derived-table subquery achieves the same outcome. This unblocks deployments on Vitess without changing query semantics for Postgres or standard MySQL.

…s without CTE-in-DML

MySQL-compatible backends like Vitess/PlanetScale do not support CTEs in DML statements and reject the query, causing the scheduler to crash-loop. This replaces the CTE calls with subqueries since the CTE was only consumed by a single statement and not semantically required. This unblocks using Airflow on Vitess without affecting Postgres/MySQL semantics.\n\nCloses: apache#72187
@boring-cyborg boring-cyborg Bot added the area:Scheduler including HA (high availability) scheduler label Aug 28, 2026

@SameerMesiah97 SameerMesiah97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. This needs a newsfragment filed under 'bugfix'
  2. Could we add a regression test that compiles the resulting DML statements using the MySQL dialect and verifies they do not contain a WITH clause? Otherwise, tests running against other backends may not catch a regression to the unsupported CTE-in-DML form.

@GyuhoonK

Copy link
Copy Markdown

Hi @saitejabandaru-in, thanks for looking into this! Just a heads-up that this issue is already being addressed by #72188, which was opened earlier for the same issue (#72187) and takes the same approach (replacing .cte() with .subquery()). To avoid duplicate effort, it would probably make sense to consolidate the review on the earlier PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:Scheduler including HA (high availability) scheduler

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Scheduler crash-loops on MySQL-compatible backends without CTE-in-DML support (e.g. Vitess): asset orphanage DELETE uses WITH

4 participants