Replace CTE with subquery in asset orphanage to support MySQL backends without CTE-in-DML - #72235
Open
saitejabandaru-in wants to merge 1 commit into
Open
Replace CTE with subquery in asset orphanage to support MySQL backends without CTE-in-DML#72235saitejabandaru-in wants to merge 1 commit into
saitejabandaru-in wants to merge 1 commit into
Conversation
…s without CTE-in-DML MySQL-compatible backends like Vitess/PlanetScale do not support CTEs in DML statements and reject the query, causing the scheduler to crash-loop. This replaces the CTE calls with subqueries since the CTE was only consumed by a single statement and not semantically required. This unblocks using Airflow on Vitess without affecting Postgres/MySQL semantics.\n\nCloses: apache#72187
SameerMesiah97
left a comment
Contributor
There was a problem hiding this comment.
- This needs a newsfragment filed under 'bugfix'
- Could we add a regression test that compiles the resulting DML statements using the MySQL dialect and verifies they do not contain a WITH clause? Otherwise, tests running against other backends may not catch a regression to the unsupported CTE-in-DML form.
|
Hi @saitejabandaru-in, thanks for looking into this! Just a heads-up that this issue is already being addressed by #72188, which was opened earlier for the same issue (#72187) and takes the same approach (replacing |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #72187.
Motivation
MySQL-compatible backends like Vitess/PlanetScale do not support CTEs in DML statements. When
SchedulerJobRunner._update_asset_orphanageruns its asset reference-count query as a CTE and executes aDELETE, the query planner rejects it. Because this runs on the parsing cleanup interval, it causes the scheduler to crash-loop.Changes
.cte()with.subquery()for bothorphan_queryandactivate_queryin_update_asset_orphanage.assets_queryin_orphan_unreferenced_assetsand_activate_referenced_assetsfromCTEtoSubquery.Since each query is consumed by a single statement, the CTE is not semantically required and a standard derived-table subquery achieves the same outcome. This unblocks deployments on Vitess without changing query semantics for Postgres or standard MySQL.