Support subquery moves across AND/OR/NOT WHERE clauses#4051
Draft
Support subquery moves across AND/OR/NOT WHERE clauses#4051
Conversation
c7da8bb to
f33c781
Compare
6669eb8 to
a1e0fc1
Compare
226f9b3 to
203f4fc
Compare
67f228e to
438e6cf
Compare
66d52fe to
fb8a7d9
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #4051 +/- ##
==========================================
- Coverage 89.20% 86.27% -2.94%
==========================================
Files 25 51 +26
Lines 2520 3547 +1027
Branches 633 640 +7
==========================================
+ Hits 2248 3060 +812
- Misses 270 485 +215
Partials 2 2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
413acaf to
4208d42
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR lets shapes with boolean combinations around subqueries stay live when dependency rows move. Previously, shapes that used
AND,OR, orNOTaround a subquery would return409on a move, forcing the client to throw away the shape and resync it from scratch. For large shapes that could mean a very expensive full resync; this PR avoids that by reconciling those changes in-stream.The core of the PR is a rewrite of the subquery foundation. Move-ins are now handled as exact log splices: buffer outer-table transactions, run the move-in query, and insert its results into the shape log at the precise point where they become valid. That gives subqueries a much more reliable ordering model, removes a large class of move-handling edge cases, and gives us a stable base for richer boolean
WHEREsupport over subqueries.Compatibility
This PR changes the wire protocol.
The sync service from this branch is not compatible with
elixir-clientfrom before this PR. For the TanStack DB client, this requires:@tanstack/db >= 0.6.2@tanstack/electric-db-collection >= 0.3.0The protocol change is needed because DNF-based subquery shapes need more than tags alone. Rows can now stay in the shape for multiple independent reasons, so the server sends real
active_conditionsvalues and position-awaremove-inbroadcast control messages. That lets clients update the truth value of the affected DNF positions for rows they already have locally, re-evaluate inclusion correctly, and avoid unnecessary deletes or full resyncs.Architecture
WHEREclause into a DNF plan and use that plan consistently for routing, move planning, SQL generation, tags, andactive_conditionsSubqueryIndexthat each shape consumer seeds and updates from its own dependency views, keeping the reverse index exactly in sync with consumer stateElectric.Shapes.Consumeraround explicit event handlers and ordered effects, with a single move queue and explicit buffering/splice handlingSecondary changes
tags,active_conditions, and move broadcasts