Skip to content

Support subquery moves across AND/OR/NOT WHERE clauses#4051

Draft
robacourt wants to merge 7 commits intomainfrom
rob/simple-subqueries-with-dnf
Draft

Support subquery moves across AND/OR/NOT WHERE clauses#4051
robacourt wants to merge 7 commits intomainfrom
rob/simple-subqueries-with-dnf

Conversation

@robacourt
Copy link
Copy Markdown
Contributor

@robacourt robacourt commented Mar 24, 2026

Summary

This PR lets shapes with boolean combinations around subqueries stay live when dependency rows move. Previously, shapes that used AND, OR, or NOT around a subquery would return 409 on a move, forcing the client to throw away the shape and resync it from scratch. For large shapes that could mean a very expensive full resync; this PR avoids that by reconciling those changes in-stream.

The core of the PR is a rewrite of the subquery foundation. Move-ins are now handled as exact log splices: buffer outer-table transactions, run the move-in query, and insert its results into the shape log at the precise point where they become valid. That gives subqueries a much more reliable ordering model, removes a large class of move-handling edge cases, and gives us a stable base for richer boolean WHERE support over subqueries.

Compatibility

This PR changes the wire protocol.

The sync service from this branch is not compatible with elixir-client from before this PR. For the TanStack DB client, this requires:

  • @tanstack/db >= 0.6.2
  • @tanstack/electric-db-collection >= 0.3.0

The protocol change is needed because DNF-based subquery shapes need more than tags alone. Rows can now stay in the shape for multiple independent reasons, so the server sends real active_conditions values and position-aware move-in broadcast control messages. That lets clients update the truth value of the affected DNF positions for rows they already have locally, re-evaluate inclusion correctly, and avoid unnecessary deletes or full resyncs.

Architecture

  • compile each shape's WHERE clause into a DNF plan and use that plan consistently for routing, move planning, SQL generation, tags, and active_conditions
  • replace the previous subquery reverse-indexing approach with a per-shape SubqueryIndex that each shape consumer seeds and updates from its own dependency views, keeping the reverse index exactly in sync with consumer state
  • refactor Electric.Shapes.Consumer around explicit event handlers and ordered effects, with a single move queue and explicit buffering/splice handling
  • emit DNF-aware row metadata and position-aware move broadcasts so visibility can change incrementally instead of forcing invalidation

Secondary changes

  • update the materializer and clients to understand DNF-aware tags, active_conditions, and move broadcasts
  • expand coverage across DNF planning, routing, move ordering, resume behavior, and boolean/subquery edge cases
  • update the oracle test harness and generators to cover the new boolean/subquery behavior more accurately

@robacourt robacourt force-pushed the rob/simple-subqueries-with-dnf branch 2 times, most recently from c7da8bb to f33c781 Compare March 25, 2026 14:38
@robacourt robacourt force-pushed the rob/simple-subqueries-with-dnf branch 2 times, most recently from 6669eb8 to a1e0fc1 Compare March 26, 2026 12:38
@robacourt robacourt force-pushed the rob/simple-subqueries-with-dnf branch from 226f9b3 to 203f4fc Compare March 26, 2026 16:51
@robacourt robacourt force-pushed the rob/simple-subqueries-with-dnf branch 6 times, most recently from 67f228e to 438e6cf Compare April 13, 2026 14:22
@robacourt robacourt force-pushed the rob/simple-subqueries-with-dnf branch 3 times, most recently from 66d52fe to fb8a7d9 Compare April 15, 2026 15:40
@electric-sql electric-sql deleted a comment from github-actions bot Apr 15, 2026
@electric-sql electric-sql deleted a comment from github-actions bot Apr 15, 2026
@electric-sql electric-sql deleted a comment from github-actions bot Apr 15, 2026
@electric-sql electric-sql deleted a comment from github-actions bot Apr 15, 2026
@electric-sql electric-sql deleted a comment from netlify bot Apr 15, 2026
@electric-sql electric-sql deleted a comment from github-actions bot Apr 15, 2026
@electric-sql electric-sql deleted a comment from github-actions bot Apr 15, 2026
@electric-sql electric-sql deleted a comment from codecov bot Apr 15, 2026
@codecov
Copy link
Copy Markdown

codecov bot commented Apr 15, 2026

Codecov Report

❌ Patch coverage is 84.44444% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.27%. Comparing base (59a96b8) to head (4208d42).
⚠️ Report is 3 commits behind head on main.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
...kages/elixir-client/lib/electric/client/message.ex 0.00% 7 Missing ⚠️
...s/elixir-client/lib/electric/client/tag_tracker.ex 94.44% 4 Missing ⚠️
packages/elixir-client/lib/electric/client/poll.ex 66.66% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4051      +/-   ##
==========================================
- Coverage   89.20%   86.27%   -2.94%     
==========================================
  Files          25       51      +26     
  Lines        2520     3547    +1027     
  Branches      633      640       +7     
==========================================
+ Hits         2248     3060     +812     
- Misses        270      485     +215     
  Partials        2        2              
Flag Coverage Δ
elixir 79.06% <84.44%> (?)
elixir-client 79.06% <84.44%> (?)
packages/experimental 87.73% <ø> (ø)
packages/react-hooks 86.48% <ø> (ø)
packages/start 82.83% <ø> (ø)
packages/typescript-client 94.30% <ø> (ø)
packages/y-electric 56.05% <ø> (ø)
typescript 89.20% <ø> (ø)
unit-tests 86.27% <84.44%> (-2.94%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@robacourt robacourt changed the title DRAFT: simple subqueries with DNF DRAFT: keep boolean subquery shapes live with DNF move handling Apr 18, 2026
@robacourt robacourt changed the title DRAFT: keep boolean subquery shapes live with DNF move handling Support subquery moves across AND/OR/NOT WHERE clauses Apr 18, 2026
@robacourt robacourt force-pushed the rob/simple-subqueries-with-dnf branch from 413acaf to 4208d42 Compare April 18, 2026 20:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants