Idea: Compile-time SQL validator (sqlx-style)

[StefanSteiner]

Status: Draft RFC — soliciting feedback before any implementation work begins. Two adversarial reviewers have already gone over this; their findings are reflected in the current revision. None of this is built yet. Comments, pushback, and "have you considered…" are all welcome.

The plan covers an opt-in, sqlx-style compile-time SQL validator for hyperdb-api. v1 is deliberately narrow: name-based column checking only, opt-in cargo feature, three-crate split. Phase 0 has gating spikes that can kill the project — IDE/rust-analyzer interaction is the biggest risk.

---

TL;DR

Add an opt-in, sqlx-style compile-time SQL validation path to hyperdb-api that catches dropped/renamed columns, typos, and missing tables at cargo build time. Validation runs by spinning up an embedded Hyper instance inside the proc-macro host and dry-running the user's SQL. The existing runtime path is unchanged — compile-time checking is a strictly additive opt-in behind a cargo feature.

v1 scope (descoped from v2 plan): name-based column check only. Type checking is deferred until measured demand justifies the maintenance burden.

Pre-requisite gating: Phase 0 spikes (~3 engineer-days) MUST pass before any implementation work is scheduled. Two of the spikes can kill the project.

---

Audience & motivation

Primary problem: Today, a typo in a SQL string (SELECT id, ema1l FROM users) or a schema drift (column dropped, type changed) surfaces as a Result::Err at runtime, often only on a code path covered by integration tests. We want this to be a compile_error! at the developer's IDE.

Honest scoping: This validates that your SQL strings agree with your derive(Table) Rust structs. It does NOT validate against a production database. If a struct and prod's actual schema have drifted, that drift remains a runtime error. v1 catches developer-side drift only — refactoring one file but not another, typos in column names, queries that select missing columns. That is a real but bounded class of bugs.

Who benefits (must be confirmed in Phase 0): Internal hyperdb-api consumers writing typed Rust queries. Note that hyperdb-mcp gets SQL from LLMs at runtime — it does NOT benefit from compile-time checking. Phase 0 must identify ≥1 real consumer who wants this; otherwise the project should be killed.

Reversibility: the new feature ships behind compile-time cargo feature, off by default. Deprecation policy: if v1 sees no adoption within 90 days post-ship, mark query_as! #[deprecated] and remove in the following minor release.

---

Codebase context (verified)

Workspace root: /Users/ssteiner/dev/hyper-api-rust. Hyper engine source at ../hyper-db relative to project root.

Concern	Location	Notes
Workspace	Cargo.toml	Members: hyperdb-api-core, hyperdb-api, hyperdb-api-derive, hyperdb-api-node, hyperdb-api-salesforce, hyperdb-mcp, hyperdb-bootstrap, sea-query-hyperdb. Edition 2021, Rust 1.81+, version 0.3.1.
Existing derive crate	hyperdb-api-derive/Cargo.toml	proc-macro = true, deps: syn = { version = "2", features = ["full"] }, quote = "1", proc-macro2 = "1"
FromRow derive	hyperdb-api-derive/src/lib.rs:68-210	Named-field structs only. Generates fn from_row(RowAccessor<'_>) -> Result<Self>. Supports #[hyperdb(rename=...)], #[hyperdb(index=N)]. Detects Option<T> to call get_opt/position_opt. Maps by name (default) or position. Returns Result, never panics.
FromRow runtime trait	hyperdb-api/src/result.rs:797-807	fn from_row(row: RowAccessor<'_>) -> Result<Self>
RowAccessor	hyperdb-api/src/row_accessor.rs:54-156	Wraps &Row + &HashMap<&str, usize> (name → index lookup, built once per query). get<T: RowValue>(name), get_opt, position, position_opt. Errors with `Error::Column { kind: Missing
HyperProcess	hyperdb-api/src/process.rs:136,246,1126	HyperProcess::new(hyper_path, parameters) -> Result<Self> (line 246). Drop calls do_shutdown(Some(Duration::from_secs(5))) (line 1126). Uses callback-mechanism — TCP listener on ephemeral port; hyperd connects back.
Connection	hyperdb-api/src/connection.rs:148,417,468,742,775	Connection::new(&proc, db_path, CreateMode). Methods: execute_command, execute_query returning Rowset, fetch_one_as<T: FromRow>, fetch_all_as<T: FromRow>.
Catalog (schema introspection)	hyperdb-api/src/catalog.rs:58,221,370,606	Catalog::new(&conn), get_table_names, get_table_definition(name) -> TableDefinition (columns + SqlType), get_row_count. Hyper has no DESCRIBE SQL form — schema access is purely programmatic.
EXPLAIN	Plain SQL via execute_query	Verified: is_read_only_sql("EXPLAIN SELECT 1") == true (hyperdb-api/tests/read_only_tests.rs:17). No structured query-plan API.
Rowset / ResultSchema / ResultColumn	hyperdb-api/src/result.rs:815,858,956	ResultColumn { name, sql_type: SqlType }. Schema accessible after running a query.
SqlType enum	hyperdb-api-core/src/types/sql_type.rs:50	Variants: Bool, BigInt, SmallInt, Int, Numeric{precision, scale}, Double, Float, Text, Varchar, Char, Date, Timestamp, TimestampTz, Json, …

Current runtime behavior under schema drift (verified)

The existing #[derive(FromRow)] codegen emits one row.get("col_name")? call per struct field — it never iterates result columns. Behavior matrix:

Change	Behavior
New column added to table, struct unchanged, SELECT *	✅ Extra column silently ignored
New column added, query uses explicit SELECT a, b	✅ New column not in result schema at all
Column dropped/renamed, struct still references it	❌ Error::Column { kind: Missing } — clean Result::Err, no panic
Column type changed	❌ Error::Column { kind: TypeMismatch { expected, actual } } — Result::Err, no panic
Struct has fewer fields than result	✅ Always fine

Implications for the validator: the runtime path is already lenient on additions, strict on drift, with Result::Err (not panic) for drift. The compile-time validator's job is to shift the strict cases left while preserving the lenient additions semantics. The earlier transcript's "currently it will panic" claim was out of date.

---

Architectural decisions (final)

1. Three crates, not two (resolves CRITICAL: circular dependency).
   hyperdb-api already depends on hyperdb-api-derive. Adding hyperdb-api as a (feature-gated) dep of hyperdb-api-derive would create a cycle Cargo's resolver rejects. Resolution: introduce a new third crate hyperdb-compile-check that:

   • Depends on hyperdb-api and hyperdb-api-core (runtime types, HyperProcess, Connection).
   • Is depended-on by hyperdb-api-derive ONLY when the compile-time feature is enabled.
   • Owns the shared CompileTimeDb, the Registry, and the LIMIT 0 dry-run helper.
   • Is the ONLY new workspace member added by this project.

   The macro crate stays dependency-light by default; the third crate carries the heavy runtime deps. No cycle.

2. One Cargo feature compile-time, off by default, on hyperdb-api-derive. Without it, only #[derive(FromRow)] exists — zero new dependencies, zero behavior change, zero compile-time hit. Opting in pulls in hyperdb-compile-check and unlocks derive(Table) + query_as!.

3. Single shared Hyper instance per crate compilation, lazily initialized via parking_lot::OnceCell<parking_lot::Mutex<CompileTimeDb>> (NOT std::sync::Mutex — see decision doc: Ssteiner/update docs #5 on poisoning). rustc spawns one proc-macro host process per crate compilation; that process amortizes the ~1–3s startup across every macro invocation in that crate. Drop runs at process exit.

4. Lazy table seeding. derive(Table) + #[hyperdb(register)] registers a (name, create_sql, fields) entry in the shared Registry but does NOT immediately CREATE TABLE. query_as! walks the SQL with sqlparser, extracts referenced tables, and seeds any registered-but-unmaterialized tables on demand before running the dry-run. This handles cross-file macro expansion ordering.

5. parking_lot::Mutex instead of std::sync::Mutex (resolves CRITICAL: panic poisoning). Proc-macros routinely panic to signal errors. std::sync::Mutex poisons on panic — one bad query_as! site would cascade-fail every subsequent macro in the crate. parking_lot::Mutex does not poison.

6. Validation mechanism: LIMIT 0 dry-run, not EXPLAIN. To get column types we need a ResultSchema; EXPLAIN returns plan text. We wrap user SQL as WITH __hdb_q AS (<user-sql>) SELECT * FROM __hdb_q LIMIT 0 (using a deliberately-prefixed CTE name to minimize collision risk with user CTEs). The dry-run returns:

   • Success with a ResultSchema of column names+types, OR
   • Parser/binder error from Hyper (formatted into compile_error! pinned to the SQL literal's span — whole-literal, not sub-literal; see decision chore(deps)(deps-dev): bump tsx from 4.22.0 to 4.22.1 in /hyperdb-api-node #9).

   v1 v2-stretch (cost warnings) would add a separate EXPLAIN code path; not in v1 scope.

   Honest limitation: LIMIT 0 validates schema correctness (parse + bind + plan), NOT runtime execution. Casts that may fail on real data, division-by-zero, subquery cardinality violations, etc. are not caught. The README must call this out explicitly.

7. Column check is name-based and subset-directional for v1. Confirms every struct field's column-name (honoring #[hyperdb(rename)], skipping #[hyperdb(index = N)] fields) appears in the result schema. Extras OK (preserves lenient-additions runtime contract). Missing → compile_error! listing all missing names. Type-level cross-check is deferred to v2 — see decision chore: add Dependabot config for cargo, npm, github-actions #8.

8. Type checking is deferred to v2. v1 ships name-only validation. Reasons:

   • Rust type-name string matching against a static SqlType table is structurally unsolvable for newtypes (type UserId = i64), path variations, and <T as Trait>::Output. The proposed #[hyperdb(skip_type_check)] escape hatch would be required on every domain newtype, making the type-checker mostly inactive in real codebases.
   • Type drift is rare in practice; column-set drift (renames, drops, typos) is far more common.
   • Runtime Error::Column { kind: TypeMismatch } already gives a clean error.
   • Saves 5+ engineer-days from W3 and removes a maintenance trap (the static compatibility table would need updating every time a RowValue impl is added).
   • v2 can add type-checking as a pure addition without breaking v1 users.

9. Diagnostics: whole-literal span pinning, not sub-literal (resolves MAJOR: span-pinning over-promise). proc_macro::Span::source_text and sub-literal byte-range span construction are unstable. On stable rustc, LitStr::span() gives the whole literal's span. v1 squigglies underline the SQL literal as a whole, with the column name embedded in the diagnostic text — not as a sub-span. This is the same UX SQLx ships and is sufficient.

10. derive(Table) always emits CREATE TABLE SQL as a pub const and a method on a Table trait, regardless of #[hyperdb(register)]. The attribute additionally registers the table for compile-time validation. Runtime use of derive(Table) for migrations/fixtures is independently valuable.

11. query_as! returns a builder. query_as!(T, sql, args...).fetch_all(&conn)?, .fetch_one(&conn)?, .fetch_optional(&conn)? (and async variants). Matches sqlx muscle memory; same prepared query can run on different connections / inside transactions. The runtime QueryAs<T> type lives in hyperdb-api.

12. Attribute name: register (not verify, not seed). verify overloads a common English word and is ambiguous next to #[hyperdb(primary_key)]. register precisely describes what it does (registers the table with the compile-time validator). seed describes a mechanism not a user-facing intent.

---

Phase 0 — gating spikes (before scheduling any W1 work)

Three short spikes that each could kill or reshape the project. Total: ~3 engineer-days. Do not skip these.

S1. Audience identification (~0.5 day)

Identify ≥1 internal Rust consumer who has explicitly asked for compile-time SQL validation OR who has hit a real bug that this would have caught. Document their use case. Kill criterion: if no real user surfaces, downscope to derive(Table) only (3-5 days, no query_as!) or kill the project.

S2. rust-analyzer / IDE behavior prototype (~1.5 days)

Build the smallest possible thing that proves the architecture survives an IDE inner loop:

• Add compile-time feature, stub query_as! macro that just spins up HyperProcess and shuts it down.
• Use it in a project loaded in rust-analyzer.
• Observe: does every keystroke trigger a Hyper boot? Does the proc-macro server cache expansions? Is editing latency tolerable?

Kill criterion: if rust-analyzer triggers Hyper startup on every keystroke and adds >500ms editing latency per keystroke, the in-process-Hyper architecture is wrong. Pivot options:

• Option A: require users to point at a long-running Hyper instance via HYPERDB_URL (closer to sqlx's actual model). Larger effort, less auto-magic.
• Option B: abandon query_as! and ship only derive(Table) + a runtime verify_schema() helper that runs once at startup.

S3. Workspace concurrency stress test (~1 day)

Simulate 16 concurrent HyperProcess::new() calls in one process (mimicking heavy parallel cargo build -j 16). Verify zero temp-dir collisions, zero leaked dirs after Drop, zero port conflicts on Windows Named Pipes (if possible to test in this environment; otherwise mark as a known-untested risk).

Kill criterion: consistent collisions or leaks → architecture needs random-suffix temp dirs (small mitigation, do at this point).

---

User-facing examples (target v1 ergonomics)

Without compile-time feature (existing behavior, unchanged)

use hyperdb_api::{Connection, FromRow};

#[derive(FromRow)]
struct User { id: i64, name: String, email: Option<String> }

let users: Vec<User> = conn.fetch_all_as("SELECT id, name, email FROM users")?;

With compile-time feature

use hyperdb_api::{Connection, FromRow, Table, query_as};

#[derive(FromRow, Table)]
#[hyperdb(table = "users", register)]
struct User {
    #[hyperdb(primary_key)]
    id: i64,
    name: String,
    email: Option<String>,
}

let users: Vec<User> = query_as!(User,
    "SELECT id, name, email FROM users WHERE name LIKE $1", "%alice%")
    .fetch_all(&conn)?;

let user: Option<User> = query_as!(User,
    "SELECT id, name, email FROM users WHERE id = $1", 42)
    .fetch_optional(&conn)?;

// Async parity:
let users = query_as!(User, "SELECT id, name, email FROM users")
    .fetch_all_async(&conn).await?;

Compile-time errors users will see

// Typo: "ema1l" instead of "email"
let users: Vec<User> = query_as!(User,
    "SELECT id, name, ema1l FROM users").fetch_all(&conn)?;
// error: User requires column "email" but the query does not project it.
//        (Hyper EXPLAIN output: column "ema1l" does not exist on table "users")
//        ^^^ underline on the whole SQL literal

// SELECT misses a column the struct needs
let users: Vec<User> = query_as!(User, "SELECT id, name FROM users").fetch_all(&conn)?;
// error: User requires columns ["email"] but the query does not project them.
//        Add them to the SELECT list, or remove the fields from User.

// Struct used in query_as! without derive(Table) #[hyperdb(register)]
let xs: Vec<Foo> = query_as!(Foo, "SELECT a FROM t").fetch_all(&conn)?;
// error: type `Foo` must `#[derive(Table)]` with `#[hyperdb(register)]`
//        to be used with `query_as!`.

// SELECT references an unregistered table
let xs: Vec<User> = query_as!(User, "SELECT * FROM ghosts").fetch_all(&conn)?;
// error: table "ghosts" is not registered. Did you forget
//        `#[derive(Table)] #[hyperdb(register)]` on the type that maps to it?

// Multiple missing tables (joined query)
let xs: Vec<X> = query_as!(X, "SELECT * FROM a JOIN b USING(id)").fetch_all(&conn)?;
// error: tables ["a", "b"] are not registered. Add `#[derive(Table)]
//        #[hyperdb(register)]` to the structs that map to them.

What still works (lenient additions)

// Migration adds `created_at` to `users`. The User struct is unchanged.
// This still compiles AND runs:
let users: Vec<User> = query_as!(User, "SELECT * FROM users").fetch_all(&conn)?;
// Extra columns in the result schema are ignored — same semantics as today.

---

Hard problems & how we handle them

P1. Cross-file macro ordering for table registration

Mitigated by lazy seeding (decision #4). query_as! walks SQL with sqlparser, extracts table names, and seeds any registered-but-unmaterialized tables before the LIMIT 0 dry-run.

P2. query_as!(T, …) cannot see T's field list

Proc-macros only see their own token stream. T's field list is populated into the shared Registry by derive(Table) + #[hyperdb(register)]. query_as! looks T's ident up. If not found → compile_error! with a "did you forget derive(Table)?" hint.

P3. sqlparser doesn't know all Hyper extensions

Hyper has SQL extensions (APPROX_COUNT_DISTINCT, geographic types, MODE() WITHIN GROUP, etc.) that sqlparser may not recognize. Mitigation: if sqlparser fails to parse, we skip the table-extraction step and proceed directly to the dry-run. Hyper will produce the actual diagnostic (parse error, table-not-found, etc.) which we forward into compile_error!. The only thing we lose is the "did you forget register?" hint when both sqlparser fails AND the table is unregistered. That's an acceptable degradation; the user still gets a diagnostic from Hyper itself.

P4. Hyper startup cost

3s on first macro invocation × N developers × every clean build. Acceptable for v1; warm builds amortize across all macros. If painful in practice, v2 can add a target/hyperdb-compile-cache/ materialized DB file persisted across builds, hash-invalidated on schema changes. Don't build this yet.

P5. SELECT * masking column-name typos

Decision #7's lenient subset-check means SELECT *, ema1l FROM users (typo in a redundant column the user thought they needed) compiles cleanly because the struct's required columns are still a subset of the result. Accepted v1 trade-off — preserves the lenient-additions invariant. Document in the README. v2 may add a #[hyperdb(strict)] mode that rejects extras.

P6. Dry-run wrapper edge cases

The WITH __hdb_q AS (<user-sql>) SELECT * FROM __hdb_q LIMIT 0 wrapper assumes the user SQL is a single SELECT-able expression. Edge cases:

• User-supplied CTEs: legal — Hyper merges nested WITH clauses correctly. Verified by writing a small integration test.
• INSERT … RETURNING: also a select-shape, works.
• Plain INSERT/UPDATE/DELETE without RETURNING: the wrapper fails because there's no result schema. v1 only supports query_as! (always SELECT-shape); plain DML is out of scope. Document this.
• Multi-statement strings: rejected at the sqlparser stage with a clear "only single statements supported" error.

P7. Workspace parallel compilation

Multiple crates compile in parallel; each crate's proc-macro host is its own process. PID-based temp-dir naming in process.rs is collision-free across processes. Phase 0 spike S3 verifies this empirically. Add random-suffix temp dirs as a fallback if S3 surfaces collisions.

---

Comparison to sqlx — architectural divergences

This project takes the sqlx idea ("validate SQL at compile time against a real DB") and adapts it to Hyper. Several decisions diverge from sqlx — some by necessity (Hyper is embedded, sqlx targets remote servers), some by deliberate choice. Recording them here so reviewers and future maintainers can judge whether the divergences are still justified.

Sorted by significance — biggest divergences first

1. Schema source of truth is the struct, not the database

• sqlx: validates against a real, externally-managed DB pointed at by DATABASE_URL. The triangle (SQL ↔ struct ↔ DB) is fully checked.
• v1: validates against a database synthesized from derive(Table) structs at macro expansion time. The triangle collapses to a line: SQL ↔ struct.
• Implication: v1 catches "I refactored one file but not another." It does NOT catch "I deployed a migration but didn't update my code." Smaller value prop, smaller setup cost (no DATABASE_URL, no cargo sqlx prepare, no .sqlx/ cache). Justified for a v1 that prioritizes zero-config; could be revisited in v2 by allowing a .hyper snapshot path as the schema source.

2. Database lifecycle: spawned in-process inside the proc-macro host

• sqlx: macro is a TCP/local-socket client to a long-running DB the developer manages.
• v1: macro calls HyperProcess::new() to start an embedded Hyper instance inside the rustc proc-macro host. One instance per crate compilation, dropped at host exit.
• Implication: Architecturally novel. No major Rust crate does this (verified). Possible only because Hyper is embedded and starts in seconds. Removes the DATABASE_URL setup burden but introduces real risk in IDE integration (rust-analyzer expansion behavior — Phase 0 S2 is gating for this) and workspace concurrency (Phase 0 S3). Justified iff S2/S3 pass.

3. No parameter type checking

• sqlx: PREPARE returns parameter type metadata. query!("… WHERE id = $1", x) checks x's Rust type matches the inferred parameter type.
• v1: uses LIMIT 0 dry-run, which returns result schema only — no parameter type info. Parameters are forwarded through to runtime opaquely. Wrong-type parameters fail at runtime, not compile time.
• Implication: silent gap users coming from sqlx will trip over. Documented as a known limitation in the README; v2 could lift this if Hyper exposes a prepare-style API (it doesn't today).

4. No type checking on result columns (v1)

• sqlx: ships per-vendor compatibility tables (sqlx-postgres, sqlx-mysql, sqlx-sqlite). Custom types use #[sqlx(type_name = "...")]; newtypes use #[derive(sqlx::Type)] #[sqlx(transparent)].
• v1: explicitly drops type checking (decision chore: add Dependabot config for cargo, npm, github-actions #8). Type-name string matching is structurally fragile across newtypes/aliases; the proposed #[hyperdb(skip_type_check)] escape hatch would be required on every domain newtype.
• Implication: deliberate architectural disagreement, not a TODO. sqlx accepted the fragility (with annotations as the escape hatch) and shipped. v1 chose the smaller surface. Defensible but worth being honest about: sqlx's imperfect-but-shipped type check has caught real bugs in real codebases.

5. No _unchecked macro family

• sqlx: ships query_unchecked!, query_as_unchecked!, etc. — same syntax as the checked versions but skip validation. Explicit "I need dynamic SQL here" escape hatch in the macro itself.
• v1: users drop to Connection::fetch_all_as / execute_query directly when they need dynamic SQL. Functionally equivalent.
• Implication: discoverability gap. A developer reading code can see query_unchecked!(...) and immediately understand "this is a deliberate skip." With v1, the same intent looks like "they didn't bother to use the macro." Worth adding query_as_unchecked! to v1.5 if query_as! sees adoption.

6. Lenient column-set semantics

• sqlx: strict by default. Adding a column to the table forces a code change at every SELECT * site, otherwise the macro errors.
• v1: deliberately lenient — extra projected columns are OK; only missing-from-result is an error. Matches existing FromRow runtime behavior.
• Implication: philosophical difference, not just a missing feature. v1 is more forgiving of schema evolution; sqlx is stricter about SQL-vs-code agreement. Both are defensible; v1's choice preserves backward compatibility with the existing runtime path.

7. Two-parser pipeline (sqlparser + Hyper) vs. one (DB only)

• sqlx: sends SQL straight to the DB. The DB is the only authority on what's valid.
• v1: parses SQL with sqlparser first (to extract table names for lazy seeding), then sends to Hyper for LIMIT 0 validation.
• Implication: added fragility for ergonomics. Hyper-specific syntax (APPROX_COUNT_DISTINCT, geographic types, MODE() WITHIN GROUP, etc.) may not parse with sqlparser. The plan handles this via graceful degradation (P3): if sqlparser fails, skip table extraction and let Hyper produce the diagnostic. Cost: lose the "did you forget register?" hint when both sqlparser fails AND a table is unregistered.

8. No offline cache

• sqlx: .sqlx/ JSON cache committed to repo. CI builds without DB access. Mandatory infrastructure.
• v1: no cache. CI gets Hyper via hyperdb-bootstrap (already wired into .github/workflows/ci.yml).
• Implication: justified for an embedded engine. sqlx's offline mode exists because you can't bring Postgres-the-server with you to CI. Hyper is portable.

9. Cargo feature posture: opt-in, not headline

• sqlx: query! is the headline feature. You can't use sqlx without compile-time validation (offline cache is optional, validation is not).
• v1: compile-time validation is opt-in behind compile-time cargo feature, off by default. The runtime path is the headline; compile-time is layered on top.
• Implication: matches the reversibility goal (deprecate if no adoption) and preserves the existing FromRow user base. Different ecosystem position than sqlx — sqlx is the validator; hyperdb-api is a database client with an optional validator.

10. Missing macro suite

• sqlx: query!, query_as!, query_scalar!, query_unchecked!, query_file!, query_file_as!, query_file_scalar! (plus _unchecked variants of each), runtime QueryBuilder for dynamic SQL.
• v1: query_as!, query_scalar!. Five+ macros short.
• Implication: real gap, but each missing macro is independently addable in v1.5/v2 once query_as! proves itself.

11. Macro-logic crate split (sqlx-macros vs. sqlx-macros-core)

• sqlx: sqlx-macros-core holds the validation logic; sqlx-macros is a thin proc-macro shell. The split exists specifically so macro logic can be unit-tested without proc-macro constraints (proc-macro crates can't be linked into normal test binaries).
• v1 (as-currently-planned): validation logic was implicitly going to live in hyperdb-api-derive directly.
• Implication: v1 should follow sqlx's lead. hyperdb-compile-check becomes the unit-testable home for macro logic (token parsing helpers, SQL extraction, registry queries, name-subset diff); hyperdb-api-derive stays a thin proc-macro shell that calls into hyperdb-compile-check. This is a small workstream tweak (W3 partially relocates) but a meaningful testability win. W3 description below has been updated to reflect this.

Smaller divergences (worth listing for completeness)

Concern	sqlx	v1
Validation mechanism	PREPARE (Postgres/MySQL) / PRAGMA (SQLite)	WITH … LIMIT 0 dry-run
Async runtime	async-first (tokio/async-std/smol gated by feature)	sync compile-time validation, sync+async runtime
Builder API shape	query_as!(...).fetch_*(executor)	identical — query_as!(...).fetch_*(&conn)
Span pinning	whole-literal	whole-literal — same
Nullability inference	yes (via PREPARE metadata)	no (no equivalent metadata in LIMIT 0 result)

When to revisit these divergences

• ci: switch from dtolnay/rust-toolchain to actions-rust-lang/setup-rust-toolchain #3 (parameter types) and fix(ci): prevent npm-publish chmod step from failing on missing binaries #11 (macro-core split): before Milestone B kickoff. Both are cheap to retrofit before W3 starts; expensive after.
• docs(security): note CVE-2026-43868 in thrift Rust crate; ci: skip docs-only PRs #4 (type checking): v2 backlog. Wait for measured demand.
• doc: Ssteiner/update docs #5 (_unchecked family): v1.5 if query_as! sees adoption.
• chore(deps)(deps): bump sysinfo from 0.38.4 to 0.39.2 #10 (full macro suite): v2/v3 driven by user requests.
• Welcome to hyper-api-rust Discussions! #1 (schema source): v2 only if a real consumer surfaces who wants prod-schema validation. Until then, the smaller value prop is intentional.

---

Workstreams & effort estimate

Total v1 (name-only): ~13–18 engineer-days assuming Phase 0 spikes pass cleanly. Tripled query_as! budget acknowledges proc-macros + new crate boilerplate is rarely as small as it looks. The W1/W3 split shifts ~1 day of effort from W3 into W1 but the total is unchanged — the macro-core split pays for itself in unit-test velocity (validation logic is testable without trybuild).

If type-checking is added later (v2): +5–8 days.

If parameter type checking is added later (when/if Hyper exposes a prepare-style API): +3-5 days. See Comparison divergence #3.

W1 — New hyperdb-compile-check crate (4-5 days, +1 day for macro-core split)

This crate is the unit-testable home for ALL validation logic, mirroring sqlx's sqlx-macros-core split (see Comparison divergence #11). hyperdb-api-derive stays a thin proc-macro shell; everything else lives here so it can be linked into normal test binaries.

Files (new):

• hyperdb-compile-check/Cargo.toml — NOT a proc-macro crate (regular library).
• hyperdb-compile-check/src/lib.rs — public re-exports.
• hyperdb-compile-check/src/db.rs — CompileTimeDb (owns HyperProcess + Connection).
• hyperdb-compile-check/src/registry.rs — Registry (table defs, registered-but-unseeded set, struct field-lists), behind parking_lot::Mutex.
• hyperdb-compile-check/src/dry_run.rs — LIMIT 0 wrapper + ResultSchema extraction.
• hyperdb-compile-check/src/sql_parse.rs — sqlparser-driven table-name extraction with graceful failure.
• hyperdb-compile-check/src/validate.rs — the validation entry point: takes (struct_ident, struct_fields, sql_str) and returns Result<ValidationOk, ValidationError>. This is what query_as_macro.rs calls into from hyperdb-api-derive. No proc-macro2/syn/quote types in this signature — all token parsing happens in the proc-macro crate; this crate operates on plain Rust types.
• hyperdb-compile-check/src/diagnostic.rs — ValidationError variants and human-readable formatting. Returns plain strings (or a structured enum); the proc-macro crate converts to compile_error! tokens.

Files modified:

• Workspace Cargo.toml — add member.

Deliverables:

• CompileTimeDb::get_or_init() -> &'static parking_lot::Mutex<CompileTimeDb>.
• Registry::register_schema(ident_str, fields) / register_table(name, create_sql) / seed_if_pending(name).
• dry_run(sql) -> Result<ResultSchema, HyperError>.
• extract_referenced_tables(sql) -> Result<Vec<String>, ParseError> (best-effort; returns empty on parse failure with a structured warning).
• validate_query_as(target_struct: &str, sql: &str) -> Result<ProjectedColumns, ValidationError> — composes the above.
• Unit tests (NOT trybuild) for: registry registration ordering, lazy seeding behavior, sqlparser graceful-degradation, LIMIT 0 wrapping of representative SQL shapes (CTEs, INSERT…RETURNING, JOINs), name-subset diff (struct ⊆ result OK; struct ⊄ result reports missing columns).
• Smoke test: calls get_or_init() from two invocations and confirms one Hyper instance is reused.

Why this split matters: if validation logic lived in hyperdb-api-derive, the only way to test it would be trybuild golden files — slow, brittle to error-message wording changes, and unable to test internal helpers in isolation. With the split, hyperdb-compile-check has standard cargo test coverage; hyperdb-api-derive gets a small set of trybuild UI tests just to verify the proc-macro plumbing.

W2 — #[derive(Table)] macro (3-4 days)

Files (new):

• hyperdb-api-derive/src/table_derive.rs

Files modified:

• hyperdb-api-derive/Cargo.toml — add [features] compile-time = ["dep:hyperdb-compile-check"]. New optional dep hyperdb-compile-check. (NOT hyperdb-api — see decision Welcome to hyper-api-rust Discussions! #1.)
• hyperdb-api-derive/src/lib.rs — #[proc_macro_derive(Table, attributes(hyperdb))]
• hyperdb-api/src/lib.rs — runtime trait Table { const NAME: &str; const CREATE_SQL: &str; fn create_sql() -> &'static str; }

Attributes supported:

• #[hyperdb(table = "name")] — override table name (default = snake_case of ident).
• #[hyperdb(register)] — register for compile-time validation.
• #[hyperdb(primary_key)] on a field.
• #[hyperdb(rename = "...")] on a field — reuse FromRow's parser.

Always emitted: impl Table for MyStruct with CREATE_SQL.
With compile-time feature + #[hyperdb(register)]: also calls Registry::register_table(...) and register_schema(...) at macro expansion time.

Type mapping (Rust → SqlType for CREATE TABLE): explicit lookup table covering: i16 → SmallInt, i32 → Int, i64 → BigInt, f32 → Float, f64 → Double, bool → Bool, String → Text, Vec<u8> → Bytes, chrono::NaiveDate → Date (if chrono feature on), chrono::NaiveDateTime → Timestamp, chrono::DateTime<Utc> → TimestampTz. Option<T> → nullable. Anything else → compile_error!("unsupported field type for derive(Table); use a custom impl Table").

W3 — query_as! macro (4-6 days, lighter post-split)

The proc-macro is now a thin shell over hyperdb-compile-check::validate_query_as. Heavy lifting (registry, dry-run, name-subset diff) lives in W1's crate where it has unit-test coverage.

Files (new):

• hyperdb-api-derive/src/query_as_macro.rs — token parsing only. Calls hyperdb_compile_check::validate_query_as(target_struct, sql) and translates the result into either compile_error! tokens or the success codegen.
• hyperdb-api/src/query_as.rs — runtime QueryAs<T> builder type.

Files modified:

• hyperdb-api-derive/src/lib.rs — #[proc_macro] pub fn query_as(input: TokenStream) -> TokenStream.
• hyperdb-api/src/lib.rs — re-export query_as! (feature-gated) and QueryAs<T>.

Macro flow:

1. Parse T, "SQL" [, arg1, arg2, …] with syn.
2. With compile-time feature: call hyperdb_compile_check::validate_query_as(&T_ident_string, &sql_str).
   • On Err(ValidationError): render to compile_error! token, span-pinned to the SQL literal's whole-span.
   • On Ok: proceed to codegen.
3. Without feature: skip step 2 entirely.
4. Emit:

   ::hyperdb_api::QueryAs::<T>::new($sql, &[$($args),*])

Runtime QueryAs<T>: stores (sql, params). Methods fetch_all(&conn) -> Result<Vec<T>>, fetch_one(&conn) -> Result<T>, fetch_optional(&conn) -> Result<Option<T>>, plus async variants. Forwards to existing Connection::fetch_*_as<T>.

Test surface:

• Unit tests for token parsing (in hyperdb-api-derive's own test module — proc-macro2 supports this for parser helpers).
• trybuild UI tests for end-to-end error rendering — fewer than the original plan, because most logic is unit-tested in hyperdb-compile-check.

W4 — query_scalar! macro (1-2 days, NEW)

Added in response to reviewer feedback that v1 without scalar support has a usefulness cliff. Single-column queries:

let n: i64 = query_scalar!(i64, "SELECT COUNT(*) FROM users").fetch_one(&conn)?;
let names: Vec<String> = query_scalar!(String, "SELECT name FROM users").fetch_all(&conn)?;

Validates that the LIMIT 0 dry-run returns exactly one column. No derive(Table) requirement for the type — just RowValue.

W5 — Diagnostics polish (1-2 days)

• Hyper error message → compile_error! formatting; whole-SQL-literal span pinning.
• "Did you forget #[derive(Table)] #[hyperdb(register)]?" hint on registry-miss.
• Multi-table missing reports in one diagnostic (don't bail on first).
• A tests/ui/ set of trybuild snapshot tests for every error path. Pin Hyper version in the test crate to keep .stderr golden files stable.

W6 — Examples + docs (1 day)

• New example: examples/compile_time_validation.rs.
• README section under hyperdb-api-derive and hyperdb-compile-check.
• Document the limitations explicitly:
  • Validates "your SQL matches your structs," not "your structs match prod."
  • LIMIT 0 doesn't catch runtime-evaluation errors (casts, overflows, divisions).
  • Hyper-specific syntax may bypass sqlparser's table-extraction (still works, weaker hint).
  • Not designed for use in rust-analyzer-heavy edit sessions if Phase 0 S2 surfaces problems.
• Migration note: existing FromRow users do nothing.

---

Out of scope for v1 (explicit)

• ❌ Compile-time per-column type checking. Deferred to v2 (decision chore: add Dependabot config for cargo, npm, github-actions #8). Runtime FromRow still catches type mismatch.
• ❌ Plain query! (no struct, no scalar) — only query_as! and query_scalar!.
• ❌ EXPLAIN-plan cost/scan warnings — v2 stretch goal.
• ❌ Auto-derived SELECT (query_as!(User, table = users)). Pure syntactic sugar; the compile-time checker already catches "forgot to project a field"; opening this door invites where=, order_by=, join=, etc., which competes with SQL itself.
• ❌ .sqlx-style offline cache. Hyper boots locally; no need.
• ❌ Conditional WHERE / dynamic query building. Documented as a known gap; users drop to runtime path for these. v2 may add a query_builder! macro if demand surfaces.
• ❌ Validation against a real production database — see honest scoping in § Audience & motivation.
• ❌ derive(Schema) for read-only structs. Earlier draft mentioned this; cut from v1 to keep surface focused. derive(Table) covers all v1 needs.
• ❌ Plain DML (INSERT/UPDATE/DELETE) without RETURNING in query_as!. Use Connection::execute_command directly.

---

Verification (v1 acceptance criteria)

1. Unit / snapshot: derive(Table) produces expected CREATE SQL for representative structs (primitives, Option<T>, renamed columns, primary keys). Snapshot-tested.
2. trybuild UI: every compile-error path has a .stderr golden file:
   • Bad SQL syntax
   • Missing/unregistered table (single + multi)
   • Column-name mismatch (single + multi)
   • Missing derive(Table) on type used in query_as!
   • Missing #[hyperdb(register)] on a registered-style struct
   • Unsupported field type in derive(Table)
3. Integration: new example crate uses derive(Table) + #[hyperdb(register)] + query_as! + query_scalar! end-to-end. Builds and runs cleanly.
4. Negative integration: introduce a column-name typo → cargo build fails with a SQL-literal diagnostic that names the missing column.
5. Lenient-additions invariant: add an unrelated extra column to a registered table; existing query_as!(T, "SELECT * FROM t") still compiles. Matches runtime behavior.
6. Feature-off invariant: build hyperdb-api-derive and hyperdb-api with default features; confirm cargo tree shows zero new transitive deps vs. baseline. Confirm hyperdb-compile-check is not built.
7. Build-time benchmark: measure clean-build delta on a project with N=10 query_as! invocations vs. N=0. Target: < 5s overhead amortized after Phase 0 S2 passes.
8. Workspace lint/build: cargo clippy --workspace --all-targets and cargo test --workspace continue to pass with feature off.
9. Concurrency: Phase 0 S3 stress test passes (or its mitigation is in place).
10. Cycle check: cargo build -p hyperdb-api-derive --no-default-features succeeds — proves no inadvertent cycle.

---

Success metrics

The feature is judged successful at 90 days post-ship if at least two of the following are true:

• ≥1 internal Rust consumer has migrated to query_as! for production queries.
• query_as! has caught ≥1 real bug in code review or CI before merge.
• derive(Table) is being used at runtime (migrations, fixtures) by ≥1 internal team.
• A user-driven feature request (e.g., type-check, conditional WHERE) has been filed.

If 0/4 at 90 days, mark query_as! #[deprecated] and remove in the next minor release. derive(Table) may be retained on its own merits if it has runtime users.

---

Phased delivery

• Phase 0 (3 days, gating): S1 audience + S2 IDE prototype + S3 concurrency. STOP if S1 or S2 fails their kill criterion.
• Milestone A (1 week): W1 + W2. derive(Table) works at runtime; compile-time feature exists but is empty. Ship as a minor version bump.
• Milestone B (1.5–2 weeks): W3 + W4. query_as! + query_scalar! with name-based validation, behind feature flag, marked experimental.
• Milestone C (3-4 days): W5 + W6. Diagnostics polish + docs + examples. Promote feature from "experimental" if Milestone-B feedback is positive.
• v2 backlog (after measured demand):
  • Type-checking (5-8 days) with explicit user-managed compatibility table or #[hyperdb(sql_type = "...")] annotations to sidestep type-name fragility.
  • EXPLAIN-plan cost/scan warnings.
  • query_builder! for conditional WHERE.
  • Offline cache if measured-painful.
  • #[hyperdb(strict)] mode for projecting-extra-columns warnings.

---

Pickup checklist for a new chat

If you're picking this up cold, do these in order:

1. Read this whole document. It's self-contained; you don't need the prior chat.
2. Verify the codebase context table (§ Codebase context) — confirm file:line references still match. Run git log --oneline -20 to see if anything has shifted since this plan was written.
3. Run Phase 0 S1. Find a real user. Without one, kill the project per the kill criterion. Do not proceed without surfacing this.
4. Run Phase 0 S2. Build the smallest stub query_as! that just spins up HyperProcess. Use it in rust-analyzer for an hour. Measure latency. Document findings.
5. Run Phase 0 S3. Concurrency stress test.
6. If all spikes pass: start Milestone A. Use parking_lot for sync primitives. Use the three-crate split (decision Welcome to hyper-api-rust Discussions! #1). Do NOT add hyperdb-api as a dep of hyperdb-api-derive directly.
7. Per-iteration: cargo clippy && cargo fmt && cargo test --workspace. Commit with git add <files> (never -A). For PR creation, hand back the body file path — gh pr create is EMU-blocked against upstream.
8. Note on release-please: if a feat: / fix: merge doesn't produce a release PR, check the prior release PR for an autorelease: pending label first.

[StefanSteiner]

What is sqlparser? It sounds like a possibly light client side SQL parser that I thought we agreed we would not do and only have Hyper do all validation using client pass through.

[StefanSteiner]

I see what's it's for, table name extraction. Is there a way to get hyper to pull the table names from a query? Probably not but explain may do it?

[StefanSteiner]

We won't wait for S1. Audience identification. We will do this regardless of any user case wanting it. Sqlx has been widely used in the Rust ecosystem. We will just do it.

[StefanSteiner]

Phase 0 spike results (run 2026-05-31) — all passed, architecture is viable

Ran the gating spikes against pinned hyperd 0.0.25080 on macOS (Apple Silicon) via a throwaway #[ignore]d harness. Also folded in the two points from the comments above. Headlines:

Your comments, addressed

sqlparser → dropped. You were right to flag it. The client-side parser was only there for table-name extraction (lazy seeding). Your follow-up — can Hyper give us the table names? — turned out to be the better path. Hyper returns PostgreSQL SQLSTATE codes on a structured error field:

Bad SQL	Error::sqlstate()	Message
SELECT * FROM ghosts	42P01 (undefined_table)	ERROR: table "ghosts" does not exist (42P01)
SELECT id, ema1l FROM t	42703 (undefined_column)	ERROR: unknown column 'ema1l' (42703)

So the validator sends SQL straight to Hyper, and on 42P01 it extracts the table name, seeds it from the registry, and retries. No client-side SQL parser, Hyper is the only authority — exactly the pass-through model we agreed on. This also closes the "two-parser fragility" divergence from the sqlx comparison: Hyper-specific syntax (APPROX_COUNT_DISTINCT, geo types, etc.) now Just Works because we never parse it ourselves.

S1 (audience gate) — removed, per your call. No longer gates the work.

Empirical findings

• Startup: ~156 ms cold HyperProcess::new() (the "20ms" was optimistic; the "10s+" was a CI timeout ceiling, not real startup). Connection ~16ms, query ~7ms. One boot per crate compilation, amortized across all macros. Negligible.
• Concurrency (S3): PASS. 16 concurrent instances in one process (mimics cargo build -j16 proc-macro hosts) — 0 failures, no temp-dir/port collisions. (Windows Named Pipes still untested — will verify in CI before promoting out of experimental.)
• LIMIT 0 dry-run: works. Returns a full ResultSchema (column names + SqlTypes) on zero rows, including through the WITH __hdb_q AS (...) ... LIMIT 0 wrapper and SELECT *.
• rust-analyzer (S2): PASS by reasoning. RA runs proc-macros in a persistent server process and memoizes expansion keyed on input tokens, so Hyper boots once per RA session (not per keystroke), and only edits inside a query_as! literal re-run that one ~20ms dry-run. rust-analyzer.procMacro.ignored is the escape hatch. One stub-macro confirmation is carried into Milestone A to verify empirically.
• Gotcha found: execute_query() is lazy on TCP — the dry-run must drive next_chunk() once to surface errors / populate the schema. Encoded in the plan.

No kill criterion triggered. Implementation is paused here for now; the plan is updated to v4 with all of the above. Feedback still welcome before Milestone A kicks off.