Feature/pg wire extended query support 01

.golangci.yml

@@ -196,6 +196,13 @@ linters:
           - lll
           - revive
         path: mcp_client\/cmd\/.*\.go
+      - linters:
+          - revive
+          - unparam
+        path: internal\/stackql\/driver\/.*\.go
+      - linters:
+          - stylecheck
+        path: internal\/stackql\/queryshape\/.*\.go
       - linters:
           - revive
         path: internal\/stackql\/acid\/tsm_physio\/.*\.go

docs/live_context.md

@@ -0,0 +1,114 @@
+# Live Context: Extended Query Protocol Implementation
+
+## Date: 2026-04-03
+
+## Current State
+
+### What's been done
+
+1. **Indirect joins tests** (complete, merged-ready):
+   - 8 new robot tests covering 3-way and 4-way INNER JOIN + LEFT OUTER JOIN across views, materialized views, subqueries, provider tables
+   - All use `Should Stackql Exec Inline Equal Both Streams` with exact output matching
+   - LEFT OUTER JOIN tests prove NULL behavior with partial matches
+   - `docs/views.md` updated with supported combinations
+
+2. **Extended query protocol stubs** (complete):
+   - `basicStackQLDriver` implements `IExtendedQueryBackend` with compile-time check
+   - `HandleParse`: passthrough (returns client OIDs as-is)
+   - `HandleBind`: no-op
+   - `HandleExecute`: `queryshape.SubstituteParams` replaces `$N` → `HandleSimpleQuery`
+   - `HandleDescribeStatement/Portal`: delegates to `queryshape.Inferrer`
+   - `HandleClose*`: no-ops
+   - 407/407 robot tests pass
+
+3. **`queryshape` package** (complete):
+   - `internal/stackql/queryshape/queryshape.go`
+   - Public `Inferrer` interface, private `standardInferrer` struct
+   - `InferResultColumns(query)` with two paths:
+     - `inferFromStoredRelation`: reads MV/table column metadata from stored DTOs (cheap)
+     - `inferFromPlan`: builds plan via `planbuilder.BuildPlanFromContext` (no execution)
+   - `SubstituteParams`: moved here from driver, replaces `$N` with bound values
+   - `extractSingleTableName`: lightweight sqlparser-based single-table detection
+   - Unit tests in `queryshape_test.go` with JSON testdata
+
+4. **Plan column metadata** (complete):
+   - `plan.Plan` has `GetColumnMetadata()`/`SetColumnMetadata()`
+   - Set in `planbuilder/entrypoint.go` from `GetSelectPreparedStatementCtx().GetNonControlColumns()`
+   - `planGraphBuilder` interface has `getRootPrimitiveGenerator()`
+
+5. **OID fidelity + value coercion** (IN PROGRESS — 10 failures remaining):
+   - Finer OID mapping in `typing/standard_column_metadata.go`:
+     - `getOidForSchema`: `integer`→`T_int8`, `boolean`→`T_bool`, `number`→`T_numeric`
+     - `getOidForParserColType`: split into `T_int2/T_int4/T_int8/T_float4/T_float8/T_json/T_jsonb`
+   - Finer OID mapping in `typing/relayed_column_metadata.go`
+   - Value coercion in `internal/stackql/psqlwire/psqlwire.go`:
+     - `coerceForOID()` function converts string/[]byte from RDBMS to Go types pgtype expects
+     - Applied in `ExtractRowElement` for non-text, non-numeric OIDs
+     - Existing `shimNumericElement` preserved for `"numeric"` type
+   - **397/407 pass, 10 failures remain** — need to check what those 10 are
+
+### What's next (from the plan)
+
+**Immediate**: Fix remaining 10 test failures from OID changes. Check what OID/coercion path they hit.
+
+**Phase 2**: `paramresolver` package
+- Resolve `$N` placeholder OIDs from method schemas during `HandleParse`
+- Add `ParameterOIDs []uint32` to `plan.Plan`
+- Populate during plan building in `entrypoint.go`
+
+**Phase 3**: Stateful driver + `paramdecoder` package
+- `paramdecoder`: decode binary-format params using `jackc/pgtype`
+- Statement/portal caches in `basicStackQLDriver`:
+  - `HandleParse`: resolve OIDs + infer columns → cache in `stmtCache`
+  - `HandleDescribeStatement`: return from cache (no re-planning)
+  - `HandleBind`: record portal→statement mapping
+  - `HandleExecute`: look up portal → decode params → substitute → execute
+  - `HandleClose*`: delete from caches
+
+**Phase 4** (separate, psql-wire repo): Respect `resultFormats` from Bind instead of hardcoding `TextFormat`
+
+### Key files modified
+
+| File | Status | Description |
+|------|--------|-------------|
+| `internal/stackql/driver/driver.go` | Modified | IExtendedQueryBackend impl, shapeInferrer field |
+| `internal/stackql/queryshape/queryshape.go` | New | Inferrer interface, SubstituteParams |
+| `internal/stackql/queryshape/queryshape_test.go` | New | Unit tests with JSON testdata |
+| `internal/stackql/queryshape/testdata/*.json` | New | Test cases |
+| `internal/stackql/psqlwire/psqlwire.go` | Modified | coerceForOID() value coercion |
+| `internal/stackql/typing/standard_column_metadata.go` | Modified | Finer OID mapping |
+| `internal/stackql/typing/relayed_column_metadata.go` | Modified | Finer OID mapping |
+| `internal/stackql/typing/oid_mapping_test.go` | New | OID mapping unit tests |
+| `internal/stackql/plan/plan.go` | Modified | ColumnMetadata field + getter/setter |
+| `internal/stackql/planbuilder/entrypoint.go` | Modified | Extract column metadata during plan build |
+| `internal/stackql/planbuilder/plan_builder.go` | Modified | getRootPrimitiveGenerator() on interface |
+| `test/robot/functional/stackql_mocked_from_cmd_line.robot` | Modified | 8 new indirect join tests + 3-way test |
+| `docs/views.md` | Modified | Updated supported join combinations |
+
+### Key architectural decisions
+
+- `queryshape.Inferrer` is the single entry point for ahead-of-time schema inference
+- Plan building (without execution) is the mechanism for inferring column types from provider schemas
+- Value coercion in `psqlwire/psqlwire.go` bridges sqlite's string-heavy output to pgtype's typed encoders
+- OID fidelity is now enabled: integer→T_int8, boolean→T_bool, fine-grained parser col types
+- The `shimNumericElement`/`shimNumericTextBytes` hacks are preserved for backward compatibility with the "numeric" pgtype path
+
+### Phase 3 Complete (2026-04-04)
+
+All Handle* methods now flow through stateful caches:
+- `HandleParse`: infers columns, caches in `stmtCache[stmtName]`
+- `HandleDescribeStatement`: returns from cache (no re-planning)
+- `HandleDescribePortal`: looks up portal→statement→columns
+- `HandleBind`: records portal→statement in `portalCache`
+- `HandleExecute`: looks up portal→OIDs, decodes params via `paramdecoder`, substitutes, executes
+- `HandleClose*`: cleans up caches
+
+New packages:
+- `internal/stackql/paramdecoder/` — decodes text AND binary format params (int2/4/8, float4/8, bool, timestamp, text)
+- Value coercion function `coerceForOID` in `psqlwire.go` — ready but not active (deferred to Phase 4 with OID fidelity)
+
+### Remaining work
+
+- **Phase 1 (OID fidelity)**: finer OIDs (integer→T_int8, bool→T_bool) break 10 tests because pgtype's text encoder formats values differently. Needs psql-wire change to bypass pgtype.Set() for text format and write strings directly. Deferred.
+- **Phase 2 (paramresolver)**: resolve $N placeholder OIDs from method schemas. pgx works without this (defaults to text). Enhancement.
+- **Phase 4 (psql-wire)**: respect resultFormats from Bind, enable binary result encoding. Separate repo.

docs/pg_wire_stackql_backend_migration.md

@@ -0,0 +1,217 @@
+# Migrating a stackql backend to extended query support
+
+This document describes how to add extended query protocol support to a stackql `ISQLBackend` implementation, replacing the default stubs provided by `psql-wire`.
+
+## Background
+
+The `psql-wire` library auto-detects whether an `ISQLBackend` also implements `IExtendedQueryBackend` via a type assertion in `connection.go`:
+
+```go
+if eb, ok := sqlBackend.(sqlbackend.IExtendedQueryBackend); ok {
+    extBackend = eb
+} else if sqlBackend != nil {
+    extBackend = sqlbackend.NewDefaultExtendedQueryBackend(sqlBackend)
+}
+```
+
+If the backend does not implement `IExtendedQueryBackend`, a `DefaultExtendedQueryBackend` wraps it. This default delegates `HandleExecute` to `HandleSimpleQuery` and stubs out everything else. Client libraries like pgx can connect and run unparameterised queries through this path.
+
+No factory or wiring changes are needed to opt in. Adding the methods to the existing struct is sufficient.
+
+## The IExtendedQueryBackend interface
+
+```go
+type IExtendedQueryBackend interface {
+    HandleParse(ctx context.Context, stmtName string, query string, paramOIDs []uint32) ([]uint32, error)
+    HandleBind(ctx context.Context, portalName string, stmtName string, paramFormats []int16, paramValues [][]byte, resultFormats []int16) error
+    HandleDescribeStatement(ctx context.Context, stmtName string, query string, paramOIDs []uint32) ([]uint32, []sqldata.ISQLColumn, error)
+    HandleDescribePortal(ctx context.Context, portalName string, stmtName string, query string, paramOIDs []uint32) ([]sqldata.ISQLColumn, error)
+    HandleExecute(ctx context.Context, portalName string, stmtName string, query string, paramFormats []int16, paramValues [][]byte, resultFormats []int16, maxRows int32) (sqldata.ISQLResultStream, error)
+    HandleCloseStatement(ctx context.Context, stmtName string) error
+    HandleClosePortal(ctx context.Context, portalName string) error
+}
+```
+
+## Migration steps
+
+### Step 1: Add no-op stubs to the existing backend
+
+Locate the struct in stackql that implements `ISQLBackend` (the one with `HandleSimpleQuery`, `SplitCompoundQuery`, and `GetDebugStr`). Add the seven methods below. These are direct copies of the `DefaultExtendedQueryBackend` behaviour, so robot tests should produce identical results.
+
+```go
+func (sb *YourBackend) HandleParse(ctx context.Context, stmtName string, query string, paramOIDs []uint32) ([]uint32, error) {
+    return paramOIDs, nil
+}
+
+func (sb *YourBackend) HandleBind(ctx context.Context, portalName string, stmtName string, paramFormats []int16, paramValues [][]byte, resultFormats []int16) error {
+    return nil
+}
+
+func (sb *YourBackend) HandleDescribeStatement(ctx context.Context, stmtName string, query string, paramOIDs []uint32) ([]uint32, []sqldata.ISQLColumn, error) {
+    return paramOIDs, nil, nil
+}
+
+func (sb *YourBackend) HandleDescribePortal(ctx context.Context, portalName string, stmtName string, query string, paramOIDs []uint32) ([]sqldata.ISQLColumn, error) {
+    return nil, nil
+}
+
+func (sb *YourBackend) HandleExecute(ctx context.Context, portalName string, stmtName string, query string, paramFormats []int16, paramValues [][]byte, resultFormats []int16, maxRows int32) (sqldata.ISQLResultStream, error) {
+    return sb.HandleSimpleQuery(ctx, query)
+}
+
+func (sb *YourBackend) HandleCloseStatement(ctx context.Context, stmtName string) error {
+    return nil
+}
+
+func (sb *YourBackend) HandleClosePortal(ctx context.Context, portalName string) error {
+    return nil
+}
+```
+
+**Verification**: run the full robot test suite. Behaviour should be unchanged.
+
+### Step 2: Add a compile-time interface check
+
+Near the top of the file, add:
+
+```go
+var _ sqlbackend.IExtendedQueryBackend = (*YourBackend)(nil)
+```
+
+This ensures the compiler catches missing methods if the interface changes.
+
+### Step 3: Implement HandleExecute with parameter substitution
+
+`HandleExecute` receives the query string and bound parameter values. The parameters arrive as:
+
+- `paramFormats []int16` — one entry per parameter: `0` = text, `1` = binary. May be empty (all text) or length 1 (applies to all).
+- `paramValues [][]byte` — raw bytes for each parameter. `nil` entry = SQL NULL.
+- `resultFormats []int16` — requested result column formats (currently safe to ignore; the wire library encodes as text).
+
+#### Option A: String interpolation (simplest)
+
+Replace positional parameters (`$1`, `$2`, ...) with their text values, applying appropriate quoting, then delegate to `HandleSimpleQuery`:
+
+```go
+func (sb *YourBackend) HandleExecute(ctx context.Context, portalName string, stmtName string, query string, paramFormats []int16, paramValues [][]byte, resultFormats []int16, maxRows int32) (sqldata.ISQLResultStream, error) {
+    resolved := substituteParams(query, paramFormats, paramValues)
+    return sb.HandleSimpleQuery(ctx, resolved)
+}
+```
+
+Where `substituteParams` replaces `$N` tokens with the corresponding text value. Rules:
+
+- `paramValues[i] == nil` → substitute `NULL` (no quotes).
+- Otherwise use the text representation from `paramValues[i]`. Quote string values with single quotes and escape embedded single quotes by doubling them (`'` → `''`).
+- If `paramFormats` is empty or has length 1, treat all parameters as that format (usually 0 = text).
+
+#### Option B: Native parameterisation
+
+If the stackql execution engine supports parameterised queries, pass the values through directly. This avoids quoting issues and is more correct long-term.
+
+**Verification**: write a test that connects with pgx, runs a parameterised query, and checks the results:
+
+```go
+rows, err := conn.Query(ctx, "SELECT $1::text, $2::int", "hello", 42)
+```
+
+### Step 4: Implement HandleDescribeStatement
+
+Client libraries call Describe after Parse to learn the result column types before any rows arrive. This allows typed scanning (e.g., pgx allocates `int32` vs `string` targets).
+
+Return parameter OIDs and result column metadata:
+
+```go
+func (sb *YourBackend) HandleDescribeStatement(ctx context.Context, stmtName string, query string, paramOIDs []uint32) ([]uint32, []sqldata.ISQLColumn, error) {
+    columns, err := sb.planQuery(query) // derive columns from query planner / schema
+    if err != nil {
+        return nil, nil, err
+    }
+    return paramOIDs, columns, nil
+}
+```
+
+Each `ISQLColumn` requires:
+- `GetName()` — column name
+- `GetObjectID()` — PostgreSQL type OID (e.g., `25` for text, `23` for int4, `16` for bool)
+- `GetWidth()` — column width in bytes (use `-1` if variable)
+- `GetTableId()`, `GetAttrNum()` — can be `0` if not applicable
+- `GetTypeModifier()` — usually `-1`
+- `GetFormat()` — `"text"` for text format
+
+If the query planner cannot derive columns (e.g., for DDL), return `nil` columns — the wire library sends `NoData`, which is valid.
+
+**Verification**: use pgx to prepare a statement and check that `FieldDescriptions()` returns the expected column metadata.
+
+### Step 5: Implement HandleDescribePortal
+
+Similar to `HandleDescribeStatement`, but for a bound portal. The portal already has its parameters bound, so column metadata may be more precise. In many cases this can delegate to the same logic:
+
+```go
+func (sb *YourBackend) HandleDescribePortal(ctx context.Context, portalName string, stmtName string, query string, paramOIDs []uint32) ([]sqldata.ISQLColumn, error) {
+    _, columns, err := sb.HandleDescribeStatement(ctx, stmtName, query, paramOIDs)
+    return columns, err
+}
+```
+
+### Step 6: Implement HandleParse with type resolution
+
+If stackql can resolve unspecified parameter types (OID = 0) from the query, do so in `HandleParse`. Otherwise, the current pass-through is fine — clients that send OID 0 will format parameters as text, which works with string interpolation.
+
+```go
+func (sb *YourBackend) HandleParse(ctx context.Context, stmtName string, query string, paramOIDs []uint32) ([]uint32, error) {
+    resolved, err := sb.resolveParamTypes(query, paramOIDs)
+    if err != nil {
+        return nil, err
+    }
+    return resolved, nil
+}
+```
+
+### Step 7: Implement HandleBind with validation
+
+If stackql can validate parameter values at bind time, do so here. Errors returned from `HandleBind` are reported to the client before execution, and the connection enters error recovery mode (messages discarded until Sync). This gives the client a chance to re-bind with corrected values.
+
+### Step 8: Implement HandleCloseStatement / HandleClosePortal
+
+If stackql caches query plans or intermediate state, release them here. If not, the no-ops from step 1 are correct.
+
+## Error recovery
+
+The wire library handles error recovery automatically. If any `IExtendedQueryBackend` method returns an error:
+
+1. An `ErrorResponse` is sent to the client.
+2. All subsequent messages are discarded until the client sends `Sync`.
+3. `Sync` sends `ReadyForQuery('E')` (failed transaction status).
+4. The client can then retry or issue new commands.
+
+Backend methods should return errors freely. They do not need to manage connection state.
+
+## Testing strategy
+
+Each step should be verified independently:
+
+1. **Step 1 (stubs)**: full robot test suite — no regressions.
+2. **Step 3 (execute)**: pgx test with parameterised `SELECT $1::text` — returns correct value.
+3. **Step 4 (describe)**: pgx `Prepare` + check `FieldDescriptions()` — column names and OIDs match.
+4. **Step 5 (describe portal)**: pgx query with `QueryRow` — typed scan works without explicit type hints.
+5. **Step 6 (parse)**: pgx query with untyped parameters — server resolves types, client formats correctly.
+
+For each step, a pgx integration test is the most realistic validation since pgx exercises the full Parse → Describe → Bind → Execute → Sync pipeline.
+
+## Common OIDs for reference
+
+| Type    | OID  | Go type       |
+|---------|------|---------------|
+| bool    | 16   | bool          |
+| int2    | 21   | int16         |
+| int4    | 23   | int32         |
+| int8    | 20   | int64         |
+| float4  | 700  | float32       |
+| float8  | 701  | float64       |
+| text    | 25   | string        |
+| varchar | 1043 | string        |
+| json    | 114  | string/[]byte |
+| jsonb   | 3802 | string/[]byte |
+
+These are defined in `github.com/lib/pq/oid` as `oid.T_bool`, `oid.T_int4`, etc.