feat(python): typed filter-expression builder as an alternative to raw SQL

[jonasdedden]

Problem

Raw SQL filters are parsed without any knowledge of the caller's intent, so three common generator patterns hit planner pitfalls:

• an integer column compared with a fractional literal fails to plan (i > 1.5 \u2192 could not convert to literal of type Int64, feature: accept a float literal against an integer column in filters #9317);
• a double-typed literal against a Float32 column casts the column and silently skips its scalar index (x > CAST(0.5 AS double) plans CAST(x AS Float64), perf: a double-typed literal against a Float32 column casts the column and skips its index #9318);
• a boolean column compared with a boolean expression fails in one operand order but not the other (flag != (id > 0) errors while (id > 0) != flag plans, bug: comparing a boolean column with a boolean expression fails to plan when the expression contains a literal #9319).

The existing typed path, pa.compute.Expression \u2192 Substrait, avoids the #9318/#9319 traps (verified: all spellings use the index, both boolean orders plan) but cannot express what filter generators need: pyarrow refuses to serialize the implicit safe cast in i > 1.5 (Substrait is only capable of representing unsafe casts), the planner rejects nested references in Substrait filters, and Expression has no constructors for string functions, temporal extraction, or list/struct access.

Chosen rule

A minimal client-side builder, lance.filter (col/lit/to_sql), rendered against the dataset schema. ScannerBuilder.filter (hence scanner/to_table), count_rows, delete, and update accept an Expr and render it with the dataset schema. Three index-preserving rules fire only with a schema in hand:

• fractional literal vs integer column \u2192 equivalent integer bound (i > 1.5 \u2192 i > 1), with null-aware tautologies for =/!=, infinities, and out-of-range bounds;
• literal vs Float32 \u2192 CAST(<literal> AS float), so the column keeps its type and index;
• boolean expression vs boolean expression \u2192 AND/OR/NOT expansion, so either operand order plans.

Negation is pushed through comparisons (the rewrites are filter-equivalent but not value-equivalent: NOT (i IS NOT NULL) would keep null rows that ~(i != 1.5) must drop). Anything without an exact spelling raises FilterError instead of guessing: division/modulo/power, NaN literals, timezone-aware timestamps, decimals, out-of-range integer literals.

Alternatives considered

• Fixing each SQL coercion in the planner (feature: accept a float literal against an integer column in filters #9317/perf: a double-typed literal against a Float32 column casts the column and skips its index #9318/bug: comparing a boolean column with a boolean expression fails to plan when the expression contains a literal #9319): owned by separate issues/PRs; this builder helps clients today and stays useful after those land (explicit literal types never hurt).
• Extending pa.compute.Expression: capped by pyarrow's serializer (safe casts, no function constructors); not something this repo controls.
• Building Substrait directly in Python: bypasses the parser but still hits the planner's nested-reference rejection and needs proto tooling for every function; larger surface for the same payoff.

Compatibility and minimum version

Pure-Python SQL generation; no Rust changes, no new dependencies, no format changes. Emitted SQL was executed verbatim on 9.0.0, 12.0.0b6, and 12.0.0b7: identical row counts everywhere except float-zero equality (f = 0.0 misses -0.0 before b7, as established for #6236). So float filters need pylance >= 12.0.0b7 (12.0.0 stable once published); every other spelling already plans on 9.0.0.

Test results

New test_filter.py section (9 tests): SQL snapshot spellings; positive and negated filters vs hand oracles over nulls, both zeros, infinities, nested fields, strings, dates, and microsecond timestamps; explain_plan assertions that the bound rewrite, the Float32 cast spelling, IN lists, and starts_with use ScalarIndexQuery while the hand-written double-cast baseline does not; count_rows/delete/update acceptance; error cases. Validated locally against the 13.0.0b4 wheel (9 passed); full CI pending. ruff check and ruff format --check pass on all touched files. Docs: new Typed filter expressions subsection in the read-and-write guide.

Limitations (why draft)

• Prototype scope: comparisons, boolean logic, null/NaN tests, IN, BETWEEN, + - *, starts_with/ends_with/contains; no temporal extraction, list/map access, or decimal/binary literals.
• Float column-vs-column comparisons render plain SQL and keep total-order semantics (bug: -0.0 and 0.0 still compare unequal between columns and in array_has #9316 scope, not handled here).
• Fragment.scanner still takes SQL strings only.
• Seeking direction: whether reviewers prefer this builder, a DataFusion-expression acceptance, or Substrait improvements before this leaves draft.

Related: #9317, #9318, #9319. Closes none of them.

[jonasdedden]

Closing in favor of #9333.