fix: treat -0.0 and 0.0 as equal between float columns

[jonasdedden]

Problem

Part of #9316. #6236 (fixing #5868) rewrites a zero literal to the encoding that answers correctly, so a = 0.0 matches both zeros. Comparisons between two float values have no literal to retarget and still compare by encoding:

import lance
import pyarrow as pa
data = pa.table({
    "id": [0, 1, 2],
    "a": [-0.0, 0.0, 1.0],
    "b": [0.0, -0.0, 1.0],
})
ds = lance.write_dataset(data, "zero.lance", mode="overwrite")
def ids(f):
    return sorted(ds.to_table(columns=["id"], filter=f)["id"].to_pylist())
print(ids("a = b"))  # [2], expected [0, 1, 2]
print(ids("a < b"))  # [0], expected []

Full truth table on main (pylance 13.0.0b4): a = b misses both opposite-zero pairs, a != b admits them, a < b admits -0 < +0, a > b admits +0 > -0, a <= b misses +0 <= -0, a >= b misses -0 >= +0. Nested fields (s.x = s.y) and computed operands (a * 1.0 = b) fail the same way.

Chosen rule

Rewrite each comparison between two non-literal float values to name both zero encodings on each side, in each side's own float width:

written	evaluated
a = b, a <= b, a >= b	(pred) OR (a IN (-0, 0) AND b IN (-0, 0))
a != b, a < b, a > b	(pred) AND NOT (a IN (-0, 0) AND b IN (-0, 0))
a IS NOT DISTINCT FROM b	(pred) OR ((both zero) IS TRUE)
a IS DISTINCT FROM b	(pred) AND ((both zero) IS NOT TRUE)

Float-ness comes from Expr::get_type(&DFSchema), so bare columns, struct fields, array_element, casts, and arithmetic all participate when they infer to Float16/Float32/Float64. Non-float pairs and any comparison with a literal side are declined (literals belong to the existing arms). NULL rows stay NULL for the six value operators (NULL IN .. is NULL); the distinctness pair stays decided through IS [NOT] TRUE. BETWEEN needs no arm: the simplifier expands non-constant bounds into >=/<= before this runs. The OR/AND dedup now also covers the emitted both-zero shapes, so the output remains a fixed point of optimize_expr (the scan path optimizes twice).

Alternatives considered

• Physical kernels with IEEE semantics: would fix every operator at once but requires new comparison kernels plus scalar-index changes (btree/bitmap order by total_cmp, bloom hashes bits). Rejected as invasive for this issue.
• Exact-encoding guards (a = -0 AND b = +0): needs bit-exact tests that the existing IN (-0, 0) spelling cannot express after fix: make float filters treat -0.0 and 0.0 as the same value #6236 (it matches both encodings). The broader both zero guard gives the same truth table for all six operators with the existing index-friendly IN shape, so the simpler rule wins.
• A BETWEEN-with-column-bounds arm: unnecessary; only fully constant BETWEEN survives unexpanded, and that arm already exists.

Compatibility

• No file-format, public-API, or dependency change (signed_zero is crate-private; Planner::optimize_expr signature unchanged).
• Behavior change is limited to float-float comparisons where at least one side could be a zero: opposite-zero pairs now compare equal, same-encoding and non-zero rows unchanged, NULL inputs still filter to NULL (distinctness stays decided), non-float comparisons byte-identical.
• NaN ordering between columns is unchanged and pinned by a new test; fixing it is bug: a NaN with its sign bit set sorts below every number in filters #9315.
• Column-column predicates fall back to filtering (explain_plan shows no ScalarIndexQuery); single-column zero-literal index pushdown is unchanged and still asserted.

Test plan

• cargo test -p lance-datafusion --lib: 199 passed (86 in signed_zero, including 6 new column shape/idempotence/width/computed cases).
• cargo test -p lance --test integration_tests --features slow_tests query::primitives::test_query_float*: 10 passed across Float16/32/64 — existing test_query_float_special_values plus new test_query_float_column_zero_comparison (12 rows: ±0 pairs both directions and same-encoding, ±1, ±inf, null/null, null/zero, zero/null, 1/-1; bare, swapped, struct-field, and * 1.0 computed forms; NOT variants; indexed via BTree/Bitmap/ZoneMap plus unindexed via assert_filter_ids, with a pinned no-ScalarIndexQuery plan check) and test_query_float_column_zero_with_nans_pinned (same-NaN equal, opposite-sign NaN ordered, zeros fixed alongside).
• cargo test -p lance --test integration_tests --features slow_tests query::primitives::test_float_zero_predicate_uses_scalar_index: passed (literal index use preserved).
• cargo test -p lance-index --lib scalar: 1084 passed.
• cargo clippy --all --tests --benches -- -D warnings clean, cargo fmt --all --check clean.

Limitations / follow-up

• array_has(list, 0.0) is intentionally untouched here; it is the sibling half of bug: -0.0 and 0.0 still compare unequal between columns and in array_has #9316.
• a IN (b, ...) with column list elements still compares by encoding.
• GROUP BY/DISTINCT/ordering, merge_insert join keys (DataFusion hashes bits), and KNN distance_range are unchanged.
• Computed operands are named twice (once in the passthrough comparison, once in the zero IN), so they evaluate twice; avoiding that needs planner-level CSE, not a local rewrite.

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The signed-zero correction is sound for stable operands, but the rewrite must not evaluate a computed operand more than once. Preserve single-evaluation semantics by normalizing each operand once, or leave volatile expressions on the pre-existing path; the current cloning changes supported filter results.

[lance-gatekeeper]

Cloning each computed operand into the zero guard evaluates volatile expressions a second time, so the guard can describe a different value from the comparison and change existing query results. Lance supports random() in filters: with one million z = +0.0 rows, the regression below selected 750,540 rows on this head; one evaluation of round(random()) should match about half. The optimized expression contains two independent round(random()) calls. Please either preserve single evaluation (for example, normalize each operand once) or decline this rewrite for expressions containing volatile functions, and cover the case.

Reproducer

#[test]
fn volatile_operand_is_evaluated_once() {
    use std::sync::Arc;
    use arrow_array::{ArrayRef, BooleanArray, Float64Array, RecordBatch};

    let row_count = 1_000_000;
    let batch = RecordBatch::try_from_iter(vec![(
        "z",
        Arc::new(Float64Array::from(vec![0.0; row_count])) as ArrayRef,
    )])
    .unwrap();
    let planner = crate::planner::Planner::new(batch.schema());
    let optimized = planner
        .optimize_expr(planner.parse_filter("round(random()) = z").unwrap())
        .unwrap();
    let values = planner
        .create_physical_expr(&optimized)
        .unwrap()
        .evaluate(&batch)
        .unwrap()
        .into_array(row_count)
        .unwrap();
    let matches = values
        .as_any()
        .downcast_ref::<BooleanArray>()
        .unwrap()
        .iter()
        .filter(|value| *value == Some(true))
        .count();

    assert!(
        (450_000..=550_000).contains(&matches),
        "matches={matches}, expression={optimized}"
    );
}

cargo test -p lance-datafusion --lib volatile_operand_is_evaluated_once -- --nocapture

Observed: matches=750540, expression=round(random()) = z OR round(random()) IN (...) AND z IN (...).

[jonasdedden]

Closing in favor of #9323.