fix: ignore NaN sign bits in filters

[lance-gatefixer]

Summary

• normalize away only the NaN sign bit for column and computed comparisons while preserving payloads and signed-zero behavior
• lower literal range predicates to indexable ranges that treat both NaN sign encodings consistently
• keep NaN-bearing zone-map regions as conservative candidates for the negative-NaN range
• cover Float16, Float32, and Float64 scan/index behavior plus multi-fragment column comparisons

Root cause

Arrow comparison and scalar-index kernels use total ordering, which places sign-bit-set NaNs below negative infinity while positive NaNs sort above finite values. Lance already rewrote signed-zero literals but did not normalize NaN signs, so logically equivalent NaNs produced different filter results.

Validation

• cargo test -p lance-datafusion
• cargo test -p lance-index scalar::zonemap::tests::test_nan_zonemap_index
• cargo test -p lance --test integration_tests --features slow_tests query::primitives::test_nan_sign_is_ignored_in_column_comparisons
• cargo test -p lance --test integration_tests --features slow_tests query::primitives::test_query_float_special_values
• cargo test -p lance --test integration_tests --features slow_tests query::primitives::test_float_zero_predicate_uses_scalar_index
• cargo fmt --all -- --check
• cargo clippy --all --tests --benches -- -D warnings

Fixes #9315

[lance-gatefixer]

Addressed in 3e35bc6: ordered literal comparisons remain scalar-indexable, including computed JSON paths, and mem-WAL combines the NaN guard into one exact range. The additional -inf boundary remains the intentional cost of correcting existing datasets without changing stored NaN bits.

[lance-gatekeeper]

⚠️ Gate recommendation: approve with a non-blocking risk.

The current head preserves scalar-index planning—including computed JSON paths—and gives mem-WAL an exact half-open < range while making NaN sign encodings compare equivalently.

The author has accepted the additional −∞ range cost. The remaining non-blocking performance exposure is recurring read-path work: ordered float-literal predicates add that range (> and >= may perform a second index search), and float column-to-column comparisons scan both operands through normalization UDFs, allocating when negative NaNs are present. Performance-sensitive adopters should benchmark representative float filters.

[lance-gatefixer]

Blocked: PR #9324 still requires an approving review before it can merge.

Remote head 08e500df0e66eb65d278493573d89f464864b55c is Ready and mergeable, all live checks pass, and GitHub reports reviewDecision=REVIEW_REQUIRED and mergeStateStatus=BLOCKED; the latest Gatekeeper review is COMMENTED with “approve with a non-blocking risk,” not an approval.

I revalidated every current Review and inline thread: there are no new actionable requests, and the existing App disposition remains later than the only prior author-facing performance concern.

A maintainer can approve the PR; alternatively, Gatekeeper can issue its canonical approval if the documented non-blocking performance risk is acceptable.

[lance-gatefixer]

Blocked: PR #9324 still requires an approving review before it can merge.

Remote head 08e500df0e66eb65d278493573d89f464864b55c is Ready and mergeable, all live checks pass, and GitHub reports reviewDecision=REVIEW_REQUIRED and mergeStateStatus=BLOCKED; the latest Gatekeeper review is COMMENTED with “approve with a non-blocking risk,” not an approval.

I revalidated all current Reviews, inline threads, and later external contributions: there are no new actionable requests, and the existing App disposition remains valid for the prior performance concern.

A maintainer can approve the PR; alternatively, Gatekeeper can issue its canonical approval if the documented non-blocking performance risk is acceptable.