Skip to content

fix(datafusion): normalize signed zero in remaining filters - #9323

Open
lance-gatefixer[bot] wants to merge 3 commits into
mainfrom
gatekeeper/fix-9316-1
Open

lance-gatefixer[bot] wants to merge 3 commits into
mainfrom
gatekeeper/fix-9316-1

Conversation

@lance-gatefixer

@lance-gatefixer lance-gatefixer Bot commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Summary

  • normalize signed zero once on each operand of non-literal float comparisons
  • lower array_has zero probes to one array_has_any call covering both floating-point zero encodings
  • preserve one evaluation for volatile haystacks and cover scans, label-list indices, NULLs, and empty lists

Root cause

The signed-zero optimizer only handled comparisons that contained a zero literal. Column-to-column comparisons therefore reached Arrow total-order kernels unchanged, and array_has probed only the literal bit pattern, so negative and positive zero compared as different values. Expanding array_has into two OR arms would evaluate a volatile haystack twice, so the repair uses a single two-needle probe.

Validation

  • cargo fmt --all -- --check
  • cargo test -p lance-datafusion --features substrait (213 unit tests and 5 doctests passed)
  • cargo test -p lance --test integration_tests --features slow_tests test_signed_zero_between_columns_and_in_array_has
  • cargo clippy --all --tests --benches -- -D warnings

Fixes #9316

@github-actions github-actions Bot added the bug Something isn't working label Sep 16, 2026
@lance-gatefixer

Copy link
Copy Markdown
Contributor Author

Blocked: The Ready PR is blocked by a flaky linux-arm Check that the lance-gatefixer App cannot rerun.

Remote head a4270ba contains the current main tip d073576. Rust workflow run 35160280317, job linux-arm (105009217457), failed only test_legacy_ivf_pq_cosine_multivec_smoke at recall 0.49 versus the 0.50 threshold after 3,894 tests passed; the PR does not change that vector-index path, its KMeans initialization draws an OS-random seed, and focused reruns on this head pass. A failed-job rerun was attempted, but GitHub rejected it because this integration lacks Actions rerun access.

The smallest action is for a maintainer to rerun the failed linux-arm job. Alternatively, grant lance-gatefixer Actions write permission and wake this FixAgent so it can request the rerun.

lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Sep 16, 2026
@lance-gatekeeper lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Sep 17, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 17, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate recommendation: approve.

The array_has_any lowering preserves signed-zero coverage while evaluating volatile haystacks once. It remains stable across repeated optimization, LabelList-indexed scans, null and empty lists, and Substrait expression round-trips. I found no remaining material issue in this revision.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: -0.0 and 0.0 still compare unequal between columns and in array_has

0 participants