fix(datafusion): normalize signed zero in remaining filters - #9323
lance-gatefixer[bot] wants to merge 3 commits into
Conversation
|
Blocked: The Ready PR is blocked by a flaky linux-arm Check that the lance-gatefixer App cannot rerun. Remote head a4270ba contains the current main tip d073576. Rust workflow run 35160280317, job linux-arm (105009217457), failed only test_legacy_ivf_pq_cosine_multivec_smoke at recall 0.49 versus the 0.50 threshold after 3,894 tests passed; the PR does not change that vector-index path, its KMeans initialization draws an OS-random seed, and focused reruns on this head pass. A failed-job rerun was attempted, but GitHub rejected it because this integration lacks Actions rerun access. The smallest action is for a maintainer to rerun the failed linux-arm job. Alternatively, grant lance-gatefixer Actions write permission and wake this FixAgent so it can request the rerun. |
There was a problem hiding this comment.
✅ Gate recommendation: approve.
The array_has_any lowering preserves signed-zero coverage while evaluating volatile haystacks once. It remains stable across repeated optimization, LabelList-indexed scans, null and empty lists, and Substrait expression round-trips. I found no remaining material issue in this revision.
Summary
Root cause
The signed-zero optimizer only handled comparisons that contained a zero literal. Column-to-column comparisons therefore reached Arrow total-order kernels unchanged, and array_has probed only the literal bit pattern, so negative and positive zero compared as different values. Expanding array_has into two OR arms would evaluate a volatile haystack twice, so the repair uses a single two-needle probe.
Validation
Fixes #9316