feat: support SQL IS [NOT] DISTINCT FROM as null-safe comparison - #9331
jonasdedden wants to merge 2 commits into
Conversation
Drop the test asserting the scalar index is not used and the asserts pinning known float total-order behavior (lance-format#9332, lance-format#9315); cover integers, strings and both zero spellings in the existing query tests instead.
|
@lance-gatekeeper: Agreed. The risk isn't specific to this spelling: |
There was a problem hiding this comment.
The author clarification confirms that this PR only exposes the existing comparison semantics and now avoids tests that pin the signed-zero or NaN total-order behavior. The direct parser lowering still matches DataFusion and preserves null-safe evaluation plus the existing signed-zero literal rewrite, so no change is requested here. The accepted residual risk is that float column-to-column and NaN distinctness can still follow Arrow total order; #9323 is the active signed-zero repair and #9315 tracks NaN ordering.
Support SQL
IS [NOT] DISTINCT FROMin Lance filters, which previously failed with "not supported SQL".The planner now maps both spellings to DataFusion's
IsDistinctFrom/IsNotDistinctFrom, as DataFusion's own SQL frontend does. Literal coercion, the signed-zero rewrite (x IS NOT DISTINCT FROM 0.0matches both zero encodings) and null-safe evaluation already handle these operators. Also fixes thesigned_zero.rsdoc table, which described an outdated rewrite shape.Column-vs-column signed zeros and NaN still follow Arrow's total order (#9316, fixed by #9323; #9315).