Improve performance of CASE WHEN x THEN y ELSE NULL expressions
#20097
+82
−29
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
|
run benchmark case_when |
|
🤖 |
Contributor
|
run benchmark tpcds |
|
🤖: Benchmark completed Details
|
|
🤖 |
|
🤖: Benchmark completed Details
|
martin-g
reviewed
Feb 2, 2026
| let return_type = self.data_type(&batch.schema())?; | ||
| let else_expr = | ||
| try_cast(Arc::clone(else_expr), &batch.schema(), return_type.clone()) | ||
| .unwrap_or_else(|_| Arc::clone(else_expr)); |
Member
There was a problem hiding this comment.
Why there is a fallback here ?
In which case the types may not match and the fallback is needed ?
Contributor
Author
There was a problem hiding this comment.
That's a very good question. I think this is actually one of the few lines of code I did not modify. Let me see if I can figure out the answer from git history.
Contributor
Author
There was a problem hiding this comment.
I traced this back to https://github.com/apache/datafusion/pull/1601/changes#diff-e27e1014e17614bf2ddd8b375475c5c31a279e58e97d2950a9a5427428645855R372 in PR #1601
. That change is from 2022, but perhaps the author @xudong963 remembers the context.
Contributor
Author
Contributor
There was a problem hiding this comment.
I don't remember the context either
Member
There was a problem hiding this comment.
@pepijnve Can't recall the reason now, imo, if not any test is broken, it's good to go
Contributor
Author
There was a problem hiding this comment.
Would we want to do that as part of this PR or a separate one? Changing this line would be completely unrelated to this PR's current description.
| EvalMethod::ScalarOrScalar | ||
| } else if body.when_then_expr.len() == 1 && body.else_expr.is_some() { | ||
| } else if body.when_then_expr.len() == 1 { | ||
| EvalMethod::ExpressionOrExpression(body.project()?) |
Member
There was a problem hiding this comment.
https://github.com/pepijnve/datafusion/blob/0d640690aa665da749d30148e6a9c6440d2020db/datafusion/physical-expr/src/expressions/case.rs#L76 says ... and both the thenandelse are expressions. But this change makes the else part optional.
Should ExpressionOrExpression's docstring be updated ?
Contributor
Author
There was a problem hiding this comment.
Yes, I should update the doc strings. I didn't check them to see if they got stale or not.
pepijnve
force-pushed
the
expr_or_expr
branch
from
727ac25 to
9124d37
Compare
Contributor
|
The TPCDS improvements are plausible as they have CASE in them
These ones don't have case 🤷 |
Contributor
|
run benchmark tpcds |
Contributor
|
run benchmark case_when |
Contributor
|
BTW I think we could add some more test coverage -- here are some cases for case.slt that are worth considering (written by codex) # single WHEN, no ELSE (absent)
query
SELECT CASE WHEN a > 0 THEN b END
FROM (VALUES (1, 10), (0, 20)) AS t(a, b);
----
10
NULL
# single WHEN, explicit ELSE NULL
query
SELECT CASE WHEN a > 0 THEN b ELSE NULL END
FROM (VALUES (1, 10), (0, 20)) AS t(a, b);
----
10
NULL
# fallible THEN expression should only be evaluated on true rows
query
SELECT CASE WHEN a > 0 THEN 10 / a END
FROM (VALUES (1), (0)) AS t(a);
----
10
NULL
# all-false path returns typed NULLs
query
SELECT CASE WHEN a < 0 THEN b END
FROM (VALUES (1, 10), (2, 20)) AS t(a, b);
----
NULL
NULL |
|
🤖 |
|
🤖: Benchmark completed Details
|
|
🤖 |
|
🤖: Benchmark completed Details
|
Contributor
Author
|
Extracting the relevant bits that illustrate the optimisation clearly |
Contributor
|
Thanks again everyone! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
While reviewing #19994 it became clear the optimised
ExpressionOrExpressioncode path was not being used when the case expression has noelsebranch or haselse null. In those situations the general evaluation strategies could end up being used.This PR refines the
ExpressionOrExpressionimplementation to also handleelse nullexpressions.What changes are included in this PR?
Use
ExpressionOrExpressionfor expressions of the formCASE WHEN x THEN y [ELSE NULL]Are these changes tested?
Covered by existing SLTs
Are there any user-facing changes?
No