PERF: add an Arrow/NumPy hybrid groupby path for decimal and string types. #63416
+344
−14
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…branches - Split test_groupby_aggregations into test_groupby_decimal_aggregations and test_groupby_string_aggregations - Split test_groupby_dropna into test_groupby_dropna_true and test_groupby_dropna_false - Use explicit Decimal values instead of range() casts for decimal tests - Parametrize values directly to avoid runtime branching 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
fangchenli
added
Performance
- Add type annotation for aggs list to handle mixed tuple types - Rename result to fallback_result to avoid type conflict 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
String types only support min, max, count - skip sum, prod, mean, std, var. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
doc/source/whatsnew/vX.X.X.rstfile if fixing a bug or adding a new feature.AGENTS.md.xref: #55234
This PR was tested for int and float as well. pandas' groupby is around 20-35% faster than the Arrow-native one for int and float types. We could easily change the condition to turn it on for int and float if Arrow's groupby got more optimized in the future.
During the reordering stage, it falls back to NumPy due to the limitation of
pyarrow.compute.scatter.And the workaround is more expensive. This hybrid approach is not perfect, but it gets us one step closer to an Arrow-native implementation.