[SPARK-58761][SQL][TEST] Add dedicated e2e test coverage for GROUP BY over nanosecond-precision timestamp keys - #57992
Closed
stevomitric wants to merge 2 commits into
Closed
Conversation
stevomitric
changed the title
… over nanosecond-precision timestamp keys ### What changes were proposed in this pull request? Add end-to-end regression-lock coverage for `GROUP BY` over the nanosecond-precision timestamp types `TIMESTAMP_NTZ(p)` / `TIMESTAMP_LTZ(p)` (`p` in `[7, 9]`). New suite `TimestampNanosGroupBySuiteBase` (with `TimestampNanosGroupByAnsiOnSuite` / `TimestampNanosGroupByAnsiOffSuite` subclasses), mirroring `TimestampNanosJoinSuiteBase`. For NTZ and LTZ, across `p` in `[7, 9]` and both whole-stage-codegen modes, it asserts: - GROUP BY keeps two keys that share `epochMicros` but differ within the microsecond in separate groups, while collapsing exact duplicates, and computes per-group `count` / `sum`; - a `UNION` of two different nanos precisions widens the key to the higher precision (`findWiderDateTimeType`) and preserves the distinction; - NULL keys form a single group (unlike an equi-join, where NULL never matches NULL). Also extends the golden files `timestamp-ntz-nanos.sql` / `timestamp-ltz-nanos.sql` with a GROUP-BY-with-aggregates-and-NULL-group case. ### Why are the changes needed? GROUP BY on a nanosecond-timestamp key already works today -- it rides on the generic nanos hashing/equality implemented in SPARK-57103 (`Murmur3Hash` / `XxHash64` / `HiveHash` over the carrier's `epochMicros: Long` and `nanosWithinMicro: Short in [0, 999]`) -- but lacked dedicated coverage. These tests lock the regression: if the nanos hash or equality path is later broken, the sub-microsecond keys collapse and the tests fail loudly. No production change. ### Does this PR introduce _any_ user-facing change? No. Test-only. ### How was this patch tested? `TimestampNanosGroupByAnsiOnSuite` / `TimestampNanosGroupByAnsiOffSuite` (24 tests, all pass) and the regenerated golden SQL files. Co-authored-by: Isaac
stevomitric
force-pushed
the
stevomitric/nanos-groupby-tests
branch
from
August 13, 2026 13:57
675f73e to
fedd08a
Compare
stevomitric
changed the title
Contributor
Author
|
cc @uros-b, @cloud-fan PTAL when you get a chance. |
uros-b
approved these changes
Aug 14, 2026
Member
|
Thank you @stevomitric! Please resolve conflicts. |
Only the nanos golden files conflicted; the .sql inputs and the new GROUP BY suite merged cleanly. Golden files regenerated via SQLQueryTestSuite. Co-authored-by: Isaac
uros-b
pushed a commit
that referenced
this pull request
Aug 15, 2026
… over nanosecond-precision timestamp keys ### What changes were proposed in this pull request? Add end-to-end regression-lock coverage for `GROUP BY` over the nanosecond-precision timestamp types `TIMESTAMP_NTZ(p)` / `TIMESTAMP_LTZ(p)` (`p` in `[7, 9]`). New suite `TimestampNanosGroupBySuiteBase` (with `TimestampNanosGroupByAnsiOnSuite` / `TimestampNanosGroupByAnsiOffSuite` subclasses), mirroring `TimestampNanosJoinSuiteBase`. For NTZ and LTZ, across `p` in `[7, 9]` and both whole-stage-codegen modes, it asserts: - GROUP BY keeps two keys that share `epochMicros` but differ within the microsecond in separate groups, while collapsing exact duplicates, and computes per-group `count` / `sum`; - a `UNION` of two different nanos precisions widens the key to the higher precision (`findWiderDateTimeType`) and preserves the distinction; - NULL keys form a single group (unlike an equi-join, where NULL never matches NULL). Also extends the golden files `timestamp-ntz-nanos.sql` / `timestamp-ltz-nanos.sql` with a GROUP-BY-with-aggregates-and-NULL-group case. ### Why are the changes needed? GROUP BY on a nanosecond-timestamp key already works today -- it rides on the generic nanos hashing/equality implemented in SPARK-57103 (`Murmur3Hash` / `XxHash64` / `HiveHash` over the carrier's `epochMicros: Long` and `nanosWithinMicro: Short in [0, 999]`) -- but lacked dedicated coverage. These tests lock the regression: if the nanos hash or equality path is later broken, the sub-microsecond keys collapse and the tests fail loudly. No production change. ### Does this PR introduce _any_ user-facing change? No. Test-only. ### How was this patch tested? `TimestampNanosGroupByAnsiOnSuite` / `TimestampNanosGroupByAnsiOffSuite` (24 tests, all pass) and the regenerated golden SQL files. ### Was this patch authored or co-authored using generative AI tooling? Co-Authored-By: Claude Opus 4.8 Closes #57992 from stevomitric/stevomitric/nanos-groupby-tests. Authored-by: Stevo Mitric <stevomitric2000@gmail.com> Signed-off-by: Uros Bojanic <221401595+uros-b@users.noreply.github.com> (cherry picked from commit a2da292) Signed-off-by: Uros Bojanic <221401595+uros-b@users.noreply.github.com>
uros-b
pushed a commit
that referenced
this pull request
Aug 15, 2026
… over nanosecond-precision timestamp keys ### What changes were proposed in this pull request? Add end-to-end regression-lock coverage for `GROUP BY` over the nanosecond-precision timestamp types `TIMESTAMP_NTZ(p)` / `TIMESTAMP_LTZ(p)` (`p` in `[7, 9]`). New suite `TimestampNanosGroupBySuiteBase` (with `TimestampNanosGroupByAnsiOnSuite` / `TimestampNanosGroupByAnsiOffSuite` subclasses), mirroring `TimestampNanosJoinSuiteBase`. For NTZ and LTZ, across `p` in `[7, 9]` and both whole-stage-codegen modes, it asserts: - GROUP BY keeps two keys that share `epochMicros` but differ within the microsecond in separate groups, while collapsing exact duplicates, and computes per-group `count` / `sum`; - a `UNION` of two different nanos precisions widens the key to the higher precision (`findWiderDateTimeType`) and preserves the distinction; - NULL keys form a single group (unlike an equi-join, where NULL never matches NULL). Also extends the golden files `timestamp-ntz-nanos.sql` / `timestamp-ltz-nanos.sql` with a GROUP-BY-with-aggregates-and-NULL-group case. ### Why are the changes needed? GROUP BY on a nanosecond-timestamp key already works today -- it rides on the generic nanos hashing/equality implemented in SPARK-57103 (`Murmur3Hash` / `XxHash64` / `HiveHash` over the carrier's `epochMicros: Long` and `nanosWithinMicro: Short in [0, 999]`) -- but lacked dedicated coverage. These tests lock the regression: if the nanos hash or equality path is later broken, the sub-microsecond keys collapse and the tests fail loudly. No production change. ### Does this PR introduce _any_ user-facing change? No. Test-only. ### How was this patch tested? `TimestampNanosGroupByAnsiOnSuite` / `TimestampNanosGroupByAnsiOffSuite` (24 tests, all pass) and the regenerated golden SQL files. ### Was this patch authored or co-authored using generative AI tooling? Co-Authored-By: Claude Opus 4.8 Closes #57992 from stevomitric/stevomitric/nanos-groupby-tests. Authored-by: Stevo Mitric <stevomitric2000@gmail.com> Signed-off-by: Uros Bojanic <221401595+uros-b@users.noreply.github.com> (cherry picked from commit a2da292) Signed-off-by: Uros Bojanic <221401595+uros-b@users.noreply.github.com>
Member
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Add end-to-end regression-lock coverage for
GROUP BYover the nanosecond-precision timestamp typesTIMESTAMP_NTZ(p)/TIMESTAMP_LTZ(p)(pin[7, 9]).New suite
TimestampNanosGroupBySuiteBase(withTimestampNanosGroupByAnsiOnSuite/TimestampNanosGroupByAnsiOffSuitesubclasses), mirroringTimestampNanosJoinSuiteBase. For NTZ and LTZ, acrosspin[7, 9]and both whole-stage-codegen modes, it asserts:epochMicrosbut differ within the microsecond in separate groups, while collapsing exact duplicates, and computes per-groupcount/sum;UNIONof two different nanos precisions widens the key to the higher precision (findWiderDateTimeType) and preserves the distinction;Also extends the golden files
timestamp-ntz-nanos.sql/timestamp-ltz-nanos.sqlwith a GROUP-BY-with-aggregates-and-NULL-group case.Why are the changes needed?
GROUP BY on a nanosecond-timestamp key already works today -- it rides on the generic nanos hashing/equality implemented in SPARK-57103 (
Murmur3Hash/XxHash64/HiveHashover the carrier'sepochMicros: LongandnanosWithinMicro: Short in [0, 999]) -- but lacked dedicated coverage. These tests lock the regression: if the nanos hash or equality path is later broken, the sub-microsecond keys collapse and the tests fail loudly. No production change.Does this PR introduce any user-facing change?
No. Test-only.
How was this patch tested?
TimestampNanosGroupByAnsiOnSuite/TimestampNanosGroupByAnsiOffSuite(24 tests, all pass) and the regenerated golden SQL files.Was this patch authored or co-authored using generative AI tooling?
Co-Authored-By: Claude Opus 4.8