Fix iceberg partition pruning by nanoseconds by ianton-ru · Pull Request #2239 · Altinity/ClickHouse

ianton-ru · 2026-08-18T20:14:03Z

Changelog category (leave one):

Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Fix partition and min/max pruning for Iceberg tables by timestamp_ns type, fix writing DateTime(9) as timestamp_ns.

Documentation entry for user-facing changes

...

CI/CD Options

Exclude tests:

Regression jobs to run:

Identity partition pruning on `timestamp_ns` over-prunes matching files. Min/max pruning also returns empty results when Iceberg `timestamp` (`DateTime64(6)`) lower/upper bounds are stored as nanoseconds, which is the customer `toDateTime64(..., 6)` case. Co-authored-by: Cursor <cursoragent@cursor.com>

Identity partition values for `timestamp_ns` arrive as Avro Int64, and some writers store nanosecond min/max bytes on Iceberg `timestamp` (`DateTime64(6)`). Both were compared at the wrong scale and dropped matching files. Write `DateTime64(9)` as `timestamp_ns` so ClickHouse does not recreate that layout. Co-authored-by: Cursor <cursoragent@cursor.com>

github-actions · 2026-08-18T20:15:36Z

Workflow [PR], commit [e47cab5]

`1e16` microseconds is ~year 2286, so spec-correct Spark `9999-12-31` and `DateTime64` max `2299-12-31` bounds were divided by 1000 and over-pruned. Convert only `|ticks| > 1e18` (true nanosecond band); fail open in between. Co-authored-by: Cursor <cursoragent@cursor.com>

`timestamp_ns` is v3-only. Mapping `DateTime64(9)` without checking `iceberg_format_version` wrote invalid v2 metadata (default is 2). Throw instead of emitting v3 types into v1/v2 tables. Co-authored-by: Cursor <cursoragent@cursor.com>

ianton-ru · 2026-08-19T08:12:18Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 381b43d5e2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-08-19T08:25:48Z

+    if (scale == 6 && magnitudeExceeds(unscaled, microseconds_ambiguous_threshold))
+    {
+        if (!magnitudeExceeds(unscaled, nanoseconds_lower_threshold))
+            return std::nullopt;
+
+        unscaled = nanosecondsToMicrosecondsForBound(unscaled, lower_bound);


Skip pruning instead of guessing nanosecond bounds

When an Iceberg timestamp manifest contains a magnitude above this threshold, the schema still identifies the value as microseconds; magnitude alone cannot prove that the writer actually used nanoseconds. Dividing the value and then trusting the resulting range allows mayBeTrueInRange to prune a file whose corrupt or differently encoded bounds do not cover its data, producing missing query results. Treat these suspicious bounds as unavailable and skip min/max pruning rather than taking a consequential fallback path.

AGENTS.md reference: AGENTS.md:L153-L153

Useful? React with 👍 / 👎.

This is not a P1. The overlapping band already fail-opens; converting |ticks| > 1e18 is the customer ns-stats case, not a guess inside the µs/ns overlap.
Iceberg timestamp is still microseconds in the schema. Magnitude cannot prove nanoseconds, which is why |ticks| in (1e16, 1e18] already returns nullopt. Callers then skip that hyperrectangle (if (!left || !right) continue) and min/max pruning for that column is skipped. That covers spec-correct far-future microseconds: Spark 9999-12-31 (~2.53e17 µs) and ClickHouse DateTime64 max 2299-12-31 (~1.04e16 µs).
Conversion runs only for |ticks| > 1e18. As microseconds that is ~year 33658, which is outside ClickHouse DateTime64(6) (max ~2299, ~1.04e16 µs) and outside normal Iceberg timestamp sentinels. As nanoseconds it is the customer layout (~1.76e18 for 2026) that previously over-pruned every file.
Fail-opening that band as well would keep query results correct (files scanned) but would disable min/max pruning again for those tables (IcebergMinMaxIndexPrunedFiles: 0). That undoes the point of this heuristic.
A corrupt 8-byte value > 1e18 that is not nanoseconds could still be converted into a fake in-range window. That is a residual, not a realistic P1 on this path.

…n_pruning_by_nanoseconds

DimensionWieldr · 2026-09-22T20:31:10Z

While writing some tests, AI found a potential issue:

Inserting into a format-version 3 table partitioned by DateTime64(9) fails with BAD_ARGUMENTS:

Unsupported type for iceberg DateTime64(9)

getAvroType in src/Storages/ObjectStorage/DataLakes/Iceberg/Utils.cpp still accepts only DateTime64 scale 6 when it encodes a partition value in the manifest. The data-file write path is fine: the same DateTime64(9) columns insert and prune on an unpartitioned v3 table.

This is the insert in test_writes_partition_pruning_datetime64_nanoseconds. Reproduced on 3948dca (clickhouse-common-static_26.6.4.20001.altinityantalya), table PARTITION BY ts with (ts DateTime64(9), value DateTime64(9), id Int32).

These cases on the same build succeed:

Reading and pruning timestamp_ns, including identity partitions written outside ClickHouse.
Unpartitioned DateTime64(9) writes stored as timestamp_ns, with min/max pruning.
DateTime64(9, 'UTC') stored as timestamptz_ns.
Create and ALTER of DateTime64(9) rejected on format versions 1 and 2; ALTER ADD Nullable(DateTime64(9)) allowed on version 3.

…n_pruning_by_nanoseconds

`getAvroType` required `timestamp-micros` after the ClickHouse#109764 backport, so identity-partitioned `timestamp_ns` writes failed. Co-authored-by: Cursor <cursoragent@cursor.com>

ianton-ru · 2026-10-01T14:18:10Z

@DimensionWieldr Thanks, must be fixed now

DimensionWieldr · 2026-10-02T19:11:22Z

CI looks good from my end.

Regression tests for timestamp nanosecond pruning are passing on both Ice and Glue catalogs: https://github.com/Altinity/clickhouse-regression/blob/main/iceberg/tests/iceberg_engine/timestamp_ns_pruning.py

DimensionWieldr · 2026-10-02T20:11:17Z

AI Audit Summary

Two follow-ups on timestamp_ns / timestamptz_ns.

DateTime64(9) with a non-UTC zone reloads as UTC. getIcebergType maps any DateTime64(9) with an explicit timezone to timestamptz_ns. IcebergSchemaProcessor::getSimpleType rebuilds that as DateTime64(9, "UTC"). shouldReloadSchemaForConsistency() returns true, and create plus every query rebuild columns from that mapping, so DateTime64(9, 'Europe/Berlin') becomes DateTime64(9, 'UTC') immediately. A string literal is parsed in the column zone, so WHERE ts = '2024-06-01 10:00:00' means 10:00 UTC afterward.

Iceberg does not keep the original zone name. Microsecond timestamptz already uses iceberg_timezone_for_timestamptz (default UTC). timestamptz_ns ignores that setting and hardcodes UTC. The read path should use the same setting.

timestamp_ns Parquet is marked UTC-adjusted. PrepareForWrite sets isAdjustedToUTC true for every DateTime64. Iceberg timestamp_ns requires TIMESTAMP(NANOS, false). An insert of DateTime64(9) with no zone writes timestamp_ns in the table metadata and a UTC-adjusted timestamp in the Parquet footer. DateTime64(9, 'UTC') is fine: that is timestamptz_ns, which is supposed to be adjusted.

ClickHouse reads the integers through the Iceberg schema, so a ClickHouse round-trip still succeeds. A reader that checks the Parquet logical type against the Iceberg type can reject the file. This flag was already forced true for every DateTime64, including ordinary timestamp. This change makes that mismatch apply to timestamp_ns. The fix belongs in the shared Parquet writer.

`getSimpleType` hardcoded UTC for nanosecond timestamptz while microsecond `timestamptz` already used the setting. Co-authored-by: Cursor <cursoragent@cursor.com>

Default Parquet still marks every `DateTime64` as `isAdjustedToUTC`. Iceberg writes set the flag from the column timezone so `timestamp_ns` matches `TIMESTAMP(NANOS, false)`. Co-authored-by: Cursor <cursoragent@cursor.com>

…n_pruning_by_nanoseconds Co-authored-by: Cursor <cursoragent@cursor.com>

DimensionWieldr · 2026-10-07T17:55:32Z

Found one more thing:

Identity timestamp_ns partition fields lose the Avro logical type

Impact: Manifests for tables partitioned by DateTime64(9) store that partition key as a bare Avro long. CH still prunes using the table schema, but a reader that trusts the file schema does not see timestamp-nanos or adjust-to-utc, so it can treat nanosecond keys as an untyped integer.

Anchor: Utils.cpp / getAvroType; contrib/avro/lang/c++/impl/Compiler.cc / makeLogicalType; IcebergWrites.cpp / extendSchemaForPartitions

Trigger: INSERT into an Iceberg v3 table partitioned by DateTime64(9) or DateTime64(9, 'UTC').

Why defect: getAvroType emits logicalType: timestamp-nanos, then compileJsonSchemaFromString maps any unknown logical type to LogicalType::NONE. DataFileWriter persists that compiled schema, which omits the annotation. The comment in getAvroType says other engines still see it.

Fix direction (short): Keep the original schema JSON in the manifest (or teach the bundled Avro writer a timestamp-nanos logical type) so the written header still contains it.

contrib Avro `toJson` drops unknown logical types, so DataFileWriter was writing a bare `long`. Persist the original JSON in `avro.schema` instead. Co-authored-by: Cursor <cursoragent@cursor.com>

…n_pruning_by_nanoseconds

DimensionWieldr · 2026-10-09T21:53:04Z

Debug build is failing.

The build is compiling unit tests, and two of them still call getIcebergType with two arguments. This PR changed that function so the third argument, format_version, is required.

DimensionWieldr · 2026-10-10T00:08:45Z

Also another possible issue:

Nanosecond bounds near the epoch are read as microseconds and can drop the whole file

Impact: A query can skip a data file that contains matching rows. If the file’s lower bound falls in a short window after the epoch and its upper bound is a normal nanosecond timestamp, the decoded range is inverted and every row in that file is pruned.

Anchor: src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergFieldParseHelpers.cpp / deserializeDateTime64FromBinaryRepr

Trigger: Iceberg timestamp (DateTime64(6)) whose 8-byte min/max are nanoseconds, with the lower bound in about 1970-01-21..1970-04-26 and the upper bound after about 2001-09-09 (the same non-spec layout this PR fixes for 2024+ bounds).

Why defect: Only |ticks| > 1e18 is divided by 1000. |ticks| <= 1e16 is kept as microseconds (~115 days). A 2024 upper bound becomes ~1.7e15 µs while a lower bound of ~1e16 ns stays 1e16 µs, so lower > upper. Range::intersectsRange then matches nothing.

Fix direction (short): If either bound is in the nanosecond band, convert both, or fail open when the decoded lower bound is greater than the upper bound.

ianton-ru and others added 2 commits August 18, 2026 21:12

ianton-ru changed the title ~~Bugfix/antalya 26.6/iceberg partition pruning by nanoseconds~~ Fix iceberg partition pruning by nanoseconds Aug 18, 2026

ianton-ru and others added 2 commits August 18, 2026 22:37

svb-alt added antalya antalya-26.6 labels Aug 19, 2026

blau-ai mentioned this pull request Aug 19, 2026

Atnalya 26.6 EXPORT PARTITION with position matching + extra columns in source #2229

Merged

11 of 28 tasks

ianton-ru marked this pull request as ready for review August 19, 2026 08:22

chatgpt-codex-connector Bot reviewed Aug 19, 2026

View reviewed changes

ianton-ru and others added 3 commits August 27, 2026 16:37

Merge branch 'antalya-26.6' into bugfix/antalya-26.6/iceberg_partitio…

ad11f98

…n_pruning_by_nanoseconds

Fix build

273e50f

Merge branch 'antalya-26.6' into bugfix/antalya-26.6/iceberg_partitio…

3948dca

…n_pruning_by_nanoseconds

subkanthi mentioned this pull request Sep 16, 2026

Iceberg v3: tracking #2390

Open

8 of 15 tasks

ianton-ru and others added 2 commits October 1, 2026 15:53

Merge branch 'antalya-26.6' into bugfix/antalya-26.6/iceberg_partitio…

eb1e381

…n_pruning_by_nanoseconds

Fix Iceberg inserts into tables partitioned by DateTime64(9)

6f0719e

`getAvroType` required `timestamp-micros` after the ClickHouse#109764 backport, so identity-partitioned `timestamp_ns` writes failed. Co-authored-by: Cursor <cursoragent@cursor.com>

ianton-ru and others added 3 commits October 7, 2026 10:48

Honor iceberg_timezone_for_timestamptz for Iceberg timestamptz_ns

d08eb59

`getSimpleType` hardcoded UTC for nanosecond timestamptz while microsecond `timestamptz` already used the setting. Co-authored-by: Cursor <cursoragent@cursor.com>

Write Iceberg timestamp_ns Parquet as not UTC-adjusted

1b0e9d6

Default Parquet still marks every `DateTime64` as `isAdjustedToUTC`. Iceberg writes set the flag from the column timezone so `timestamp_ns` matches `TIMESTAMP(NANOS, false)`. Co-authored-by: Cursor <cursoragent@cursor.com>

Merge branch 'antalya-26.6' into bugfix/antalya-26.6/iceberg_partitio…

3f49934

…n_pruning_by_nanoseconds Co-authored-by: Cursor <cursoragent@cursor.com>

Keep Iceberg timestamp-nanos in identity-partition Avro schemas

8ddc7a2

contrib Avro `toJson` drops unknown logical types, so DataFileWriter was writing a bare `long`. Persist the original JSON in `avro.schema` instead. Co-authored-by: Cursor <cursoragent@cursor.com>

ianton-ru added the port-antalya PRs to be ported to all new Antalya releases label Oct 8, 2026

Merge branch 'antalya-26.6' into bugfix/antalya-26.6/iceberg_partitio…

e47cab5

…n_pruning_by_nanoseconds

Navigation Menu

Repository navigation

Fix iceberg partition pruning by nanoseconds - #2239

Fix iceberg partition pruning by nanoseconds#2239
ianton-ru wants to merge 14 commits into
antalya-26.6from
bugfix/antalya-26.6/iceberg_partition_pruning_by_nanoseconds

ianton-ru commented Aug 18, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 18, 2026 •

edited

Loading

Uh oh!

ianton-ru commented Aug 19, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Aug 19, 2026

Uh oh!

ianton-ru Aug 19, 2026

Uh oh!

DimensionWieldr commented Sep 22, 2026 •

edited

Loading

Uh oh!

ianton-ru commented Oct 1, 2026

Uh oh!

DimensionWieldr commented Oct 2, 2026 •

edited

Loading

Uh oh!

DimensionWieldr commented Oct 2, 2026

Uh oh!

DimensionWieldr commented Oct 7, 2026

Uh oh!

DimensionWieldr commented Oct 9, 2026

Uh oh!

DimensionWieldr commented Oct 10, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

ianton-ru commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Documentation entry for user-facing changes

CI/CD Options

Exclude tests:

Regression jobs to run:

Uh oh!

github-actions Bot commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ianton-ru commented Aug 19, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Aug 19, 2026

Choose a reason for hiding this comment

Uh oh!

ianton-ru Aug 19, 2026

Choose a reason for hiding this comment

Uh oh!

DimensionWieldr commented Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ianton-ru commented Oct 1, 2026

Uh oh!

DimensionWieldr commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DimensionWieldr commented Oct 2, 2026

AI Audit Summary

Uh oh!

DimensionWieldr commented Oct 7, 2026

Uh oh!

DimensionWieldr commented Oct 9, 2026

Uh oh!

DimensionWieldr commented Oct 10, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ianton-ru commented Aug 18, 2026 •

edited

Loading

github-actions Bot commented Aug 18, 2026 •

edited

Loading

DimensionWieldr commented Sep 22, 2026 •

edited

Loading

DimensionWieldr commented Oct 2, 2026 •

edited

Loading