test_num_rows_cache_no_collision_across_buckets fails on every run since #2183 #2264

Describe the situation

test_storage_iceberg_multistorage/test.py::test_num_rows_cache_no_collision_across_buckets fails on every run of antalya-26.6 since 2026-08-21. It is not flaky.

test_storage_iceberg_multistorage/test.py:605
    assert cache_lookups_a >= 1
E   assert 0 >= 1

The counts themselves are correct — only the assertion that the num-rows cache was consulted fails.

Evidence

Daily results for this test on antalya-26.6 — CI database query:

Bisected using branch runs only (pull_request_number = 0, no PR code involved) — CI database query:

The same four jobs in both runs: arm_binary, distributed plan, 3/4, amd_msan, 5/10, amd_asan_ubsan, db disk, old analyzer, 3/8, amd_tsan, 1/6.

Only two merges sit between them:

• 4b7cecaa3cf — Iceberg deletion vector support (attempt #2) #2183, 21/08 09:18
• 9bcfda40574 — Antalya 26.6 Backport of #109199 - Fix alter with custom disk #2246 ("Fix alter with custom disk"), 22/08 07:03 — touches AlterCommands.cpp and MergeTreeData.cpp only

The first failure was at 21/08 16:00, i.e. after #2183 and before #2246. The CI commits of that day (DNS, dataset mirror, proxy) are all later than 9bcfda40574 and therefore outside the window.

Confirmed locally by @CarlosFelipeOR: the release build of e1da95b648168e4fdcba804dd5ce98b3c84fca21 passes; the release build of #2183's head c113b0c95074c8cfc99f2c9a69e312bc1b9f5c79 fails with the same assertion.

Recent failing jobs on antalya-26.6 (branch runs, no PR):

• amd_msan, 5/10
• amd_asan_ubsan, db disk, old analyzer, 3/8
• amd_tsan, 1/6
• arm_binary, distributed plan, 3/4

Integration tests did not run on #2183. That is why this was not seen on the PR itself.

How to reproduce

python -m ci.praktika run "Integration tests (arm_binary, distributed plan, 1/4)" \
  --test "test_storage_iceberg_multistorage/test.py::test_num_rows_cache_no_collision_across_buckets"

Expected behavior

Either the test passes, or its assertion is updated to match intended behaviour.

Open question for the author: is this failure expected after #2183?

AI-generated analysis

Everything above is measured from CI logs, the CI database and a local run. The section below is AI-generated analysis of the source and has not been validated by instrumenting a build.

The server log for the failing query shows zero requests to the data buckets and Read 1 rows, 16.00 B — the count is answered entirely from Iceberg metadata, the data file is never opened, so the per-file num-rows cache is never consulted and the ProfileEvents counter stays at 0.

#2183 appears to have changed two things here, both deliberate, in response to #2236 / #2237 / #2238:

1. A metadata count path that sums per-file record_count from the manifests and subtracts position deletes, instead of trusting the snapshot summary's total-records (count() trusts a corrupted snapshot-summary total-records verbatim instead of the manifest row counts #2238).
2. canUseCountFromFilesCache(), which disables the per-file cache when its key (path + mtime) cannot represent the truth — live deletes, bucket subsets (Trivial count() optimization is applied to Iceberg tables with live delete files and trusts summary statistics over table data #2237).

The test strips value_counts and total-records so that count() must read the file, but it does not strip record_count — which is the field the new path uses. So the premise the test relies on no longer holds. If the reading above is right, this is a stale assertion rather than a defect.

Possible fixes

1. Drop the two cache_lookups >= 1 assertions. Two lines, but the test then passes even if the cache-key collision regressed — worth a comment saying the coverage was lost.
2. Rewrite the check without Iceberg (e.g. plain s3() over two buckets sharing an object key). The bug being guarded is in the generic num-rows cache key, not in Iceberg, and nothing short-circuits count() there.
3. Mark it in broken_tests.yaml as a stopgap so it stops polluting every run while a decision is made.

Note the test came from upstream ClickHouse#90740 and already failed there with this same assertion on 2026-08-07, on three builds, before it was merged.