Antalya 26.8: pr-2159: combined port of 10 PRs

[zvonand]

Dropped from this backport: the AI dropped these surfaces rather than pulling in a missing prerequisite. Reviewers: confirm each is genuinely optional.

• ci/tests/test_print_stacktraces.py budget tests — file deleted on antalya-26.8 by 3a69545

Changelog category (leave one):

• Experimental Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Add experimental content-addressed storage (cas) for object-storage disks, which stores MergeTree parts as deduplicated blobs in a shared S3/GCS pool that multiple servers can mount, replicates by relinking existing blobs instead of copying them, and comes with lease-coordinated garbage collection, online integrity checks, SYSTEM CAS commands, system tables, clickhouse-disks tools, ALTER TABLE ... EXPORT PARTITION support and configurable mount lease timing. Also stop object-storage listings from parsing every listed key as a JSON command, which removes one thrown and caught exception per key on every object storage.
Add experimental content-addressed storage (cas) for object-storage disks, which stores MergeTree parts as deduplicated blobs in a shared S3/GCS pool that multiple servers can mount, replicates by relinking existing blobs instead of copying them, and comes with lease-coordinated garbage collection, online integrity checks, SYSTEM CAS commands, system tables, clickhouse-disks tools, ALTER TABLE ... EXPORT PARTITION support and configurable mount lease timing. Also stop object-storage listings from parsing every listed key as a JSON command, which removes one thrown and caught exception per key on every object storage (#2159 by @filimonov, #2300 by @filimonov, #2307 by @k-morozov, #2312 by @k-morozov, #2326 by @k-morozov, #2349 by @k-morozov, #2360 by @filimonov, #2384 by @filimonov, #2385 by @filimonov, #2393 by @filimonov).

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Performance tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All with Aarch64
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)
• Tiered Storage (2h)

Combined port of 10 PR(s) (group pr-2159). Cherry-picked from #2159, #2300, #2307, #2312, #2326, #2349, #2360, #2384, #2385, #2393.

• [experimental] CAS (Content-addressed storage) over shared object storage for antalya-26.6 #2159 — [experimental] CAS (Content-addressed storage) over shared object storage for antalya-26.6
• CAS improvements #2300 — CAS improvements
• CAS: configurable mount lease timing #2307 — CAS: configurable mount lease timing
• CAS: doc for GC round #2312 — CAS: doc for GC round
• CAS: gc scheduler stop join race #2326 — CAS: gc scheduler stop join race
• MemoryWorker: use sanitizer info for allocated counter #2349 — MemoryWorker: use sanitizer info for allocated counter
• Fix the coverage build: unused variable in gtest_cas_throttling_gate.cpp #2360 — Fix the coverage build: unused variable in gtest_cas_throttling_gate.cpp
• CI: cap the RustFS blocking-thread pool on the CAS-S3 lanes and collect its log #2384 — CI: cap the RustFS blocking-thread pool on the CAS-S3 lanes and collect its log
• CI: lldb stack budget by build flavor, and keep lldb's JIT breakpoint out of the server #2385 — CI: lldb stack budget by build flavor, and keep lldb's JIT breakpoint out of the server
• Temporary workaround: do not probe object-storage listing keys as JSON #2393 — Temporary workaround: do not probe object-storage listing keys as JSON

[github-actions]

Workflow [PR], commit [e330851]

[zvonand]

@blau-ai

[blau-ai]

CI triage for #2461 (combined CAS port, pr-2159 → antalya-26.8)

Summary — 6 failing checks: 3 PR-related (stateless suites), 3 not PR-related (Grype image scans).

Check	Verdict
Stateless tests (amd_debug, parallel)	PR-related
Stateless tests (arm_binary, parallel)	PR-related
Stateless tests (arm_binary, sequential)	PR-related
GrypeScanKeeper / Server / Server(-alpine)	Not PR-related (pre-existing base-image CVEs)

All of the stateless failures are in the feature this PR introduces (cas tests are brand-new here, so they can't be "pre-existing"), plus one pre-existing test (04812) that this PR regresses. None of them look like generic flakiness — the two that ran the randomized-settings diagnosis both came back reproducible. Evidence and concrete fixes below.

Report: result_pr.json for e330851.

---

1. 04812_clickhouse_disks_move_directory — PR-caused regression, clean fix ✅

(arm_binary parallel; deterministic — the diagnosis re-ran it 18× and it failed 18/18, "fails without randomization too")

Reason in the report is return code: 83, not a reference diff — the test script exits nonzero.

Root cause: this PR (cherry-pick of #2159) adds exit-code propagation to programs/disks/DisksApp.cpp:

if (query.has_value() && last_command_exit_code != 0)
    return last_command_exit_code;

last_command_exit_code does not exist on antalya-26.8 — before this PR, clickhouse-disks --query always returned EXIT_OK (0). The 04812 test explicitly depends on that old contract — its own comment says so:

# clickhouse-disks always exits 0, so assert the disk state.

…and it deliberately issues commands that error (e.g. move src aa/bb/cc into a missing parent fails with ENOENT on a local disk; remove -r can't empty a dir holding a virtual child on plain_rewritable). With the new behavior those surface as a nonzero process exit (83) and fail the test. The test asserts disk state via list, it does not care about exit codes.

Suggested fix (restore the contract for this test; stdout/reference are unchanged):

 function disks()
 {
-    clickhouse-disks -C "$config" --disk "$disk" --query "$1" 2>/dev/null
+    clickhouse-disks -C "$config" --disk "$disk" --query "$1" 2>/dev/null || true
 }

Alternative if you'd rather not touch a test owned by #2379: scope the new exit-code propagation to the cas-* commands only (the stated motivation was CI-gating cas-fsck), rather than making it global for every clickhouse-disks --query.

---

2. 04295_cas_mutation_no_leftovers — PR-related; points at a real CAS + packed-parts gap ⚠️

(arm_binary sequential; diagnosis: passes 16/16 without randomization, culprit --min_bytes_for_full_part_storage 399407315)

-grew_above_baseline 1
+grew_above_baseline 0 (baseline=0 after_insert=0)

The test inserts 4×100k rows and asserts the CAS pool directory grew (find …/ca/{blobs,packs,trees}). With a high min_bytes_for_full_part_storage, parts are written as packed parts (DataPartStorageOnDiskPacked) and no objects land in the counted pool subdirs — see finding #3, which is a crash in exactly that packed path. A one-line test fix (SETTINGS … min_bytes_for_full_part_storage = 0) would turn it green, but that would mask the packed-part behavior rather than fix it. I'd recommend confirming whether CAS is meant to support packed parts before pinning the setting.

---

3. Server logical error (crash) — most serious, PR-related 🔴

(amd_debug parallel — this is why that suite also shows "Server liveness check failed": the server aborts)

Logical error: 'Uncommitted shared transaction already exists'.
src/Storages/MergeTree/DataPartStorageOnDiskPacked.cpp:520: DB::DataPartStorageOnDiskPacked::beginTransaction()
src/Storages/MergeTree/MergeTask.cpp:630:  ExecuteAndFinalizeHorizontalPart::prepare()
…  DB::StorageMergeTree::optimize(...)   ← triggered by OPTIMIZE / a merge

beginTransaction() throws when transaction || writer is already set and has_shared_transaction is true — i.e. a packed part on a CAS disk gets beginTransaction() called while a shared transaction is already open, during a merge. has_shared_transaction is a CAS addition, so this path is new in this PR. In a debug build the LOGICAL_ERROR aborts the server, which cascades into the liveness failure and masks the rest of the amd_debug run.

This one needs the CAS authors — it can't be verified/fixed statically here (no local build/run in this environment). Same DataPartStorageOnDiskPacked packed path as #2, so I'd treat #2 and #3 as one underlying issue: CAS × packed-part merges.

---

4. 05008_cas_gc_snapshot_prune, 05009_cas_event_log, 05023_cas_dropns_leaked_namespace — PR-related 🔴

(failed in both amd_debug and arm_binary parallel — reproducible across arches, and arm had no crash, so these are genuine behavior diffs, not just crash fallout)

All three are GC/blob-accounting reference diffs in the new feature:

• 05009: has_blob_put 1 → 0 (no blob-put recorded)
• 05023: captured_rows_absent_after_gc_fixpoint 1 → 0
• 05008: -1 1 → 0 1

These are the CAS GC/event-log oracles disagreeing with the ported runtime. Like #2/#3 the recurring theme is blob/pool objects not appearing as the tests expect — plausibly the same packed/compact-part storage interaction. Needs the feature authors to confirm whether it's a port regression or a test-robustness gap; I can't settle it without running the server.

---

5. Grype scans (keeper / server / server-alpine) — NOT PR-related

fail = "image contains N high/critical CVEs" from the base OS/library layers (keeper: 4, server-alpine: 1, server: 0-but-policy-fail). This PR changes only C++/tests/CI config — no docker/, contrib/, or requirements.txt changes — so it introduces none of these, and they fail branch-wide. Safe to ignore for this PR / handle separately.

---

What I'd do next

• Update README.md #1 is a ready, safe fix — happy to apply it.
• Update README.md #2–Update README.md #4 are in the CAS runtime (packed parts + GC) and need @filimonov / @k-morozov; Update README.md #2/Update README.md #3 look like one bug (CAS × packed-part merge), Update README.md #4 like the GC oracles.

I have not pushed anything. Want me to open a blau/* PR with the 04812 fix (and, if you confirm it's acceptable to mask, the 04295 setting pin), or commit the 04812 fix directly to feature/antalya-26.8/pr-2159?

I can't build or run ClickHouse in this environment — findings are from the praktika S3 report + static reading; any fix is validated by CI on the PR.

[zvonand]

waiting from input from @filimonov on test fails related to new packed parts

[zvonand]

Superseded: #2159 is now ported as part of group auto-grp-pr-2159.