Skip to content

feat: add support for dynamic KRaft quorum scaling - #1010

Open
razvan wants to merge 14 commits into
mainfrom
feat/kraft-dynamic-voter-membership
Open

feat: add support for dynamic KRaft quorum scaling#1010
razvan wants to merge 14 commits into
mainfrom
feat/kraft-dynamic-voter-membership

Conversation

@razvan

@razvan razvan commented Aug 18, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Description

Fixes #1009

See CHANGELOG for a high level view of everything that changed.

✅ : OKD integration tests work. I ran them many times but there was always one or two tests that failed due to flakiness. I added some cleanup steps so that kuttl doesn't timeout during namespace cleanups.

Definition of Done Checklist

  • Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
  • Please make sure all these things are done and tick the boxes

Author

  • Changes are OpenShift compatible
  • CRD changes approved
  • CRD documentation for all fields, following the style guide.
  • Helm chart can be installed and deployed operator works
  • Integration tests passed (for non trivial changes)
  • Changes need to be "offline" compatible
  • Links to generated (nightly) docs added
  • Release note snippet added

Reviewer

  • Code contains useful comments
  • Code contains useful logging statements
  • (Integration-)Test cases added
  • Documentation added or updated. Follows the style guide.
  • Changelog updated
  • Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

  • Feature Tracker has been updated
  • Proper release label has been added
  • Links to generated (nightly) docs added
  • Release note snippet added
  • Add type/deprecation label & add to the deprecation schedule
  • Add type/experimental label & add to the experimental features tracker

razvan and others added 6 commits August 18, 2026 14:24
@razvan @claude
feat: add quorum-manager sidecar for dynamic KRaft voter membership
571bf23 
Controllers now run a quorum-manager sidecar that admits itself into the
KRaft voter set on startup (add-controller) and removes itself before
termination (remove-controller via preStop), so controller role groups
can be scaled up/down on a running cluster without a full rolling
restart. controller.quorum.bootstrap.servers now points at each
controller role group's headless Service DNS name instead of individual
pod addresses, keeping container commands stable across replica changes.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@razvan @claude
feat: add startup/liveness/readiness probes for KRaft controllers
059e7d9 
Controllers get a startupProbe (plain TCP, generous failure threshold
for slow metadata-log replay on boot), a plain-TCP livenessProbe, and a
readinessProbe that checks the node's Raft state via its metrics
endpoint instead of a bare TCP check, so a controller stuck rejoining
the quorum is correctly reported as not ready.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@razvan @claude
feat: use OrderedReady pod management for controller StatefulSets
f98c117 
Controller pods now start/scale sequentially (OrderedReady) instead of
in parallel, since the quorum-manager sidecar's admission flow assumes
one voter joins at a time. Brokers are unaffected and keep Parallel.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@razvan @claude
fix: reject and gracefully handle KRaft controllers scaled to zero
645061b 
Scaling controllers to 0 replicas while brokers keep running is now
rejected at validation time with an actionable error, instead of
failing much later and confusingly while building the broker's
ConfigMap. Scaling controllers and brokers to 0 together (a coordinated
whole-cluster stop) is still allowed and now actually builds, since
downstream resource builders no longer assume a non-empty controller
quorum whenever KRaft mode is active.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@razvan @claude
test: cover KRaft controller scale-up/down/shutdown in kuttl operatio…
930a497 
…ns tests

Enable the previously version-gated scale-up/down steps (Kafka 3.7 no
longer needs special-casing), assert quorum voter counts via
kafka-metadata-quorum.sh after each scale, and add a final step scaling
both controllers and brokers to 0 to exercise the whole-cluster-stop
path before namespace teardown.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@razvan @claude
docs: document KRaft dynamic voter membership and update CHANGELOG
3074345 
Update the KRaft controller usage guide for scale-up/down support,
record the design spec and implementation plan, add the CHANGELOG
entries for this branch's changes, and ignore .worktrees/ for local
worktree checkouts.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@razvan razvan self-assigned this Aug 18, 2026
@razvan

razvan commented Aug 18, 2026

Copy link
Copy Markdown
Member Author

The failed test was due to namespace deletion timeout

--- FAIL: kuttl (2409.35s)
    --- FAIL: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/logging_kafka-3.9.2_zookeeper-latest-3.9.5_openshift-false (134.90s)
        --- FAIL: kuttl/harness/smoke-kraft_kafka-kraft-4.2.1_openshift-false (634.74s)
        --- PASS: kuttl/harness/smoke_kafka-3.9.2_zookeeper-3.9.5_use-client-tls-false_openshift-false (81.56s)
        --- PASS: kuttl/harness/upgrade_upgrade_old-3.9.2_upgrade_new-4.2.1_use-client-tls-false_use-client-auth-tls-false_openshift-false (120.71s)
        --- PASS: kuttl/harness/kerberos_kafka-3.9.2_zookeeper-latest-3.9.5_openshift-false_krb5-1.21.1_kerberos-realm-PROD.MYCORP_kerberos-backend-mit_broker-listener-class-cluster-internal_bootstrap-listener-class-external-unstable (113.64s)
        --- PASS: kuttl/harness/tls_kafka-3.9.2_zookeeper-latest-3.9.5_use-client-tls-false_use-client-auth-tls-false_openshift-false (33.64s)
        --- PASS: kuttl/harness/cluster-operation_kafka-latest-3.9.2_zookeeper-latest-3.9.5_openshift-false (41.49s)
        --- PASS: kuttl/harness/configuration_kafka-latest-3.9.2_openshift-false (11.45s)
        --- PASS: kuttl/harness/operations-kraft_kafka-kraft-4.2.1_openshift-false (1079.48s)
        --- PASS: kuttl/harness/opa_kafka-latest-3.9.2_zookeeper-latest-3.9.5_opa-latest-1.16.2_use-opa-tls-false_openshift-false_krb5-1.21.1 (128.33s)
        --- PASS: kuttl/harness/delete-rolegroup_kafka-3.9.2_zookeeper-latest-3.9.5_openshift-false (29.40s)
FAIL
ERROR:root:kuttl failed

@razvan
razvan marked this pull request as draft August 18, 2026 15:57
@razvan razvan mentioned this pull request Aug 20, 2026
Draft
20 tasks
@razvan
razvan marked this pull request as ready for review August 20, 2026 11:10
@razvan razvan moved this to Development: Waiting for Review in Stackable Engineering Aug 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@razvan razvan

Labels

None yet

Projects

Stackable Engineering
Status: Development: Waiting for Review

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Feat: KRaft quorum scalability

1 participant

@razvan