Skip to content

fix(kubernetes): recover compute driver watches - #2842

Open
jhjaggars wants to merge 1 commit into
NVIDIA:mainfrom
jhjaggars:fix-kubernetes-watch-recovery/jj
Open

fix(kubernetes): recover compute driver watches#2842
jhjaggars wants to merge 1 commit into
NVIDIA:mainfrom
jhjaggars:fix-kubernetes-watch-recovery/jj

Conversation

@jhjaggars

Copy link
Copy Markdown
Contributor

Summary

Keep Kubernetes compute-driver watches alive when kube-runtime emits recoverable watcher errors. This lets kube-runtime relist after expired resource versions instead of forcing the gateway to recreate the driver stream and introducing an avoidable watch gap.

Related Issue

No issue required: this is an obvious localized bug fix for periodic Kubernetes watcher recovery failures.

Changes

  • Apply kube-runtime's default watcher backoff to Sandbox and Kubernetes Event watches
  • Log and consume emitted watcher errors so continued polling drives kube-runtime's recovery state machine
  • Preserve stream closure and receiver cancellation behavior in single-namespace and cluster-wide modes
  • Distinguish Sandbox-resource and Kubernetes Event watcher failures in structured logs
  • Add 410 Expired-to-Restarted coverage through the mpsc-backed outward compute-driver stream
  • Document in-place watcher recovery in the Kubernetes driver README

Testing

  • mise run pre-commit passes
  • Unit tests added/updated
    • cargo test -p openshell-driver-kubernetes (211 passed)
    • cargo test -p openshell-driver-kubernetes outward_watch_stream_survives_expired_error_and_backoff_recovery -- --nocapture
  • E2E tests added/updated (not applicable; recovery is covered with paused-time watcher stream unit tests)

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
@copy-pr-bot

copy-pr-bot Bot commented Aug 20, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant