[BUG] Upgrade Failures: Lack of Persistent Webhook and Multi-Replica Support

Summary

We are self-hosting Kubefleet and have encountered critical issues impacting pod
creation and upgrade reliability. These issues hinder our ability to meet
operational requirements as a regulated industry. Below, we outline the challenges
and propose potential solutions.

Issues and Observations

1. Pod Creation Blocked Outside Fleet-`* and Kube-`* Namespaces

The ValidatingWebhookConfiguration blocks the creation of pods in namespaces
other than fleet-* and kube-*.
As a regulated industry, we require agents and other critical workloads to run in
custom namespaces, but the webhook prevents this.
Manual patching of the webhook (e.g., setting failurePolicy: Ignore) is required
to unblock pod creation, which is not sustainable.

As a workaround, we have resorted to creating more namespaces with fleet-* and
kube-* prefixes. However, this approach is not scalable and introduces
operational complexity. For instance, CustomResourcePlacement (CRP) objects fail
to be created in the hub namespace (e.g., hub-namespace) but are successfully
created in member namespaces (e.g., member-namespace-1).

Alternatively, if we use a fleet-* namespace, the CRP throws the following error:
This inconsistency leads to:
- Increased namespace sprawl, making cluster management more challenging.
- Potential conflicts with existing naming conventions and policies.
- Difficulty in maintaining compliance with regulatory requirements.
What do you propose as a sustainable solution to address this limitation, and what
are the potential implications of continuing with this workaround?

2. Upgrade Failures for Node Image and Kubernetes Upgrades

During node image and Kubernetes upgrades, we observe failures due to the
inability of the hub-agent or other pods to come up because of the
ValidatingWebhookConfiguration.
The two alternatives we tried are as follows:
1. Webhook Configuration:
  - The ValidatingWebhookConfiguration must be manually patched after every
    upgrade or node pool eviction to unblock pod creation.
  - Manual changes are lost because the controller deletes and recreates the webhook
    configuration using a delete-and-create pattern:
```
if err := w.mgr.GetClient().Create(ctx, &validatingWebhookConfig); err != nil {
      if !apierrors.IsAlreadyExists(err) {
            return err
      }
      // Overwrite: delete and recreate
      if err := w.mgr.GetClient().Delete(ctx, &validatingWebhookConfig); err != nil {
            return err;
      }
      if err = w.mgr.GetClient().Create(ctx, &validatingWebhookConfig); err != nil {
            return err;
      }
}
```
2. Multi-Replica Hub-Agent:
  - To improve high availability, we added a rolling update strategy, set replicas
    to 5, and configured a PodDisruptionBudget.
  - Certificates are hardcoded in the webhook server setup, as seen in
    cmd/hubagent/main.go,
    where FleetWebhookCertDir and FleetWebhookPort are used without dynamic
    synchronization.
  - Each replica must use the same TLS certificate stored in FleetWebhookCertDir.
    If the certificates are not synchronized across replicas, the API server might
    receive different certificates from different replicas, leading to TLS handshake
    failures.
  - When multiple replicas are behind a Kubernetes Service, the API server's
    requests are load-balanced across the replicas. If the replicas do not share the
    same certificate, the API server might encounter inconsistent certificates
    during successive requests.
  - The API server logs errors like:
```
x509: certificate signed by unknown authority
```
    This disrupts the admission process, potentially causing delays or failures in
    pod creation.
  - The caBundle field in the ValidatingWebhookConfiguration is critical for
    ensuring the API server trusts the webhook service. For more details, refer to
    the Kubernetes documentation on
    Admission Webhooks.

Proposed Solutions

Persistent Webhook Configuration:
- Support persistent configuration for webhook fields (e.g., failurePolicy, ) to
  avoid manual patching.
- Can other namespaces be ignored to simplify webhook configuration and reduce
  operational overhead?
Multi-Replica Hub-Agent:
- Address TLS handshake errors by:
  - Using a shared certificate source (e.g., Kubernetes secrets) to prevent
    mismatches.
- Wire the replicas and strategy fields into the hub-agent deployment and add
  PodDisruptionBudgets for upgrade reliability.
Namespace Flexibility:
- Allow pod creation in custom namespaces by updating the webhook configuration to
  support broader namespace scopes.

Request and Questions

Can you assist in resolving the TLS issues with multiple replicas to ensure
upgrades do not fail?
Would you be open to supporting persistent webhook configuration and namespace
flexibility?

Divya1388

Divya1388 , Thank you very much for raising the issue. I wonder if you actually used azure/fleet or kubefleet (https://github.com/kubefleet-dev/kubefleet) which is the CNCF project. This repo is served as the backend for the azure kubernete fleet manager (https://docs.azure.cn/en-us/kubernetes-fleet/). They are close but not entirely the same.

Pod Creation Blocked Outside Fleet-* and Kube-* Namespaces

This is by design, the hub cluster is not designed to run workload. I wonder if you have any reason to run the pod in the hub cluster?

I wonder if you can describe to us your use cases?

Upgrade Failures for Node Image and Kubernetes Upgrades

We will look into this and get back to you.

Divya1388

Possibly related: kubefleet-dev/kubefleet#290

Divya1388 is on the same team as me. We are using the CNCF version to self-host. We recognize that this issue has perhaps been filed in the wrong repo. Sorry, it gets confusing.

Use cases are outlined in the other issue. Things like:

Prometheus
OpenTelemetry
Security scanning software (Wiz, Crowdstrike)

If it's an Azure-owned product (like a "packaged thing") then we have less need for including the extra monitoring and security items because we can "trust Microsoft." If it's self-hosted, we have to treat it like a standard Kubernetes cluster and include all the extra bits.

Probably good if we pick up over there to avoid confusion.
kubefleet-dev/kubefleet#290

Ryan Zhang (@ryanzhang-oss)

Hi Ryan Zhang (@ryanzhang-oss)
Thanks for looking into this!
As Travis mentioned, things are a bit confusing due to the presence of two repositories.
For the issues outlined above, we've conducted a POC using both:

Regarding your question I wonder if you have any reason to run the pod in the hub cluster?
There are couple considerations:

As a regulated industry, we need to install enterprise monitoring software on each of the clusters we provision.
We've observed the following behavior:
- When creating a CRP with a namespace that does not start with fleet-* or kube-*, the objects are created in member clusters, but the deployments fail in the hub.
- Conversely, when the namespace does start with fleet-* or kube-*, we encounter the following error:

Could you please advise on the recommended strategy for handling this? We're trying to determine the best approach moving forward.
Thanks again!

Hi, I'm on the same team as Divya and Travis.

We tested another approach by disabling the webhook while keeping the hub-agent's replicaCount at 1. As expected, without the ValidatingWebhookConfiguration, there was nothing preventing pods from being initiated. The Kubernetes version upgrade completed successfully. However, I'm still unsure about the potential implications of disabling the webhook.

Interestingly, even though the webhook was disabled, its Kubernetes Service was still created. It's unclear if this service is actually required in this scenario.

Wei Weng (@weng271190436)

Based on our discussion Wei Weng (@weng271190436) , I wanted to highlight an issue we're encountering when attempting to delete or unjoin member clusters. When we try to remove a membercluster resource, the deletion fails and we're seeing errors in the hub-agent logs

I1118 16:26:51.614247       1 membercluster/membercluster_validating_webhook.go:57] "Validating webhook member cluster DELETE" memberCluster="/member-0"
I1118 16:26:51.618015       1 rest/request.go:1632] body was not decodable (unable to check for Status): couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string "json:\"apiVersion,omitempty\""; Kind string "json:\"kind,omitempty\"" }
E1118 16:26:51.618536       1 membercluster/membercluster_validating_webhook.go:61] "Failed to list internalServiceExportList when validating" err="no matches for kind \"InternalServiceExport\" in version \"networking.fleet.azure.com/v1alpha1\""
I1118 16:26:52.067456       1 fleetresourcehandler/fleetresourcehandler_webhook.go:88] "handling namespaced resource in fleet reserved namespaces" GVK="apps/v1, Kind=DaemonSet" namespacedName="kube-system/azure-ip-masq-agent" operation="UPDATE" subResource=""

This is a critical concern for our operations going forward. If we need to deprecate or decommission clusters and remove their support from the hub-agent by deleting the MemberCluster resource, this could become a significant blocker.

We've already attempted several troubleshooting steps, including patching the webhook and removing the CRD, but neither approach has resolved the issue. Could you please take a look at this when you have a chance?

cc - Divya1388 Ryan Zhang (@ryanzhang-oss)

Krunal Hingu (@krunalhinguu)

Krunal Hingu (@krunalhinguu) , thank you very much for your interests in fleet. I wonder if you can clarify on how did you create the fleet? Are you creating an upstream fleet using kubefleet or you are using the Azure fleet manager? The error you got seems to indicate that you use kubefleet. This error is caused by the related network agents and their CRD (https://github.com/Azure/fleet-networking/) are not installed.

Travis Illig (@tillig)

I think we need more sessions to go over all the issues.
btw, Travis Illig (@tillig), we now have exactly what you were looking for a few months back. We now allow you to have a separte rollout for a deployment inside a namespace. You can now place namespaced resources individualy. Arvind Thirumurugan (@Arvindthiru) can give you a session on exactly how that works

Ryan Zhang (@ryanzhang-oss)

Ryan Zhang (@ryanzhang-oss) Thank you for response.

That's correct. We are using the upstream fleet using kubefleet for our implementation. Regarding the networking CRDs, We assumed they were optional and therefore didn't install them.

Could you confirm if there's a way to handle this without installing fleet-networking, or if these CRDs are mandatory for basic fleet operations?

it will be better to install the network agents since the hub agent assumed that we have the network agents. However, we designed the leave behavior with an opt out option.

https://github.com/kubefleet-dev/kubefleet/blob/4ce199e9d99bc961c39fe95cdd0593002cf208a0/apis/cluster/v1beta1/membercluster_types.go#L100C2-L100C16

if you set the deleteOption as
// DeleteValidationModeSkip skips the validation when deleting a MemberCluster.
DeleteValidationModeSkip = "Skip"

then we will not check the networking CRD.

Ryan Zhang (@ryanzhang-oss)

Thanks for pointing this out, Ryan Zhang (@ryanzhang-oss). This is very helpful.

Just to confirm, is this what you're suggesting?

apiVersion: cluster.kubernetes-fleet.io/v1beta1
kind: MemberCluster
metadata:
  name: <member-cluster-name>
spec:
  identity:
    name: <name>
    kind: ServiceAccount
    namespace: <namespace>
    apiGroup: ""
  heartbeatPeriodSeconds: 15
  deleteOptions:
    validationMode: "Skip"

Krunal Hingu (@krunalhinguu)

Krunal Hingu (@krunalhinguu)

My understanding is that Wei Weng (@weng271190436) suggested using the Azure Fleet repository for our installations instead of the upstream Fleet. Ryan Zhang (@ryanzhang-oss) and Wei Weng (@weng271190436) , could you please confirm which Helm chart we should be installing?

We have demonstrated how we pull Helm charts and push them to our internal Helm repositories over a call. However, the upstream repository presents the following challenges:

Lack of tags
Extensive use of symlinks
Lack of support for beta1 APIs in hub agent CRD's
Presence of conflicting APIs, which we have to manually clean up

These issues are making the process more complex for us.

Divya1388

Divya1388 , please use the kubefleet repo helm chart. To address your concerns

I just created one release.
I am not sure why that is a problem. Does the helm install command not work?
I am not sure what does "support" mean for a community driven CNCF project. We have support for Azure fleet if you need support.
I wonder if you can give an example of "conflicting API"?

Thank you very much.

If you have more questions, can you please leave more questions in our slack channel?

Or join our community calls in APAC or US/EU time?

Ryan Zhang (@ryanzhang-oss)

Ryan Zhang (@ryanzhang-oss) , I discussed with Wei Weng (@weng271190436) regarding this support issue. Currently, we rely on the kubefleet-dev repository for the charts, but for release tags, we use azure/fleet. Since these two repositories are not in sync, it would be helpful to start tagging the kubefleet-dev repository as well. This way, we can rely on a single source of truth, simplifying our workflow moving forward.

Ryan Zhang (@ryanzhang-oss)

Ryan Zhang (@ryanzhang-oss) i will get on slack, i think Krunal is covering our team for now. Thanks for all the support

Follow up on your questions:

I am not sure why that is a problem. Does the helm install command not work?

Helm repositories does not support symlinks and within our company we are only allowed to helm install from a internal helm repository

I am not sure what does "support" mean for a community driven CNCF project. We have support for Azure fleet if you need support.

I meant this, thanks for taking care

Presence of conflicting APIs, which we have to manually clean up

I honestly do not have screenshot of the error now. But when we were POC'ing we had challenges with v1beta1 and v1 and v1alpha1 schemas being present in the CRD's schema definitions, For Example: https://github.com/kubefleet-dev/kubefleet/blob/main/config/crd/bases/placement.kubernetes-fleet.io_clusterapprovalrequests.yaml#L35
I will try to re-collect the error but this is something we discussed with Wei Weng (@weng271190436)

Re: helm install - bugs already filed for fixing up the charts:

[BUG] Helm charts do not pass helm lint kubefleet-dev/kubefleet#212
[BUG] Helm charts incorrectly attempt to create the namespace kubefleet-dev/kubefleet#213
Publish Helm charts to a Helm repo or OCI repository kubefleet-dev/kubefleet#214
[BUG] Hub agent Helm chart hardcodes service account name kubefleet-dev/kubefleet#278

Re: conflicting APIs, I had a related bug here: kubefleet-dev/kubefleet#103

Divya1388

Divya1388 we just published v0.1.1 from kubefleet with a new enableWorkload=true option, which allows pod and replicaset deployments in all namespaces in hub. https://github.com/kubefleet-dev/kubefleet/tree/v0.1.1
We published the docker images to ghcr.io. https://github.com/orgs/kubefleet-dev/packages
ghcr.io/kubefleet-dev/kubefleet/hub-agent:v0.1.1
ghcr.io/kubefleet-dev/kubefleet/member-agent:v0.1.1
ghcr.io/kubefleet-dev/kubefleet/refresh-token:v0.1.1

I also sent thsi to CNCF slack earlier https://cloud-native.slack.com/archives/C08KR7589R8/p1764626643472149 tagging Krunal

Can you try to see if this fixes your top 2 issues?

Priority 1: Stable Node Image Upgrades
Node image upgrades are currently failing due to validating webhooks on pods and replica sets. This is blocking our ability to maintain cluster security and stability.

Priority 2: Monitoring Workloads and Namespace Support
We need to run monitoring workloads on the hub cluster and allow other namespaces, so CRP doesn't fail when we're not using envelope objects.

Note: the docker images we publish to ghcr only work for linux/amd64 currently but not for linux arm64 because of some issue between the cross compile tool we use and ghcr

Wei Weng (@weng271190436)

one question Wei Weng (@weng271190436) just to confirm going forward, can we expect Kubefleet tags to be published regularly as we do with azure/fleet and treated as the source of truth? I want to ensure we can rely on these tagged releases as a stable reference for our automation.

Krunal Hingu (@krunalhinguu)

Krunal Hingu (@krunalhinguu) yes you can expect kubefleet tags to be published regularly and we intend to make that the recommended way to use kubefleet.

Wei Weng (@weng271190436)

Perfect. Sounds good. Thanks for confirming Wei Weng (@weng271190436)

Ryan Zhang (@ryanzhang-oss)

Hi Ryan Zhang (@ryanzhang-oss), while testing the eviction of member-agent, I encountered an issue with the membercluster CRD having both v1 and v1beta1 versions. The v1 version, which lacked the deleteOptions property, was taking precedence over v1beta1, causing the process to fail during the first attempt. After recreating the cluster, we verified the CRDs beforehand and identified the problem. I resolved the issue by removing the v1 version, and it is now working as expected.

This aligns with the concerns Divya1388 previously highlighted related to crds. While we were able to resolve this issue, it took significant time to identify the root cause. Additionally, we are uncertain how many other CRDs might have similar issues, which prevents us from fully automating the process and necessitates manual intervention with extra testing efforts.

If you could look into this further, it would be greatly appreciated.

cc - Wei Weng (@weng271190436)

Travis Illig (@tillig)

Travis Illig (@tillig) Divya1388 , thank you for the report. We have fixed all the issues.

Krunal Hingu (@krunalhinguu)

Krunal Hingu (@krunalhinguu), I wonder if it would be possible to use v1beta1 CRDs instead of V1. We only add new optional fields in v1beta1.

Wei Weng (@weng271190436)

Thanks for fixing it guys Wei Weng (@weng271190436) Ryan Zhang (@ryanzhang-oss). It is really helpful. For now, this is all we have, but I will check with the team and thoroughly test this with v1beta1 CRDs to ensure everything is working correctly.

Navigation Menu

Repository navigation

[BUG] Upgrade Failures: Lack of Persistent Webhook and Multi-Replica Support #1224

Summary

Issues and Observations

1. Pod Creation Blocked Outside Fleet-`* and Kube-`* Namespaces

2. Upgrade Failures for Node Image and Kubernetes Upgrades

Proposed Solutions

Request and Questions

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[BUG] Upgrade Failures: Lack of Persistent Webhook and Multi-Replica Support #1224

Description

Summary

Issues and Observations

1. Pod Creation Blocked Outside Fleet-* and Kube-* Namespaces

2. Upgrade Failures for Node Image and Kubernetes Upgrades

Proposed Solutions

Request and Questions

Activity

ryanzhang-oss commented on Nov 7, 2025

tillig commented on Nov 7, 2025

Divya1388 commented on Nov 9, 2025

mchaves27 commented on Nov 12, 2025

krunalhinguu commented on Nov 18, 2025

ryanzhang-oss commented on Nov 18, 2025

ryanzhang-oss commented on Nov 18, 2025

krunalhinguu commented on Nov 18, 2025

ryanzhang-oss commented on Nov 18, 2025

krunalhinguu commented on Nov 18, 2025

Divya1388 commented on Nov 19, 2025

ryanzhang-oss commented on Nov 20, 2025

ryanzhang-oss commented on Nov 20, 2025

ryanzhang-oss commented on Nov 20, 2025

krunalhinguu commented on Dec 1, 2025

Divya1388 commented on Dec 1, 2025

tillig commented on Dec 1, 2025

weng271190436 commented on Dec 2, 2025

krunalhinguu commented on Dec 2, 2025

weng271190436 commented on Dec 2, 2025

krunalhinguu commented on Dec 2, 2025

krunalhinguu commented on Dec 4, 2025

ryanzhang-oss commented on Jan 22, 2026

ryanzhang-oss commented on Jan 22, 2026

krunalhinguu commented on Jan 26, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

1. Pod Creation Blocked Outside Fleet-`* and Kube-`* Namespaces