fix(kubelet): remove streamingConnectionIdleTimeout for k8s >= 1.34

[abigailliang-aks-sig-node]

Summary

Remove streamingConnectionIdleTimeout from both the kubelet command line and the kubelet config file for Kubernetes >= 1.34, where this field was removed from KubeletConfiguration.

Context: RP-side flow (scriptless path)

Step 1: defaults_kubelet.go — removeKubeletFlags()
        └── k8s >= 1.34: delete(kubeletConfig, "--streaming-connection-idle-timeout")
                ↓
        kubeletConfig map (flag is already gone)
                ↓
Step 2: scriptless_direct.go — processKubeletConfigFileContent()
        └── cfg.StreamingConnectionIdleTimeout = kubeletFlags["--streaming-connection-idle-timeout"]
            → empty string (key was deleted in step 1)
                ↓
Step 3: scriptless_direct.go — processKubeletFlags()
        └── filterKubeletFlags() strips translated flags when enableKCF=true
            → flag not on command line either way
                ↓
Step 4: Proto sent to aks-node-controller
        └── KubeletConfigFileConfig.StreamingConnectionIdleTimeout = "" (omitted by protojson)

RP already handles this in removeKubeletFlags(). However, the field may still arrive in KubeletConfigFileConfig from:

• Older RP versions that haven't rolled out yet
• The GetAKSNodeConfig (NBC conversion) path where the removal may not be applied

Changes (defense-in-depth)

File	Change
aks-node-controller/helpers/utils.go	ValidateAndSetLinuxKubeletFlags(): delete --streaming-connection-idle-timeout for >= 1.34
aks-node-controller/parser/parser.go	getCSEEnv(): clear KubeletConfigFileConfig.StreamingConnectionIdleTimeout for >= 1.34 before marshaling
pkg/agent/baker.go	ValidateAndSetLinuxNodeBootstrappingConfiguration(): same flag deletion for the old NBC path

Why both flag deletion AND config file clearing?

• Flag deletion (helpers/utils.go, baker.go): covers enableKCF=false case where translated flags stay on the command line
• Config file clearing (parser.go): covers enableKCF=true case where the field would appear in kubeletconfig.json if RP sets it

Test plan

• unit tests
• local e2e tests
• CI passing: https://github.com/Azure/AgentBaker/actions/runs/29296756647/job/86971871744?pr=8812

[Copilot]

Pull request overview

This PR aims to prevent streamingConnectionIdleTimeout from being propagated to kubelet for Kubernetes >= 1.34, where the corresponding KubeletConfiguration field/flag has been removed, by stripping it from kubelet CLI flags and clearing it from the kubelet config-file content.

Changes:

• Delete --streaming-connection-idle-timeout from kubelet flags for Kubernetes >= 1.34 in both the legacy NodeBootstrappingConfiguration path and the aks-node-controller helper path.
• Clear KubeletConfigFileConfig.StreamingConnectionIdleTimeout for Kubernetes >= 1.34 before marshaling kubelet config-file JSON in aks-node-controller.
• Add version gating comments explaining why the field must not appear for 1.34+.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
pkg/agent/baker.go	Removes --streaming-connection-idle-timeout from kubelet flags for k8s >= 1.34 in the NodeBootstrappingConfiguration validation path.
aks-node-controller/parser/parser.go	Clears StreamingConnectionIdleTimeout before kubelet config-file JSON marshaling for k8s >= 1.34.
aks-node-controller/helpers/utils.go	Removes --streaming-connection-idle-timeout from kubelet flags for k8s >= 1.34 in helper validation.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

[abigailliang-aks-sig-node]

this change block due to gofmt/goimports column alignment.

When a struct literal contains multiple fields, gofmt attempts to align the values ​​following the colons into the same column. The width of the aligned column is determined by the longest field name in the group:

[abigailliang-aks-sig-node]

same gofmt behave

[github-actions]

The latest Buf updates on your PR. Results from workflow Buf CI / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
✅ passed	✅ passed	✅ passed	✅ passed	Jul 18, 2026, 3:43 AM

[aks-node-assistant]

AgentBaker Linux gate detective

Run: https://msazure.visualstudio.com/CloudNativeCompute/_build/results?buildId=172264217
Failed job/stage/task: �2e / Run AgentBaker E2E / Run AgentBaker E2E

TL;DR: Ubuntu 24.04 Gen2 E2E package validation reached the node and then failed because expected moby-containerd 2.3.2-ubuntu24.04u1 was missing from installed packages.

Likely cause/signature: deterministic VHD/package-content or expectation skew ($e2eSig).

Confidence: High.

Recommended owner/action: VHD/package owner should compare Ubuntu 24.04 package feed/build inputs against installed package inventory and either fix the package source/image content or correct expected metadata.

Strongest alternative: test expectation advanced before package/feed availability; less likely to be PR-change-caused because PR #8812 changes kubelet streaming idle timeout logic, not containerd packaging.

Evidence: E2E est-log.json reports the package assertion; sibling Ubuntu 24.04 Gen2 leaves show the same signature; retry evidence is deterministic rather than transient.

Wiki signature: ubuntu2404-gen2-moby-containerd-232-ubuntu2404u1-missing

[abigailliang-aks-sig-node]

AgentBaker Linux gate detective

Run: https://msazure.visualstudio.com/CloudNativeCompute/_build/results?buildId=172264217 Failed job/stage/task: �2e / Run AgentBaker E2E / Run AgentBaker E2E

TL;DR: Ubuntu 24.04 Gen2 E2E package validation reached the node and then failed because expected moby-containerd 2.3.2-ubuntu24.04u1 was missing from installed packages.

Likely cause/signature: deterministic VHD/package-content or expectation skew ($e2eSig).

Confidence: High.

Recommended owner/action: VHD/package owner should compare Ubuntu 24.04 package feed/build inputs against installed package inventory and either fix the package source/image content or correct expected metadata.

Strongest alternative: test expectation advanced before package/feed availability; less likely to be PR-change-caused because PR #8812 changes kubelet streaming idle timeout logic, not containerd packaging.

Evidence: E2E est-log.json reports the package assertion; sibling Ubuntu 24.04 Gen2 leaves show the same signature; retry evidence is deterministic rather than transient.

Wiki signature: ubuntu2404-gen2-moby-containerd-232-ubuntu2404u1-missing

fix at #8959

[Copilot]

Pull request overview

Copilot reviewed 14 out of 15 changed files in this pull request and generated 1 comment.

Files not reviewed (1)

• aks-node-controller/pkg/gen/aksnodeconfig/v1/kubelet_config.pb.go: Generated file

[aks-node-assistant]

Detective summary

Run: https://msazure.visualstudio.com/CloudNativeCompute/_build/results?buildId=172411891

Failed job/stage/task: Build VHD across the VHD matrix during Packer azure-arm ARM deployment validation.

TL;DR: the gate failed before image provisioning because ARM validation hit QuotaExceeded for standardDDSv5Family cores in WestUS3. The missing BCC log publish failures are downstream/collateral.

Likely cause / signature

• Signature: Packer azure-arm → InvalidTemplateDeployment → QuotaExceeded for standardDDSv5Family cores → packer.mk:38 → packer.mk:98 → exit code 2.
• Likely cause: infrastructure/regional VM-family quota exhaustion in the VHD gate subscription/region, not the kubelet flag changes in PR fix(kubelet): remove streamingConnectionIdleTimeout for k8s >= 1.34 #8812.
• Classification: Infrastructure / Packer capacity failure.
• Confidence: Very high.
• Strongest alternative: PR changed SKU/matrix demand; less likely because changed files are kubelet/ANC/parser-related and ARM rejects quota before repo code runs.

Recommended owner / action

Node Lifecycle / VHD gate infra owner: increase standardDDSv5Family quota in WestUS3 or reduce/throttle concurrent matrix jobs using that family; re-run after capacity is restored.

Evidence

• Timeline: multiple Build VHD task failures; BCC log publish failures are secondary.
• Logs: Build VHD logs 191, 199, and 221 show InvalidTemplateDeployment / QuotaExceeded for standardDDSv5Family.
• Wiki signature: linux-vhd-packer-westus3-vm-family-quota-exceeded

[Copilot]

Pull request overview

Copilot reviewed 14 out of 15 changed files in this pull request and generated no new comments.

Files not reviewed (1)

• aks-node-controller/pkg/gen/aksnodeconfig/v1/kubelet_config.pb.go: Generated file

Comments suppressed due to low confidence (2)

pkg/agent/utils.go:424

• This comment is misleading: the function still mutates the caller's custom config by merging in missing default flags (see the loop just above). Suggest rewording to clarify that only deprecated-flag removal is avoided, not all mutation.

	// Filter out deprecated flags at output time rather than mutating the caller's CustomConfiguration.

aks-node-controller/helpers/utils.go:235

• The word "KubeletConfiguration" is broken across two comment lines ("Kube" / "letConfiguration"), which reads like a typo in rendered docs and makes grepping harder. Rewrap the comment without splitting the identifier.

	// streamingConnectionIdleTimeout was removed from Kube
	// letConfiguration in k8s 1.34+.
	// It must not appear on the command line or in the config file for those versions.