fix: ensure kubelet and containerd in kube.slice via systemd drop-ins by titilambert · Pull Request #8958 · Azure/AgentBaker

titilambert · 2026-07-15T19:38:16Z

What this PR does / why we need it:

When node hardening is enabled, ensureKubeletCgroupHierarchy() creates a dedicated kubelet.slice for kube-reserved cgroup enforcement. However the existing drop-in for kubelet.service only declared Wants= and After= ordering — it did not set Slice=kubelet.slice, so kubelet remained in system.slice. Additionally, containerd.service had no drop-in at all and was never placed in the slice.

This PR fixes both issues:

Adds Slice=kubelet.slice to the kubelet drop-in so kubelet actually runs inside the dedicated slice.
Creates an equivalent drop-in for containerd.service (10-kubelet-slice.conf) so containerd is co-located in the same slice.

Without this fix, cgroup resource accounting under kubelet.slice is incomplete — node hardening's kube-reserved budget doesn't capture the real resource usage of kubelet and containerd, which blocks nodes to be ready since they are affected to system.slice which has a low max memory limit

Testing:

Updated ShellSpec tests to validate both drop-ins are written with the
correct [Service] Slice=kubelet.slice directive.

 $ systemctl status containerd | grep "CGroup"
 CGroup: /kubelet.slice/containerd.service
 $ systemctl status kubelet | grep "CGroup"
 CGroup: /kubelet.slice/kubelet.service

Which issue(s) this PR fixes:

Fixes #

titilambert · 2026-07-15T19:38:17Z

This change is part of the following stack:

fix: ensure kubelet and containerd in kube.slice via systemd drop-ins #8958 ◀

_{Change managed by git-spice.}

Copilot

Pull request overview

This PR updates the Linux CSE helper responsible for Node Memory Hardening cgroup hierarchy setup so that both kubelet and containerd are configured to run under kubelet.slice, and adjusts ShellSpec coverage accordingly.

Changes:

Extend ensureKubeletCgroupHierarchy to set Slice=kubelet.slice for kubelet.service and add a matching drop-in for containerd.service.
Update ShellSpec assertions to validate the additional systemd drop-in content for containerd.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
`parts/linux/cloud-init/artifacts/cse_helpers.sh`	Adds systemd drop-ins to place kubelet and containerd into `kubelet.slice` when hardening cgroups are enabled.
`spec/parts/linux/cloud-init/artifacts/cse_helpers_spec.sh`	Updates tests to assert `Slice=` and validates the containerd drop-in is created.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

SriHarsha001

We need to look into the AI's comments, apart from that changes look good to me.

Copilot

Pull request overview

Copilot reviewed 11 out of 12 changed files in this pull request and generated no new comments.

Files not reviewed (1)

aks-node-controller/pkg/gen/aksnodeconfig/v1/kubelet_config.pb.go: Generated file

Comments suppressed due to low confidence (3)

pkg/agent/utils.go:512

isNodeHardeningEnabled() only strips surrounding '[' and ']' from --enforce-node-allocatable, but existing test data documents values that can be additionally wrapped in quotes (e.g. "[pods]"). In that case hardening detection will be skipped and the cgroup flags won’t be set/overwritten as intended.

	raw := strings.TrimSpace(kubeletFlags["--enforce-node-allocatable"])
	raw = strings.TrimPrefix(raw, "[")
	raw = strings.TrimSuffix(raw, "]")
	enforced := strings.Split(raw, ",")

parts/linux/cloud-init/artifacts/cse_config.sh:392

This logs_to_events key uses the ensureKubelet prefix even though it’s executed from ensureContainerd(), which can make event attribution/telemetry confusing when diagnosing provisioning failures.

      if ! logs_to_events "AKS.CSE.ensureKubelet.ensureKubeletCgroupHierarchy" ensureKubeletCgroupHierarchy; then

parts/linux/cloud-init/artifacts/cse_config.sh:888

This rewrites /etc/systemd/system/kubelet.service.d/10-containerd-base-flag.conf but does not ensure the drop-in directory exists first. On images/scenarios where kubelet.service.d isn’t present yet, tee will fail and kubelet may start without the intended --runtime-cgroups refresh.

    tee "/etc/systemd/system/kubelet.service.d/10-containerd-base-flag.conf" > /dev/null <<EOF
[Service]
Environment="KUBELET_CONTAINERD_FLAGS=--runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock --runtime-cgroups=${containerd_runtime_cgroups}"
EOF

Copilot

Pull request overview

Copilot reviewed 11 out of 12 changed files in this pull request and generated 1 comment.

Files not reviewed (1)

aks-node-controller/pkg/gen/aksnodeconfig/v1/kubelet_config.pb.go: Generated file

Comments suppressed due to low confidence (2)

parts/linux/cloud-init/artifacts/cse_helpers.sh:1490

The PR description and title still refer to placing services under kubelet.slice / Slice=kubelet.slice, but the code and tests in this PR consistently use kubereserved.slice. Please update the PR description (or rename the slice) so reviewers/operators aren’t left with conflicting slice names.

# It MUST be called before kubelet starts so that /kubereserved.slice and /system.slice
# exist and are managed by systemd before the first kubelet enforcement pass.
#
# The function:

parts/linux/cloud-init/artifacts/cse_config.sh:393

This logs_to_events tag is emitted from ensureContainerd, but the event name is prefixed with ensureKubelet. This makes telemetry harder to query/triage (the same helper is already tagged correctly under ensureContainerd.* elsewhere in this function).

  resolveKubeletReservedCgroups
  if [ -n "${KUBE_RESERVED_CGROUP}" ] || [ -n "${SYSTEM_RESERVED_CGROUP}" ]; then
      if ! logs_to_events "AKS.CSE.ensureKubelet.ensureKubeletCgroupHierarchy" ensureKubeletCgroupHierarchy; then
          exit $ERR_KUBELET_START_FAIL

…de hardening is on

… enabled

…bled

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

…elet.slice

… RP-supplied values

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 11 out of 12 changed files in this pull request and generated 1 comment.

Files not reviewed (1)

aks-node-controller/pkg/gen/aksnodeconfig/v1/kubelet_config.pb.go: Generated file

+        # Enable the slice for subsequent boots AND materialise the cgroup tree
+        # at /sys/fs/cgroup/kubereserved.slice on this boot before kubelet starts.
+        # systemctlEnableAndStart wraps both operations with retry logic to
+        # survive transient systemd failures during CSE.
+        if ! systemctlEnableAndStart kubereserved.slice 30; then
+            echo "ensureKubeletCgroupHierarchy: failed to enable and start kubereserved.slice"


Copilot

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated 1 comment.

Files not reviewed (1)

aks-node-controller/pkg/gen/aksnodeconfig/v1/kubelet_config.pb.go: Generated file

Comments suppressed due to low confidence (1)

parts/linux/cloud-init/artifacts/cse_config.sh:394

This is inside ensureContainerd(), but failures from ensureKubeletCgroupHierarchy are logged under the ensureKubelet event name and exit with ERR_KUBELET_START_FAIL. That makes telemetry/error classification misleading for containerd startup failures (and can confuse troubleshooting). Use an ensureContainerd-scoped event name and an error code consistent with other systemctl failures in this function.

      if ! logs_to_events "AKS.CSE.ensureKubelet.ensureKubeletCgroupHierarchy" ensureKubeletCgroupHierarchy; then
          exit $ERR_KUBELET_START_FAIL
      fi

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated 3 comments.

Files not reviewed (1)

aks-node-controller/pkg/gen/aksnodeconfig/v1/kubelet_config.pb.go: Generated file

Comments suppressed due to low confidence (2)

parts/linux/cloud-init/artifacts/cse_helpers.sh:1616

ensureKubeletCgroupHierarchy calls systemctlEnableAndStart for kubereserved.slice, but systemctlEnableAndStart uses systemctl restart internally. For a slice unit, restart is more disruptive than necessary and can impact already-running units in that slice during the idempotent refresh path (e.g., nodePrep). Prefer enabling + starting the slice explicitly with retries (without restart).

        if ! systemctlEnableAndStart kubereserved.slice 30; then
            echo "ensureKubeletCgroupHierarchy: failed to enable and start kubereserved.slice"

parts/linux/cloud-init/artifacts/cse_config.sh:394

In ensureContainerd, a failure to create the hardening slice/drop-ins is not a kubelet start failure; it’s a systemd/unit setup failure before containerd is started. Exiting with ERR_KUBELET_START_FAIL (34) will misclassify the failure; ERR_SYSTEMCTL_START_FAIL (4) is a closer match.

  resolveKubeletReservedCgroups
  if [ -n "${KUBE_RESERVED_CGROUP}" ] || [ -n "${SYSTEM_RESERVED_CGROUP}" ]; then
      if ! logs_to_events "AKS.CSE.ensureKubelet.ensureKubeletCgroupHierarchy" ensureKubeletCgroupHierarchy; then
          exit $ERR_KUBELET_START_FAIL
      fi

+// Node Hardening cgroup slice names. AgentBaker is the single
+// source of truth for these values: cse_helpers.sh::ensureKubeletCgroupHierarchy
+// is what actually creates (or validates) the systemd slice unit on the node, so
+// the name must be decided here rather than accepted verbatim from the RP.
+const (


+    # Refresh --runtime-cgroups for PIS nodes where basePrep may have baked an older
+    # 10-containerd-base-flag.conf pointing at /system.slice/containerd.service.
+    local containerd_runtime_cgroups="/system.slice/containerd.service"
+    if [ "${KUBE_RESERVED_CGROUP:-}" = "/kubereserved.slice" ] || [ "${KUBE_RESERVED_CGROUP:-}" = "kubereserved.slice" ]; then
+        containerd_runtime_cgroups="/kubereserved.slice/containerd.service"
+    fi
+    tee "/etc/systemd/system/kubelet.service.d/10-containerd-base-flag.conf" > /dev/null <<EOF
+[Service]
+Environment="KUBELET_CONTAINERD_FLAGS=--runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock --runtime-cgroups=${containerd_runtime_cgroups}"
+EOF


+
+// ValidateServiceInSlice asserts that the given systemd service is running in the expected slice.
+func ValidateServiceInSlice(ctx context.Context, s *Scenario, service, expectedSlice string) {
+	s.T.Helper()
+	// Avoid accidental shell injection / option smuggling.
+	if !regexp.MustCompile(`^[A-Za-z0-9_.@:-]+$`).MatchString(service) {
+		s.T.Fatalf("invalid systemd unit name: %q", service)
+	}


Copilot AI review requested due to automatic review settings July 15, 2026 19:38

titilambert requested review from AbelHu, Devinwong, SriHarsha001, awesomenix, calvin197, cameronmeissner, djsly, ganeshkumarashok, karenychen, lilypan26, mxj220, pdamianov-dev, phealy, r2k1, runzhen, sulixu, surajssd, timmy-wright, xuexu6666 and zachary-bailey as code owners July 15, 2026 19:38

Copilot started reviewing on behalf of titilambert July 15, 2026 19:39 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

Comment thread parts/linux/cloud-init/artifacts/cse_helpers.sh Outdated

Comment thread spec/parts/linux/cloud-init/artifacts/cse_helpers_spec.sh

Copilot AI review requested due to automatic review settings July 15, 2026 19:42

titilambert force-pushed the fix-test-ensure-kubelet-and branch from 39476df to 4045739 Compare July 15, 2026 19:42

Copilot started reviewing on behalf of titilambert July 15, 2026 19:43 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

Comment thread parts/linux/cloud-init/artifacts/cse_helpers.sh Outdated

Comment thread spec/parts/linux/cloud-init/artifacts/cse_helpers_spec.sh

titilambert changed the title ~~fix(test): ensure kubelet and containerd are in kubelet.slice when node hardening is on~~ fix: place kubelet and containerd in kubelet.slice via systemd drop-ins Jul 15, 2026

SriHarsha001 reviewed Jul 15, 2026

View reviewed changes

Copilot AI reviewed Jul 22, 2026

View reviewed changes

cameronmeissner approved these changes Jul 22, 2026

View reviewed changes

Copilot AI reviewed Jul 22, 2026

View reviewed changes

Comment thread parts/linux/cloud-init/artifacts/cse_helpers.sh

titilambert and others added 21 commits July 23, 2026 10:52

fix(test): ensure kubelet and containerd are in kubelet.slice when no…

65bd4a8

…de hardening is on

fix PR comments

2ead406

Make things simplier and idempotent

f8daa6f

Remove useless test

a4005a2

Add e2e tests

9067c21

Ensure runtime-cgroups match with containerd cgroups

3bf4790

Fix PR comments

57fc240

Fix PR comments

7ca3f57

Place containerd in dedicated containerd.slice when node hardening is…

0036c8f

… enabled

Place kubelet and containerd in kube.slice when node hardening is ena…

478c1b4

…bled

Rename kube.slice to kube-reserved.slice

b92dafe

Potential fix for pull request finding

4ba0f00

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

7c83f71

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Rename kube.slice to kube-reserved.slice with backward compat for kub…

643abb9

…elet.slice

Keep only kube-reserved.slice

b5c24ad

agentbaker owns Node Hardening cgroup slice names instead of trusting…

c695e65

… RP-supplied values

rename from kube-reserved to kubereserved

deeee05

Potential fix for pull request finding

669e743

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

03f6f21

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

a3a8493

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

coalesce enable and start into existing systemctlEnableAndStart command

4eaeb2f

Copilot AI reviewed Jul 23, 2026

View reviewed changes

Fix tests

975c93d

Copilot AI reviewed Jul 23, 2026

View reviewed changes

Comment thread e2e/vmss.go Outdated

Potential fix for pull request finding

6fa4b31

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot AI reviewed Jul 23, 2026

View reviewed changes

mxj220 approved these changes Jul 23, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: ensure kubelet and containerd in kube.slice via systemd drop-ins#8958

fix: ensure kubelet and containerd in kube.slice via systemd drop-ins#8958
mxj220 merged 23 commits into
mainfrom
fix-test-ensure-kubelet-and

titilambert commented Jul 15, 2026 •

edited

Loading

Uh oh!

titilambert commented Jul 15, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

SriHarsha001 left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Conversation

titilambert commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

titilambert commented Jul 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

SriHarsha001 left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

titilambert commented Jul 15, 2026 •

edited

Loading