feat(linux): onboard Ubuntu 26.04 minimal image builds for AMD64 + ARM64#8964
feat(linux): onboard Ubuntu 26.04 minimal image builds for AMD64 + ARM64#8964cameronmeissner wants to merge 116 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
This PR onboards Ubuntu 26.04 (Resolute) minimal VHD builds for Gen2 AMD64 + ARM64, replacing the retired 22.04 minimal SKUs, and refactors VHD build + provisioning logic to support 22.04 / 24.04 / 26.04 (including minimal variants) with fewer ad-hoc SKU checks.
Changes:
- Add Ubuntu 26.04 minimal build jobs and image definitions (including ARM64) plus supporting components.json release key (
r2604) and schema updates. - Refactor Linux VHD build/provision scripts for better portability on minimal images (notably removing
gawkdependency) and update LSM/BPF + CIS/hardening behaviors for 26.04. - Extend E2E coverage to include Ubuntu 26.04 minimal scenarios (bootstrap, networking, NPD, secure TLS bootstrap fallback, RCV1P, etc.).
Reviewed changes
Copilot reviewed 33 out of 33 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| vhdbuilder/scripts/linux/ubuntu/tool_installs_ubuntu.sh | Removes templated comment markers and adds basic error-guarding for pushd/popd and cd in BCC install. |
| vhdbuilder/packer/vhd-scanning.sh | Marks Ubuntu 26.04 as CIS-unsupported (pending upstream benchmarks). |
| vhdbuilder/packer/trivy-scan.sh | Adds Ubuntu 26.04 Trivy package version and adjusts PMC key + az CLI installation logic (pip fallback for 26.04). |
| vhdbuilder/packer/test/linux-vhd-content-test.sh | Updates tests for 26.04 (LSM BPF gating and skipping BCC assertions). |
| vhdbuilder/packer/pre-install-dependencies.sh | Switches OS parsing away from gawk; installs minimal build dependencies for Ubuntu minimal images. |
| vhdbuilder/packer/post-install-dependencies.sh | Switches OS parsing away from gawk; avoids PackageKit purge on minimal images. |
| vhdbuilder/packer/packer_source.sh | Includes Ubuntu 26.04 in PAM config selection logic. |
| vhdbuilder/packer/install-dependencies.sh | Major refactor: components-driven caching/install flow; LSM BPF ordering change; skip eBPF tool install on 26.04; extracts udev rules and caching into helpers. |
| schemas/components.cue | Adds r2604 release key to schema. |
| pkg/agent/datamodel/types.go | Treats Ubuntu 26.04 distro as containerd v2-capable. |
| pkg/agent/baker.go | Updates template functions for cgroup v2 and containerd 2.0+ ulimit behavior to include 26.04. |
| parts/linux/cloud-init/artifacts/ubuntu/cse_install_ubuntu.sh | Adds Ubuntu 26.04 handling, minimal-build prerequisites, batch apt installs helper, and updated repo/key handling (including 2025 MS key). |
| parts/linux/cloud-init/artifacts/mariner/cse_install_mariner.sh | Adds stub installMinimalBuildDeps for interface parity. |
| parts/linux/cloud-init/artifacts/flatcar/cse_install_flatcar.sh | Adds stub installMinimalBuildDeps for interface parity. |
| parts/linux/cloud-init/artifacts/cse_install.sh | Switches OS parsing away from gawk. |
| parts/linux/cloud-init/artifacts/cse_helpers.sh | Switches OS parsing away from gawk; adds isMinimalImage helper. |
| parts/linux/cloud-init/artifacts/cse_config.sh | Extends systemd-resolved workaround to Ubuntu 26.04 and fixes typo. |
| parts/linux/cloud-init/artifacts/cis.sh | Updates root password hashing for Python 3.14 (26.04) and selects classic sudo to preserve CIS file logging. |
| parts/linux/cloud-init/artifacts/azlosguard/cse_install_osguard.sh | Adds stub installMinimalBuildDeps for interface parity. |
| parts/linux/cloud-init/artifacts/acl/cse_install_acl.sh | Adds stub installMinimalBuildDeps for interface parity. |
| parts/common/components.json | Adds Ubuntu r2604 entries for core components (containerd, runc, kubelet/kubectl, ig, etc.). |
| e2e/validators.go | Minor string cleanup; kernel log panic matcher now excludes benign DRM panic-handler registration lines. |
| e2e/test_helpers.go | Renames GPU scenario helper to be Ubuntu 24.04-specific. |
| e2e/scenario_test.go | Adds extensive Ubuntu 26.04 minimal E2E scenarios (AMD64 + ARM64) and adjusts some existing GPU tests to new helper name. |
| e2e/scenario_rcv1p_test.go | Adds RCV1P cert-mode scenario for Ubuntu 26.04 minimal. |
| e2e/scenario_localdns_hosts_test.go | Adds Ubuntu 26.04 minimal to localdns hosts plugin test matrix (with cluster selection tweak). |
| e2e/e2e-local.sh | Adds support for running against a specific VHD build ID via env. |
| e2e/config/vhd.go | Adds Ubuntu 26.04 minimal image definitions for AMD64 and ARM64. |
| e2e/cache.go | Adds “latest k8s” cluster variants for kubenet/azure network/overlay dualstack and updates model cache name. |
| e2e/aks_model.go | Refactors AKS cluster model builders into mutators and relocates latest-GA version resolver. |
| .pipelines/templates/.builder-release-template.yaml | Derives SKU name “minimal” suffix from FEATURE_FLAGS instead of IMG_SKU substring matching. |
| .pipelines/.vsts-vhd-builder-release.yaml | Replaces 22.04 minimal jobs with 26.04 minimal Gen2 AMD64 + ARM64 jobs and sets new offer/SKU + feature flags. |
| .github/workflows/no-sudo-check.yml | Allows update-alternatives usage (needed for sudo selection on 26.04). |
AgentBaker Linux gate detectiveRun: 173660533 Detective summary: Multiple SecondaryNIC DualStack scenarios timed out waiting for the VMSS VM before SSH/node validation; boot diagnostics then returned Compute 409 because another operation was still in progress. Likely cause / signature: Azure Compute/VMSS provisioning delay in the dual-stack secondary NIC shared-cluster path; PR #8964 changes are a weaker theory because failures span existing Ubuntu/AzureLinux scenarios and stop before node evidence. Confidence: Medium-high Recommended owner/action: AgentBaker E2E / test-infra owner to rerun and track VMSS provisioning flake; escalate if recurring. Strongest alternative: PR 26.04 onboarding affected provisioning, but this is less likely because the failure is pre-SSH VM readiness across multiple existing OS scenarios. Evidence: timeline failed task exit 1; failed test run 545097787; task log 562 around line 987; build metadata/PR metadata. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 33 out of 33 changed files in this pull request and generated 3 comments.
Comments suppressed due to low confidence (1)
e2e/scenario_test.go:2113
- The scenario description is just "T", which makes test output hard to interpret and search. Please replace with a descriptive sentence matching the test’s intent (VHD caching validation for Ubuntu 26.04 minimal).
RunScenario(t, &Scenario{
Description: "T",
Config: Config{
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 33 out of 33 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (4)
e2e/scenario_test.go:1887
- The blobfuse2 validation is currently commented out, so the Ubuntu 26.04 minimal Azure CNI scenario is not asserting blobfuse2 is installed. Rather than leaving commented code, gate the validation on whether components.json defines an expected version for r2604 so the check automatically becomes active once blobfuse2 is available.
// ValidateInstalledPackageVersion(ctx, s, "blobfuse2", components.GetExpectedPackageVersions("blobfuse2", "ubuntu", "r2604")[0])
e2e/scenario_test.go:2265
- The blobfuse2 validation is commented out for the Ubuntu 26.04 minimal ARM64 scenario. This leaves dead code and makes it easy to accidentally reintroduce a panic by indexing into an empty expected-versions slice. Gate the validation on whether an expected version exists so it auto-enables when components.json is updated.
// ValidateInstalledPackageVersion(ctx, s, "blobfuse2", components.GetExpectedPackageVersions("blobfuse2", "ubuntu", "r2604")[0])
e2e/scenario_test.go:2290
- The blobfuse2 validation is commented out for the Ubuntu 26.04 minimal ARM64 Azure CNI scenario. Prefer a conditional check over commented code so this automatically starts validating once the expected version is added to components.json (and avoids slice index panics).
// ValidateInstalledPackageVersion(ctx, s, "blobfuse2", components.GetExpectedPackageVersions("blobfuse2", "ubuntu", "r2604")[0])
e2e/scenario_test.go:2112
- This scenario description is currently just "T", which makes test output and failures hard to interpret. Please replace it with a meaningful description (even if the validator is intentionally empty as a smoke test for VHD caching).
Description: "T",
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 33 out of 33 changed files in this pull request and generated 3 comments.
Comments suppressed due to low confidence (1)
e2e/scenario_test.go:2112
- This scenario description is currently just "T", which makes it hard to understand what the VHDCaching test is covering when reading logs or filtering scenarios.
Description: "T",
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 33 out of 33 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (1)
parts/linux/cloud-init/artifacts/cis.sh:20
- On Ubuntu 26.04 this switches to passlib for SHA-512 password hashing, but nothing here ensures passlib is installed. If the base/minimal image lacks python3-passlib, this will fail with ModuleNotFoundError and break CIS hardening. Ensure python3-passlib is present before constructing the passlib-based CMD (or provide a non-passlib fallback).
CMD="import crypt, getpass, pwd; print(crypt.crypt('$SECRET', '\$6\$$SALT\$'))"
if [ "${VERSION}" = "26.04" ]; then
# Ubuntu 26.04 ships Python 3.14, which removed the 'crypt' module (removed in 3.13).
# passlib's sha512_crypt has a pure-Python backend that produces the same SHA-512 ($6$)
# hash without needing the crypt module. rounds=5000 matches the crypt(3) default cost.
# SALT is base64 and may contain chars (e.g. '+', '=') outside passlib's salt alphabet,
# so filter it to sha512_crypt.salt_chars before use.
CMD="import getpass, pwd; from passlib.hash import sha512_crypt; salt=''.join(c for c in '$SALT' if c in sha512_crypt.salt_chars); print(sha512_crypt.using(salt=salt, rounds=5000).hash('$SECRET'))"
fi
AgentBaker Linux gate detectiveRun: 173700978 Detective summary: LocalDNS hosts plugin validation failed: after cold start and hosts-file canary insertion, the canary did not resolve within the 60s polling window. Likely cause / signature: Recurring LocalDNS hosts plugin hot-reload timeout; this build crossed the repair threshold and is now tracked by repair item #38940381. Confidence: High Recommended owner/action: Node Lifecycle / LocalDNS owner should investigate hosts plugin reload behavior or adjust the validator budget/retry path; repair item #38940381 tracks follow-up. Strongest alternative: Transient DNS/CoreDNS readiness delay, but less likely because this exact signature has recurred across seven distinct builds. Evidence: timeline failed task exit 1; failed test run 545494657; task log 562 around line 1500; build/PR metadata. |
What this PR does / why we need it:
Onboards Ubuntu 26.04 (Resolute) minimal VHD builds for both AMD64 and ARM64 (Gen2), replacing the retired 22.04 minimal SKUs. Confirmed with @allyford that removal of the 2204 minimal SKUs is correct.
Alongside the 26.04 enablement, this PR does a fair amount of refactoring of the VHD build + CSE provisioning scripts so that a single code path cleanly supports 22.04 / 24.04 / 26.04 (and the minimal image variant) instead of branching on ad-hoc
IMG_SKUstring matching.Important
Follow-up PRs are required to add
acr-mirror(artifact streaming) andblobfuse2(Azure Blob CSI fuse driver) support on the new 26.04 images. Both packages are not yet published in the Resolute (26.04) PMC repos, so they are intentionally stubbed out on 26.04 for now and marked withTODO(2604)ininstall-dependencies.sh. See the Known gaps / follow-ups section below.Important
GPU drivers and the managed GPU experience are NOT supported on Ubuntu 26.04 in this PR — they are deferred to GA. The NVIDIA GRID driver currently pinned for AKS does not compile against the 26.04 kernel (
linux-azure7.0 removedin_irq()and restructuredvm_area_struct's private__vm_flags, which the current driver source still relies on). GPU driver installation and the managed GPU experience are intentionally out of scope for 26.04 and will be enabled for GA in a follow-up.Scope of the 26.04 onboarding
Build pipeline / SKUs
.vsts-vhd-builder-release.yaml: swapped the2204 minimal gen1/gen2build jobs for2604 minimal gen2(AMD64,Standard_D16ds_v5) and2604 minimal arm64 gen2(Standard_D16pds_v5). New offer/SKU:ubuntu-26_04-lts/minimal+minimal-arm64, Gen V2,FEATURE_FLAGS=minimal..builder-release-template.yaml: SKU-name derivation now appendsminimalfromFEATURE_FLAGS(instead of matchingIMG_SKU).Package definitions
parts/common/components.json: addedr2604entries for the core set (containerd, runc, cni-plugins, cri-tools, kubelet/kubectl, azure-acr-credential-provider, ig/inspektor-gadget, aks-secure-tls-bootstrap-client, node-exporter).schemas/components.cue: added ther2604release key.CSE provisioning (
cse_install_ubuntu.sh)linux-modules-extra-*is dropped from the package list (its contents are bundled intolinux-modules-*on Resolute).microsoft-2025.asc).installMinimalBuildDeps/installUbuntu2604MinimalBuildDepsto pull minimal-image build prerequisites (rsyslog,gpg).LSM / BPF configuration (
install-dependencies.sh)bpfis added as an LSM:configureLsmWithBpfnow appendsbpfto the end of the active LSM list (lsm=<current>,bpf) instead of prepending it (bpf,<current>). On the 26.04 stack (kernel 7.0 + systemd 259 + AppArmor 5.0), placingbpfahead ofapparmorshadowed AppArmor on the exclusivesocket_getpeersec_stream(SO_PEERSEC) hook, which broke AppArmor's D-Bus peer mediation —dbus-daemonthen rejected every new system-bus connection,networkctl reloadfailed withTransport endpoint is not connected, and localdns → CSE failed with exit 216. Orderingbpflast keeps AppArmor as the owner of the peersec/label interfaces (restoring D-Bus mediation) while preserving full BPF-LSM enforcement (Cilium, systemdRestrictFileSystems=). This ordering is applied for Ubuntu 24.04 / 26.04 and Azure Linux 3.0; the 24.04 reorder is intentional and behavior-equivalent for enforcement (every LSM still runs; only the exclusive peersec owner changes).CIS / hardening (
cis.sh)cryptmodule, so we switch topasslib's pure-Pythonsha512_crypt(rounds=5000 to match crypt(3)).preferClassicSudo: 26.04 defaults to sudo-rs, which doesn't supportDefaults logfile; we select classic sudo viaupdate-alternativesso the CIS/var/log/sudo.logaudit trail is preserved.Build / scan / test infra
pre-install-dependencies.sh: installs minimal build deps whenisMinimalImage && isUbuntu.post-install-dependencies.sh: PackageKit purge now guarded byisMinimalImage(minimal doesn't ship it).trivy-scan.sh: added 2604 trivy deb version; restructuredinstall_azure_cli(guard clauses + PMC helper); 26.04 installs az via pip (--break-system-packages) as aTODO(2604)until the Resolute az package exists; import 2025 PMC key.vhd-scanning.sh: skip CIS scanning on 26.04 (TODO(2604)— upstream CIS benchmarks not yet available).linux-vhd-content-test.sh: skip BCC-tools check on 26.04;packer_source.shreuses the 2204 PAM config for 26.04.E2E coverage (
e2e/)VHDUbuntu2604MinimalGen2Containerd+...Arm64...image defs.ValidateKernelLogs: the PANIC/CRASH matcher now excludes the benigndrm panichandler-registration lines (Registered N planes with drm panic) emitted bysimple-framebuffer/hyperv_drmon kernels withCONFIG_DRM_PANIC(6.10+, e.g. 26.04 / linux-azure 7.0), so they no longer false-positive as kernel panics.General refactoring
gawk 'match(...)'ID/VERSION_ID/VARIANT_IDparsing with a portablesed | head | trpipeline acrossinstall-dependencies.sh,pre/post-install-dependencies.sh,cse_helpers.sh, andcse_install.sh—gawkis not present on the minimal image.install-dependencies.sh: removed the large inlined top-level per-package install block and drove it from a components.json-based case dispatch loop; extracted the sharedaptGetBatchInstallPackagesWithFallbackhelper; minimal-vs-full gating now viaisMinimalImage/FEATURE_FLAGS=minimalrather thanIMG_SKUsubstring matching. eBPF/BCC tooling is skipped on 26.04.isMinimalImagehelper added tocse_helpers.sh;installMinimalBuildDepsstubbed for the non-Ubuntu OSes (mariner, flatcar, acl, osguard).tool_installs_ubuntu.sherror-code comments de-templatized + shellcheck|| exit 1guards;no-sudo-check.ymlallowlistsupdate-alternatives; typo fixes (Ingoring→Ignoring).Known gaps / follow-ups
These are intentionally deferred and must land in follow-up PRs:
in_irq()removed,vm_area_struct.__vm_flagsrestructured); to be added once a kernel-7.0-compatible driver is available.acr-mirror(artifact streaming) —TODO(2604)ininstall-dependencies.sh; not yet in resolute PMC.blobfuse2(Azure Blob Storage CSI fuse driver) —TODO(2604)ininstall-dependencies.sh; not yet in resolute PMC. E2E already asserts its presence, so the 2604 bootstrap tests will drive closure of this gap.trivy-scan.sh; swap to the PMC package when published (TODO(2604)).vhd-scanning.shonce upstream CIS publishes Resolute benchmarks (TODO(2604)).