Skip to content

gitserver: make PVC access modes configurable via storageAccessModes - #921

Open
marcleblanc2 wants to merge 1 commit into
mainfrom
gitserver-storage-access-modes
Open

gitserver: make PVC access modes configurable via storageAccessModes#921
marcleblanc2 wants to merge 1 commit into
mainfrom
gitserver-storage-access-modes

Conversation

@marcleblanc2

Copy link
Copy Markdown
Contributor

Problem

The gitserver StatefulSet volumeClaimTemplate hardcodes accessModes: [ReadWriteOnce].

On SELinux-enforcing nodes (Bottlerocket, EKS Auto Mode), every gitserver pod (re)start makes containerd recursively relabel every file on the repos volume with the new container's random MCS categories, because an RWO volume cannot use the -o context mount option (the SELinuxMount feature gate is off by default in Kubernetes <= 1.36, and managed offerings like EKS Auto Mode do not allow enabling it). On a repos volume with millions of git objects, this holds gitserver in ContainerCreating for 15-30+ minutes, with kubelet logging failed to reserve container name ... another CreateContainer request is in progress as its CRI calls time out and retry behind the still-running relabel walk.

Fix

Add gitserver.storageAccessModes, defaulting to the previous hardcoded ["ReadWriteOnce"] so existing deployments render identically.

Setting it to ["ReadWriteOncePod"] enables the mount-option labeling path (SELinuxMountReadWriteOncePod, GA in Kubernetes 1.36): kubelet mounts the volume with -o context=<label> and the relabel walk is skipped entirely. Requires the pod to pin gitserver.podSecurityContext.seLinuxOptions.level and a CSI driver with seLinuxMount: true (the EBS CSI driver qualifies).

Notes

  • No behavior change for existing users: default renders the same manifest (verified with helm template; all 103 unit tests and 12 snapshots pass).
  • Access modes are immutable on PVCs and volumeClaimTemplates are immutable on StatefulSets, so switching an existing deployment requires deleting the StatefulSet (--cascade=orphan) and recreating the PVC. Noted in the CHANGELOG.
  • Diagnosed on an EKS Auto Mode cluster where a 3.7M-inode repos volume was fully relabeled (millions of xattr writes, ~9 GB of metadata IO) on every gitserver restart.

Test plan

  • helm unittest charts/sourcegraph: 24 suites, 103 tests, 12 snapshots pass
  • helm template with defaults renders ReadWriteOnce (unchanged)
  • helm template --set 'gitserver.storageAccessModes={ReadWriteOncePod}' renders ReadWriteOncePod
  • README regenerated with helm-docs

The gitserver StatefulSet volumeClaimTemplate hardcoded
accessModes: [ReadWriteOnce]. On SELinux-enforcing nodes (Bottlerocket,
EKS Auto Mode), every gitserver pod (re)start makes containerd
recursively relabel every file on the repos volume with the new
container's MCS categories, because the volume cannot use the
-o context mount option. On a volume with millions of git objects this
holds the pod in ContainerCreating for 15-30+ minutes.

Since Kubernetes 1.36, SELinuxMountReadWriteOncePod is GA: a
ReadWriteOncePod PVC (plus a pinned seLinuxOptions.level and a CSI
driver with seLinuxMount: true) is mounted with -o context and skips
the relabel walk entirely.

Adds gitserver.storageAccessModes, defaulting to the previous
hardcoded [ReadWriteOnce], so existing deployments are unchanged.

Amp-Thread-ID: https://ampcode.com/threads/T-019fdaa3-9594-762b-8593-8de01deb566e
Co-authored-by: Amp <amp@ampcode.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant