Minion/master connection-resilience defaults, dedicated namespaces, GHCR-first quickstart

README.md

@@ -10,66 +10,65 @@ Helm charts for deploying different flavours of Salt images to Kubernetes.
 | [salt-minion-kubernetes](salt-minion-kubernetes) | Installs Salt Minion and RBAC. Has built-in support to run CIS Kubernetes compliance assessments via kube-bench on-demand Jobs. Supports in-cluster (minion runs as a pod) and external (RBAC only) modes. Like `salt-minion-vcf`, this is a full project directory - its own `Dockerfile` builds a Salt minion preloaded with `saltext.vault` and `saltext.kubernetes`, with `kubectl` bundled in. |
 | [salt-minion-vcf](salt-minion-vcf) | Extensible Salt Minion image (Docker, Docker Compose, Kubernetes, and Helm) preloaded with configurable Salt extensions - `saltext.vcf` (VMware Cloud Foundation automation: vCenter, NSX, SDDC-M, VCF Ops) by default, but not limited to it. Includes `saltext.vault` integration for sourcing credentials from HashiCorp Vault into Pillar instead of storing them on disk. Unlike the other entries here, this directory is the full project (Dockerfile, Docker Compose, scripts, docs), not a chart-only directory - the Helm chart itself lives at [`salt-minion-vcf/helm/salt-minion-vcf`](salt-minion-vcf/helm/salt-minion-vcf). |
 
-## Usage
+## Quick install (published images/charts)
 
-### Kubernetes / Helm
+The fastest path to a working deployment: install straight from the
+published GHCR OCI charts, no local Docker/Helm build required. See
+[`docs/releasing.md`](docs/releasing.md) for the full list of published
+artifacts, and each component's own `CHANGELOG.md` for exactly which
+Salt/extension versions a given tag carries (the tag itself is just that
+component's own semver, independent of Salt's version).
 
-```bash
-helm install salt-master-kubernetes ./salt-master-kubernetes -f my-values.yaml
-```
+### salt-master-kubernetes
 
 ```bash
-helm install salt-minion-kubernetes ./salt-minion-kubernetes -f my-values.yaml
+helm install salt-master-kubernetes \
+  oci://ghcr.io/saltstack/salt-helm/charts/salt-master-kubernetes \
+  --version 0.1.0 \
+  --set agent.image.repository=ghcr.io/saltstack/salt-helm/salt-master \
+  --set agent.image.tag=1.0.0
 ```
 
+### salt-minion-kubernetes
+
 ```bash
-helm install salt-minion-vcf ./salt-minion-vcf/helm/salt-minion-vcf \
-  --set salt.master=salt-master.example.com
+helm install salt-minion-kubernetes \
+  oci://ghcr.io/saltstack/salt-helm/charts/salt-minion-kubernetes \
+  --version 0.1.0 \
+  --set agent.saltMasterHost=salt-master.example.com
 ```
 
-See each chart's `values.yaml` for configurable parameters.
+(`agent.image.repository` already defaults to the published
+`ghcr.io/saltstack/salt-helm/salt-minion-kubernetes` image - no override
+needed there.)
 
-Charts and images are also published to GHCR — see
-[`docs/releasing.md`](docs/releasing.md) for the full list, and each
-component's own `CHANGELOG.md` for exactly which Salt/extension versions a
-given release tag carries (the tag itself is just that component's own
-semver, independent of Salt's version):
+### salt-minion-vcf
 
 ```bash
-helm install salt-master-kubernetes \
-  oci://ghcr.io/saltstack/salt-helm/charts/salt-master-kubernetes \
-  --version 0.1.0
+helm install salt-minion-vcf \
+  oci://ghcr.io/saltstack/salt-helm/charts/salt-minion-vcf \
+  --version 0.1.0 \
+  --set image.repository=ghcr.io/saltstack/salt-helm/salt-minion-vcf \
+  --set image.tag=0.1.0 \
+  --set salt.master=salt-master.example.com
 ```
 
-### Docker
-
-`salt-minion-vcf` also runs as a plain Docker container or via Docker
-Compose, without Kubernetes - see
+`salt-minion-vcf` also runs as a plain Docker container, pulling the
+published image directly - no Kubernetes required. See
 [`salt-minion-vcf/README.md`](salt-minion-vcf/README.md) for the full guide,
-including air-gapped builds and the local Vault-backed testing workflow.
+including Docker Compose and the local Vault-backed testing workflow.
 
 ```bash
-docker build -t salt-minion-vcf:0.1.0 ./salt-minion-vcf
-
 docker run -d \
   --name salt-minion-vcf \
   -e SALT_MASTER=salt-master.example.com \
   -v salt-minion-vcf-pki:/etc/salt/pki/minion \
-  salt-minion-vcf:0.1.0
+  ghcr.io/saltstack/salt-helm/salt-minion-vcf:0.1.0
 ```
 
-or with Docker Compose:
-
-```bash
-cd salt-minion-vcf
-cp .env.example .env   # set SALT_MASTER, etc.
-docker compose up -d --build
-```
+See each chart's `values.yaml` for the full list of configurable parameters.
 
-`salt-minion-kubernetes` and `docker/salt-master` build the same way, each
-with its own directory as context:
+## Building from source
 
-```bash
-docker build -t salt-minion-kubernetes:0.1.0 ./salt-minion-kubernetes
-docker build -t salt-master:3008.2 ./docker/salt-master
-```
+Need a patched image, a different Salt version, or an air-gapped build with
+no route to GHCR? See [`docs/building.md`](docs/building.md).

docker/salt-master/entrypoint.sh

@@ -8,6 +8,12 @@ mkdir -p "$CONF_DIR"
     [ -z "${SALT_AUTO_ACCEPT:-}" ] || printf 'auto_accept: %s\n' "$SALT_AUTO_ACCEPT"
     [ -z "${SALT_MASTER_ID:-}" ] || printf 'id: %s\n' "$SALT_MASTER_ID"
 
+    # Fires salt/presence/present (and /change) events on the event bus so
+    # `salt-run manage.present`/`manage.status` reflect which minions are
+    # actually connected right now - useful in Kubernetes, where minion pods
+    # come and go independently of their accepted-key status.
+    printf 'presence_events: %s\n' "${SALT_PRESENCE_EVENTS:-True}"
+
     # Keep pidfile/sock_dir off /var/run: the container runtime remounts
     # /run fresh (root-owned, 0755) on every pod start regardless of what's
     # baked into the image, so a non-root master can never mkdir under

docs/building.md

@@ -0,0 +1,95 @@
+# Building From Source
+
+Most users don't need this — the published images and charts on GHCR (see
+the main [`README.md`](../README.md)) are the fastest path to a working
+deployment. Build from source instead when you need a patched/local image,
+a Salt version other than what's published, or an air-gapped build with no
+route to GHCR.
+
+## Prerequisites
+
+- Docker (or another OCI builder) for the images
+- Helm 3+ for the charts
+
+## Docker images
+
+Each image is self-contained (pip-installed Salt extensions from PyPI rather
+than a checkout of the extension's own repo, where applicable), so all three
+build directly from this repo with that image's own directory as the build
+context:
+
+```bash
+docker build -t salt-master:3008.2 docker/salt-master
+docker build -t salt-minion-vcf:0.1.0 salt-minion-vcf
+docker build -t salt-minion-kubernetes:0.1.0 salt-minion-kubernetes
+```
+
+See each image's own README for the full build-arg reference (Salt version,
+extension selection, internal PyPI/apt mirrors for air-gapped builds, etc.):
+
+- [`docker/salt-master/README.md`](../docker/salt-master/README.md)
+- [`salt-minion-vcf/README.md`](../salt-minion-vcf/README.md)
+- [`salt-minion-kubernetes/README.md`](../salt-minion-kubernetes/README.md)
+
+### Running a locally-built image
+
+`salt-minion-vcf` also runs as a plain Docker container or via Docker
+Compose, without Kubernetes — see
+[`salt-minion-vcf/README.md`](../salt-minion-vcf/README.md) for the full
+guide, including the local Vault-backed testing workflow:
+
+```bash
+docker run -d \
+  --name salt-minion-vcf \
+  -e SALT_MASTER=salt-master.example.com \
+  -v salt-minion-vcf-pki:/etc/salt/pki/minion \
+  salt-minion-vcf:0.1.0
+```
+
+or with Docker Compose:
+
+```bash
+cd salt-minion-vcf
+cp .env.example .env   # set SALT_MASTER, etc.
+docker compose up -d --build
+```
+
+### Getting a locally-built image onto a cluster
+
+- **With a registry:** tag and push, then point the chart's
+  `agent.image.repository`/`image.repository` value at your registry path.
+- **Without a registry** (air-gapped/test clusters): `docker save` the image
+  and `ctr -n k8s.io images import` it directly into every node's containerd
+  store, then set `image.pullPolicy: Never` (or `agent.image.pullPolicy`) so
+  Kubernetes doesn't try to pull.
+
+## Helm charts
+
+The charts themselves need no build step — install straight from a checkout
+of this repo:
+
+```bash
+helm install salt-master-kubernetes ./salt-master-kubernetes -f my-values.yaml
+helm install salt-minion-kubernetes ./salt-minion-kubernetes -f my-values.yaml
+helm install salt-minion-vcf ./salt-minion-vcf/helm/salt-minion-vcf \
+  --set salt.master=salt-master.example.com
+```
+
+Remember to also point each chart's image value at whatever you built above
+(the charts otherwise default to either the published GHCR image or an
+unqualified local tag — check `values.yaml`).
+
+To produce a distributable `.tgz` instead (e.g. for your own chart repo or
+registry):
+
+```bash
+helm package salt-master-kubernetes
+helm package salt-minion-kubernetes
+helm package salt-minion-vcf/helm/salt-minion-vcf
+```
+
+## Cutting an actual release
+
+Building locally doesn't publish anything. To push a build to
+`ghcr.io/saltstack/salt-helm/...` under a real version tag, see
+[`docs/releasing.md`](releasing.md).

docs/kubernetes-compliance-guide.md

@@ -104,8 +104,8 @@ for the full list):
 Verify the master came up healthy:
 
 ```bash
-kubectl -n kube-system get pods -l app=salt-master-kubernetes
-kubectl -n kube-system exec -it deploy/salt-master-kubernetes -- salt-key -L
+kubectl -n salt-master get pods -l app=salt-master-kubernetes
+kubectl -n salt-master exec -it deploy/salt-master-kubernetes -- salt-key -L
 ```
 
 ### 2b. salt-minion-kubernetes
@@ -134,9 +134,9 @@ jobs), and both must be reachable at whatever address you set
 Accept the new minion's key from the master:
 
 ```bash
-kubectl -n kube-system exec -it deploy/salt-master-kubernetes -- salt-key -L
-kubectl -n kube-system exec -it deploy/salt-master-kubernetes -- salt-key -a <minion-id>
-kubectl -n kube-system exec -it deploy/salt-master-kubernetes -- salt '<minion-id>' test.ping
+kubectl -n salt-master exec -it deploy/salt-master-kubernetes -- salt-key -L
+kubectl -n salt-master exec -it deploy/salt-master-kubernetes -- salt-key -a <minion-id>
+kubectl -n salt-master exec -it deploy/salt-master-kubernetes -- salt '<minion-id>' test.ping
 ```
 
 `test.ping` returning `True` confirms both ports are wired up correctly.
@@ -179,6 +179,10 @@ or `run_assessment` won't find the CronJob it's looking for:
 | `namespace` | `namespace` |
 | `cronJob.name` | `kubeBench.cronJobName` |
 
+`salt-minion-kubernetes` now defaults `namespace` to `salt` (not `kube-system`),
+so install `kube-bench-job` with `--set namespace=salt` unless you've
+overridden `salt-minion-kubernetes`'s own `namespace` value too.
+
 See [`salt-k8s-compliance/helm/kube-bench-job/README.md`](https://github.com/saltstack/salt-k8s-compliance/blob/main/helm/kube-bench-job/README.md)
 for the full configuration reference, including host-path mounts for
 non-kubeadm distros (RKE2/k3s) and the RBAC gaps to watch for on multi-node
@@ -192,7 +196,7 @@ execution module (installed on the minion as part of `saltext.kubernetes`).
 Force a fresh assessment right now, regardless of any cached result:
 
 ```bash
-kubectl -n kube-system exec -it deploy/salt-master-kubernetes -- \
+kubectl -n salt-master exec -it deploy/salt-master-kubernetes -- \
   salt '<minion-id>' kube_bench_cache.run_assessment
 ```
 
@@ -205,7 +209,7 @@ assessment first only if the cache is stale — see `pillar.ttlSeconds`
 below):
 
 ```bash
-kubectl -n kube-system exec -it deploy/salt-master-kubernetes -- \
+kubectl -n salt-master exec -it deploy/salt-master-kubernetes -- \
   salt '<minion-id>' kube_bench_cache.status_for_check test_number=1.1.11
 ```
 

salt-master-kubernetes/README.md

@@ -40,11 +40,12 @@ The following table lists the most commonly overridden values. See
 
 | Parameter | Description | Default |
 | --- | --- | --- |
-| `namespace` | Namespace for all chart resources. | `kube-system` |
+| `namespace` | Namespace for all chart resources. | `salt-master` |
 | `agent.image.repository` | Salt master image repository. | `salt-master` |
 | `agent.image.tag` | Salt master image tag. | `3007.1` |
 | `agent.autoAccept` | Auto-accept new minion keys instead of requiring `salt-key -a` per minion. Leave `false` for a production master. | `false` |
 | `agent.masterId` | Sets `id:` on the master itself. Empty uses the pod hostname. | `""` |
+| `agent.presenceEvents` | Fires `salt/presence/present`/`change` events so `manage.present`/`manage.status` reflect which minions are actually connected right now. | `true` |
 | `agent.persistence.enabled` | Persist the master's `/etc/salt/pki` (its own keypair *and* the accepted-minion key list) across pod restarts. | `false` |
 | `agent.persistence.type` | `pvc` or `hostPath`. `hostPath` requires `agent.nodeSelector`. | `pvc` |
 | `agent.nodeSelector` | Pins the pod to a node. Required when `agent.persistence.type=hostPath`. | `{}` |

salt-master-kubernetes/templates/deployment.yaml

@@ -48,6 +48,8 @@ spec:
           env:
             - name: SALT_AUTO_ACCEPT
               value: {{ .Values.agent.autoAccept | quote }}
+            - name: SALT_PRESENCE_EVENTS
+              value: {{ .Values.agent.presenceEvents | quote }}
             {{- if .Values.agent.masterId }}
             - name: SALT_MASTER_ID
               value: {{ .Values.agent.masterId | quote }}

salt-master-kubernetes/values.yaml

@@ -1,5 +1,5 @@
 # Namespace where all salt-master-kubernetes objects are deployed.
-namespace: kube-system
+namespace: salt-master
 
 agent:
   image:
@@ -56,6 +56,12 @@ agent:
   # Sets `id:` on the master itself. Leave empty to use the pod hostname.
   masterId: ""
 
+  # Fires salt/presence/present (and /change) events on the event bus, so
+  # `salt-run manage.present`/`manage.status` reflect which minions are
+  # actually connected right now - useful in Kubernetes, where minion pods
+  # come and go independently of their accepted-key status.
+  presenceEvents: true
+
   # Name of an EXISTING Kubernetes Secret (created out-of-band - e.g. by
   # whatever generates and registers this master's identity with RaaS/the
   # minion side before this chart is ever installed - never via Helm

salt-minion-kubernetes/README.md

@@ -54,7 +54,7 @@ Then issue a token for the created ServiceAccount and use it in the external
 minion's kubeconfig:
 
 ```bash
-kubectl create token salt-minion-kubernetes -n kube-system
+kubectl create token salt-minion-kubernetes -n salt
 ```
 
 ## Uninstalling the chart
@@ -70,14 +70,18 @@ The following table lists the most commonly overridden values. See
 
 | Parameter | Description | Default |
 | --- | --- | --- |
-| `namespace` | Namespace for all chart resources. Must match `kube-bench-job`'s namespace. | `kube-system` |
+| `namespace` | Namespace for all chart resources. Must match `kube-bench-job`'s namespace. | `salt` |
 | `agent.authMode` | `in_cluster` or `external`. | `in_cluster` |
 | `agent.image.repository` | Salt minion image repository. | `ghcr.io/saltstack/salt-helm/salt-minion-kubernetes` |
 | `agent.image.tag` | Salt minion image tag. | `0.1.0` |
 | `agent.kubectl.bundled` | Skip the install-kubectl init container — true when `agent.image` already bundles `kubectl` (the default image does). | `true` |
 | `agent.saltMasterHost` | Salt master address. Required for `in_cluster` mode. | `""` |
 | `agent.saltMasterPort` | Salt master "ret" port (`master_port`). Override alongside `agent.saltPublishPort` when the master isn't reachable on its default ports, e.g. behind a Kubernetes NodePort Service. | `4506` |
 | `agent.saltPublishPort` | Salt master "publish" port (`publish_port`). | `4505` |
+| `agent.authTimeout` | Seconds to wait for master auth before retrying - reduces thundering-herd retry storms. | `60` |
+| `agent.masterAliveInterval` | Seconds between checks that the master TCP connection is still alive; reconnects if not. | `60` |
+| `agent.reconDefault` / `agent.reconMax` | ZeroMQ transport reconnect backoff range (ms). | `1000` / `5000` |
+| `agent.reconRandomize` | Jitters reconnect delay so minions don't all retry in lockstep. | `true` |
 | `agent.minion.id` | Salt minion ID. Empty uses the pod hostname. | `""` |
 | `agent.persistence.enabled` | Persist the minion's generated keypair (`/etc/salt/pki`) across pod restarts. | `false` |
 | `agent.persistence.type` | `pvc` or `hostPath`. `hostPath` requires `agent.nodeSelector`. | `pvc` |

salt-minion-kubernetes/scripts/docker-entrypoint.sh

@@ -35,6 +35,11 @@ master_port: ${SALT_MASTER_PORT}
 publish_port: ${SALT_PUBLISH_PORT}
 master_tries: -1
 retry_dns: 30
+auth_timeout: ${SALT_AUTH_TIMEOUT:-60}
+master_alive_interval: ${SALT_MASTER_ALIVE_INTERVAL:-60}
+recon_default: ${SALT_RECON_DEFAULT:-1000}
+recon_max: ${SALT_RECON_MAX:-5000}
+recon_randomize: ${SALT_RECON_RANDOMIZE:-True}
 EOF
 
   # Preferred: pre-seed the master's actual public key so the minion trusts

salt-minion-kubernetes/templates/deployment.yaml

@@ -78,6 +78,16 @@ spec:
               value: {{ .Values.agent.saltMasterPort | quote }}
             - name: SALT_PUBLISH_PORT
               value: {{ .Values.agent.saltPublishPort | quote }}
+            - name: SALT_AUTH_TIMEOUT
+              value: {{ .Values.agent.authTimeout | quote }}
+            - name: SALT_MASTER_ALIVE_INTERVAL
+              value: {{ .Values.agent.masterAliveInterval | quote }}
+            - name: SALT_RECON_DEFAULT
+              value: {{ .Values.agent.reconDefault | quote }}
+            - name: SALT_RECON_MAX
+              value: {{ .Values.agent.reconMax | quote }}
+            - name: SALT_RECON_RANDOMIZE
+              value: {{ .Values.agent.reconRandomize | quote }}
             {{- if .Values.agent.minion.id }}
             - name: SALT_MINION_ID
               value: {{ .Values.agent.minion.id | quote }}