fix(hosting): deploy ClickHouse from the official image instead of Bitnami

docs/self-hosting/docker.mdx

@@ -362,6 +362,17 @@ TRIGGER_IMAGE_TAG=v4.5.0
 
 We patch the latest released version line only, so keep an eye on new releases to receive security fixes. See [Security & vulnerability reporting](/self-hosting/security).
 
+You can also lock the versions of the bundled services, for example with `CLICKHOUSE_IMAGE_TAG`. If you do, or if you bring your own ClickHouse via `CLICKHOUSE_URL`, note that Trigger.dev requires ClickHouse 25.8 or newer.
+
+<Note>
+  The bundled ClickHouse now uses the official `clickhouse/clickhouse-server` image. Your existing
+  data volume carries over automatically. If you previously pinned `CLICKHOUSE_IMAGE_TAG` to a
+  Bitnami tag (for example `25.7.5-debian-12-r0`), update it to an official image tag such as
+  `26.2` — Bitnami tags don't exist in the official repository. Note the switch is one-way: the
+  official image takes ownership of the data files, so rolling back to the Bitnami image requires
+  manually restoring their previous owner (`chown -R 1001:1001` on the volume).
+</Note>
+
 <Note>
   Trigger.dev 4.5.0 is the last version we officially support for running v3 (SDK v3) tasks. If
   you still have v3 tasks, pin `TRIGGER_IMAGE_TAG` to exactly `v4.5.0` or [migrate to

docs/self-hosting/kubernetes.mdx

@@ -80,6 +80,56 @@ kubectl port-forward svc/trigger-webapp 3040:3030 -n trigger
 kubectl logs -n trigger deployment/trigger-webapp | grep -A1 "magic link"
 ```
 
+## Upgrading
+
+`helm upgrade` keeps generated secrets, PVCs, and datastore volumes, so most upgrades need no preparation. These chart versions are the exceptions.
+
+### 4.5.6: two new required secret keys
+
+The webapp now reads `PROVIDER_SECRET` and `COORDINATOR_SECRET`. When you supply `secrets.existingSecret` the chart generates nothing and reads every key from your Secret, so both keys must exist there before you upgrade. Inline `secrets.*` values are ignored while `existingSecret` is set.
+
+Add them with two fresh 32-character hex values:
+
+```bash
+kubectl patch secret my-trigger-secrets -n trigger --type merge -p \
+  "{\"stringData\":{\"PROVIDER_SECRET\":\"$(openssl rand -hex 16)\",\"COORDINATOR_SECRET\":\"$(openssl rand -hex 16)\"}}"
+```
+
+The chart checks your Secret in a pre-upgrade hook and aborts with the list of missing keys, leaving the running release untouched. Without that check a missing key surfaces as a `CreateContainerConfigError` partway through the webapp rollout.
+
+<Note>
+  GitOps tools that render with `helm template` have no cluster access, so the check is
+  skipped and a missing key still reaches the rollout. Verify the keys yourself before syncing.
+</Note>
+
+### 4.5.6: ClickHouse credentials moved
+
+The bundled ClickHouse password moved out of the subchart-generated Secret and into the chart-managed datastore Secret:
+
+|        | Up to 4.5.5            | 4.5.6 and later             |
+| ------ | ---------------------- | --------------------------- |
+| Secret | `<release>-clickhouse` | `trigger-datastore`         |
+| Key    | `admin-password`       | `clickhouse-admin-password` |
+
+The webapp and the ClickHouse server both read the new location, so the upgrade itself needs no action. Repoint anything outside the chart that reads the old Secret — a maintenance CronJob, a Grafana datasource, an external secret sync:
+
+```bash
+kubectl get secret trigger-datastore -n trigger \
+  -o jsonpath='{.data.clickhouse-admin-password}' | base64 -d
+```
+
+The same move applies to the bundled PostgreSQL (`postgres-password`) and MinIO (`minio-root-user`, `minio-root-password`).
+
+### Bitnami ClickHouse to the official image
+
+The bundled ClickHouse runs the official `clickhouse/clickhouse-server` image instead of the Bitnami subchart. The chart adopts your existing data volume automatically, with three caveats covered under [ClickHouse](#clickhouse) in external services: rendering without cluster access, pinned Bitnami image tags, and storage that doesn't support `fsGroup`.
+
+<Note>
+  The data volume is `ReadWriteOnce`, so during the upgrade the new ClickHouse pod may sit in
+  `ContainerCreating` with a multi-attach warning for a minute while the old pod still holds the
+  volume. This clears itself once the old pod is removed later in the same upgrade - don't abort.
+</Note>
+
 ## Configuration
 
 Most values map directly to the environment variables documented in the [webapp](/self-hosting/env/webapp) and [supervisor](/self-hosting/env/supervisor) environment variable overview.
@@ -138,8 +188,9 @@ secrets:
 # - PROVIDER_SECRET
 # - COORDINATOR_SECRET
 # - MANAGED_WORKER_SECRET
-# - OBJECT_STORE_ACCESS_KEY_ID
-# - OBJECT_STORE_SECRET_ACCESS_KEY
+# Plus s3-auth-access-key-id and s3-auth-secret-access-key if you deploy the
+# bundled MinIO with s3.auth.existingSecret cleared. The chart lists any keys
+# it can't find and fails the install before touching a running release.
 secrets:
   enabled: false
   existingSecret: "your-existing-secret"
@@ -279,26 +330,64 @@ redis:
 
 #### ClickHouse
 
+<Note>Trigger.dev requires ClickHouse 25.8 or newer.</Note>
+
+<Note>
+  When upgrading from a chart version that bundled ClickHouse via the Bitnami subchart, the chart
+  automatically adopts the existing data volume, so no manual migration is needed. If you render
+  manifests without cluster access (for example with GitOps tools that use `helm template`), set
+  `clickhouse.persistence.existingClaim` to the old PVC name
+  (`data-<release>-clickhouse-shard0-0`) to keep your data — auto-detection can't run there, and
+  skipping this starts ClickHouse on a fresh empty volume. If that happened, your old data is
+  still on the old PVC: delete the ClickHouse StatefulSet with `--cascade=orphan` (its volume
+  configuration is immutable), set `existingClaim`, and sync again. If you render without cluster
+  access, pin `clickhouse.persistence.existingClaim` for good once the volume is adopted: a later
+  render where the lookup can't see the PVC would otherwise try to re-add `volumeClaimTemplates`,
+  which the API server rejects on an existing StatefulSet.
+</Note>
+
+<Note>
+  If you pinned `clickhouse.image` to a Bitnami repository or tag in your values, update it to
+  the official `clickhouse/clickhouse-server` image — Bitnami tags don't exist there. The
+  bundled ClickHouse is single-node: the old Bitnami subchart keys (`shards`, `replicaCount`,
+  `keeper`) are no longer supported, so use an external ClickHouse for clustered setups.
+</Note>
+
+<Note>
+  When a volume is adopted from the Bitnami-based chart, a one-time init container fixes its
+  ownership for the non-root ClickHouse server. This runs automatically on `helm upgrade` (and
+  when you set `clickhouse.persistence.existingClaim`), so storage that doesn't support `fsGroup`
+  ownership changes (NFS, hostPath, local-path) works without manual steps. Set
+  `clickhouse.volumePermissions.enabled: true` to force it in other cases.
+</Note>
+
 **Direct configuration:**
 
 ```yaml
 clickhouse:
   deploy: false
   external:
     host: "my-clickhouse.example.com"
-    port: 8123
+    httpPort: 8123
     username: "my-username"
     password: "my-password"
 ```
 
+<Note>
+  An inline external `username`/`password` is percent-encoded into the connection URL for you, so
+  store the **raw** value - special characters like `@ : / %` are handled automatically. If you
+  previously percent-encoded the password by hand to work around this, switch back to the raw value.
+  (Credentials from `existingSecret` are injected at runtime and are unaffected.)
+</Note>
+
 **Using existing secrets (recommended):**
 
 ```yaml
 clickhouse:
   deploy: false
   external:
     host: "my-clickhouse.example.com"
-    port: 8123
+    httpPort: 8123
     username: "my-username"
     existingSecret: "clickhouse-credentials"
     # existingSecretKey: "clickhouse-password" # default (optional)

hosting/docker/.env.example

@@ -136,7 +136,7 @@ OBJECT_STORE_SECRET_ACCESS_KEY=
 # POSTGRES_IMAGE_TAG=14
 # REDIS_IMAGE_TAG=7
 # ELECTRIC_IMAGE_TAG=1.0.13
-# CLICKHOUSE_IMAGE_TAG=latest
+# CLICKHOUSE_IMAGE_TAG=26.2
 # REGISTRY_IMAGE_TAG=2
 # MINIO_IMAGE_TAG=latest
 # DOCKER_PROXY_IMAGE_TAG=latest

hosting/docker/clickhouse/data-paths.xml

@@ -0,0 +1,17 @@
+<!--
+    Keeps ClickHouse's on-disk layout compatible with data volumes created by
+    the previous Bitnami-based setup, which stored everything under a data/
+    subdirectory of the volume. Fresh installs get the same layout. tmp lives
+    outside data/ because old volumes contain a dangling tmp symlink there.
+-->
+<clickhouse>
+    <path>/var/lib/clickhouse/data/</path>
+    <tmp_path>/var/lib/clickhouse/tmp/</tmp_path>
+    <user_files_path>/var/lib/clickhouse/data/user_files/</user_files_path>
+    <format_schema_path>/var/lib/clickhouse/data/format_schemas/</format_schema_path>
+    <user_directories>
+        <local_directory>
+            <path>/var/lib/clickhouse/data/access/</path>
+        </local_directory>
+    </user_directories>
+</clickhouse>

hosting/docker/webapp/docker-compose.yml

@@ -80,11 +80,11 @@ services:
       TRIGGER_BOOTSTRAP_WORKER_GROUP_NAME: bootstrap
       TRIGGER_BOOTSTRAP_WORKER_TOKEN_PATH: /home/node/shared/worker_token
       # ClickHouse configuration
-      CLICKHOUSE_URL: ${CLICKHOUSE_URL:-http://default:${CLICKHOUSE_PASSWORD}@clickhouse:8123?secure=false}
+      CLICKHOUSE_URL: ${CLICKHOUSE_URL:-http://${CLICKHOUSE_USER:-default}:${CLICKHOUSE_PASSWORD}@clickhouse:8123?secure=false}
       CLICKHOUSE_LOG_LEVEL: ${CLICKHOUSE_LOG_LEVEL:-info}
       # Run replication
       RUN_REPLICATION_ENABLED: ${RUN_REPLICATION_ENABLED:-1}
-      RUN_REPLICATION_CLICKHOUSE_URL: ${RUN_REPLICATION_CLICKHOUSE_URL:-http://default:${CLICKHOUSE_PASSWORD}@clickhouse:8123}
+      RUN_REPLICATION_CLICKHOUSE_URL: ${RUN_REPLICATION_CLICKHOUSE_URL:-http://${CLICKHOUSE_USER:-default}:${CLICKHOUSE_PASSWORD}@clickhouse:8123}
       RUN_REPLICATION_LOG_LEVEL: ${RUN_REPLICATION_LOG_LEVEL:-info}
       # Limits
       # TASK_PAYLOAD_OFFLOAD_THRESHOLD: 524288 # 512KB
@@ -157,18 +157,26 @@ services:
       start_period: 10s
 
   clickhouse:
-    image: bitnamilegacy/clickhouse:${CLICKHOUSE_IMAGE_TAG:-latest}
+    image: clickhouse/clickhouse-server:${CLICKHOUSE_IMAGE_TAG:-26.2}
     restart: ${RESTART_POLICY:-unless-stopped}
     logging: *logging-config
     ports:
       - ${CLICKHOUSE_PUBLISH_IP:-127.0.0.1}:9123:8123
       - ${CLICKHOUSE_PUBLISH_IP:-127.0.0.1}:9090:9000
+    ulimits:
+      nofile:
+        soft: 262144
+        hard: 262144
     environment:
-      CLICKHOUSE_ADMIN_USER: ${CLICKHOUSE_USER:-default}
-      CLICKHOUSE_ADMIN_PASSWORD: ${CLICKHOUSE_PASSWORD:?Set CLICKHOUSE_PASSWORD in .env - run ./generate-secrets.sh}
+      CLICKHOUSE_USER: ${CLICKHOUSE_USER:-default}
+      CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD:?Set CLICKHOUSE_PASSWORD in .env - run ./generate-secrets.sh}
+      CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT: 1
     volumes:
-      - clickhouse:/bitnami/clickhouse
-      - ../clickhouse/override.xml:/bitnami/clickhouse/etc/config.d/override.xml:ro
+      # The same volume works across upgrades from the previous Bitnami-based
+      # setup: data-paths.xml keeps the on-disk layout compatible.
+      - clickhouse:/var/lib/clickhouse
+      - ../clickhouse/data-paths.xml:/etc/clickhouse-server/config.d/data-paths.xml:ro
+      - ../clickhouse/override.xml:/etc/clickhouse-server/config.d/override.xml:ro
     networks:
       - webapp
     healthcheck:

hosting/k8s/helm/Chart.lock

@@ -5,11 +5,8 @@ dependencies:
 - name: redis
   repository: oci://registry-1.docker.io/bitnamicharts
   version: 21.2.6
-- name: clickhouse
-  repository: oci://registry-1.docker.io/bitnamicharts
-  version: 9.4.4
 - name: minio
   repository: oci://registry-1.docker.io/bitnamicharts
   version: 17.0.9
-digest: sha256:e1b572ab8eca0cc376311398c27b1734d8a598095fccc81dd9c32b2c8b9c1149
-generated: "2026-05-05T10:31:58.493590751+01:00"
+digest: sha256:a735954c8b78fcf5b30689bdcdfed66b9be57135368ea696aff2b52ecd731474
+generated: "2026-07-13T16:36:15.800113+01:00"

hosting/k8s/helm/Chart.yaml

@@ -26,10 +26,6 @@ dependencies:
     version: "21.2.6"
     repository: "oci://registry-1.docker.io/bitnamicharts"
     condition: redis.deploy
-  - name: clickhouse
-    version: "9.4.4"
-    repository: "oci://registry-1.docker.io/bitnamicharts"
-    condition: clickhouse.deploy
   - name: minio
     version: "17.0.9"
     repository: "oci://registry-1.docker.io/bitnamicharts"

hosting/k8s/helm/templates/_helpers.tpl

@@ -408,45 +408,81 @@ http://{{ include "trigger-v4.fullname" . }}-s2:{{ .Values.s2.service.port }}/v1
 {{- end -}}
 {{- end }}
 
+{{/*
+Percent-encode a string for the userinfo part of a URL. urlquery encodes
+spaces as `+` (query semantics), which userinfo decoding keeps literal; a
+real `+` becomes `%2B`, so any `+` left in the output is a space and can be
+rewritten to `%20`.
+*/}}
+{{- define "trigger-v4.urlencode" -}}
+{{- . | urlquery | replace "+" "%20" -}}
+{{- end }}
+
+{{/*
+ClickHouse data-paths config. Keeps the on-disk layout compatible with data
+volumes created by the Bitnami subchart this chart used previously, which
+stored everything under a data/ subdirectory of the volume. Fresh installs
+get the same layout. tmp lives outside data/ because old volumes contain a
+dangling tmp symlink there. Users can override by defining their own
+data-paths.xml in clickhouse.configdFiles.
+*/}}
+{{- define "trigger-v4.clickhouse.dataPathsConfig" -}}
+<clickhouse>
+  <path>/var/lib/clickhouse/data/</path>
+  <tmp_path>/var/lib/clickhouse/tmp/</tmp_path>
+  <user_files_path>/var/lib/clickhouse/data/user_files/</user_files_path>
+  <format_schema_path>/var/lib/clickhouse/data/format_schemas/</format_schema_path>
+  <user_directories>
+    <local_directory>
+      <path>/var/lib/clickhouse/data/access/</path>
+    </local_directory>
+  </user_directories>
+</clickhouse>
+{{- end }}
+
 {{/*
 ClickHouse hostname
 */}}
 {{- define "trigger-v4.clickhouse.hostname" -}}
 {{- if .Values.clickhouse.host }}
 {{- .Values.clickhouse.host }}
 {{- else if .Values.clickhouse.deploy }}
-{{- printf "%s-clickhouse" .Release.Name }}
+{{- printf "%s-clickhouse" (include "trigger-v4.fullname" .) }}
 {{- end }}
 {{- end }}
 
 {{/*
 ClickHouse URL for application (with secure parameter)
 
-Note on the external+existingSecret branch: the password is expanded via
-Kubernetes' `$(VAR)` syntax, not shell `${VAR}`. Kubelet substitutes
-`$(CLICKHOUSE_PASSWORD)` at container-creation time from the
+Note on the deploy and external+existingSecret branches: the password is
+expanded via Kubernetes' `$(VAR)` syntax, not shell `${VAR}`. Kubelet
+substitutes `$(CLICKHOUSE_PASSWORD)` at container-creation time from the
 CLICKHOUSE_PASSWORD env var declared just before CLICKHOUSE_URL in
 webapp.yaml. Shell-style `${...}` does not work here because
 `docker/scripts/entrypoint.sh` assigns CLICKHOUSE_URL to GOOSE_DBSTRING
 with a single-pass expansion (`export GOOSE_DBSTRING="$CLICKHOUSE_URL"`),
 so any inner `${...}` reaches goose verbatim and fails URL parsing.
 
 CLICKHOUSE_PASSWORD must contain only URL-userinfo-safe characters — the
-value is substituted verbatim, so `@ : / ? # [ ] %` break the URL. Use a
-hex-encoded password or percent-encode before storing in the Secret.
+value is substituted verbatim, so `@ : / ? # [ ] %` break the URL. The
+chart-generated datastore password is hex, which is safe; a pinned
+auth.password or external Secret value must be URL-safe too.
+
+Inline credentials (usernames and the external plain password) are
+percent-encoded, so any special characters are safe there.
 */}}
 {{- define "trigger-v4.clickhouse.url" -}}
 {{- if .Values.clickhouse.deploy -}}
 {{- $protocol := ternary "https" "http" .Values.clickhouse.secure -}}
 {{- $secure := ternary "true" "false" .Values.clickhouse.secure -}}
-{{ $protocol }}://{{ .Values.clickhouse.auth.username }}:$(CLICKHOUSE_PASSWORD)@{{ include "trigger-v4.clickhouse.hostname" . }}:8123?secure={{ $secure }}
+{{ $protocol }}://{{ include "trigger-v4.urlencode" .Values.clickhouse.auth.username }}:$(CLICKHOUSE_PASSWORD)@{{ include "trigger-v4.clickhouse.hostname" . }}:{{ .Values.clickhouse.service.ports.http }}?secure={{ $secure }}
 {{- else if .Values.clickhouse.external.host -}}
 {{- $protocol := ternary "https" "http" .Values.clickhouse.external.secure -}}
 {{- $secure := ternary "true" "false" .Values.clickhouse.external.secure -}}
 {{- if .Values.clickhouse.external.existingSecret -}}
-{{ $protocol }}://{{ .Values.clickhouse.external.username }}:$(CLICKHOUSE_PASSWORD)@{{ .Values.clickhouse.external.host }}:{{ .Values.clickhouse.external.httpPort | default 8123 }}?secure={{ $secure }}
+{{ $protocol }}://{{ include "trigger-v4.urlencode" .Values.clickhouse.external.username }}:$(CLICKHOUSE_PASSWORD)@{{ .Values.clickhouse.external.host }}:{{ .Values.clickhouse.external.httpPort | default 8123 }}?secure={{ $secure }}
 {{- else -}}
-{{ $protocol }}://{{ .Values.clickhouse.external.username }}:{{ .Values.clickhouse.external.password }}@{{ .Values.clickhouse.external.host }}:{{ .Values.clickhouse.external.httpPort | default 8123 }}?secure={{ $secure }}
+{{ $protocol }}://{{ include "trigger-v4.urlencode" .Values.clickhouse.external.username }}:{{ include "trigger-v4.urlencode" .Values.clickhouse.external.password }}@{{ .Values.clickhouse.external.host }}:{{ .Values.clickhouse.external.httpPort | default 8123 }}?secure={{ $secure }}
 {{- end -}}
 {{- end -}}
 {{- end }}
@@ -460,13 +496,13 @@ applies to the replication URL.
 {{- define "trigger-v4.clickhouse.replication.url" -}}
 {{- if .Values.clickhouse.deploy -}}
 {{- $protocol := ternary "https" "http" .Values.clickhouse.secure -}}
-{{ $protocol }}://{{ .Values.clickhouse.auth.username }}:$(CLICKHOUSE_PASSWORD)@{{ include "trigger-v4.clickhouse.hostname" . }}:8123
+{{ $protocol }}://{{ include "trigger-v4.urlencode" .Values.clickhouse.auth.username }}:$(CLICKHOUSE_PASSWORD)@{{ include "trigger-v4.clickhouse.hostname" . }}:{{ .Values.clickhouse.service.ports.http }}
 {{- else if .Values.clickhouse.external.host -}}
 {{- $protocol := ternary "https" "http" .Values.clickhouse.external.secure -}}
 {{- if .Values.clickhouse.external.existingSecret -}}
-{{ $protocol }}://{{ .Values.clickhouse.external.username }}:$(CLICKHOUSE_PASSWORD)@{{ .Values.clickhouse.external.host }}:{{ .Values.clickhouse.external.httpPort | default 8123 }}
+{{ $protocol }}://{{ include "trigger-v4.urlencode" .Values.clickhouse.external.username }}:$(CLICKHOUSE_PASSWORD)@{{ .Values.clickhouse.external.host }}:{{ .Values.clickhouse.external.httpPort | default 8123 }}
 {{- else -}}
-{{ $protocol }}://{{ .Values.clickhouse.external.username }}:{{ .Values.clickhouse.external.password }}@{{ .Values.clickhouse.external.host }}:{{ .Values.clickhouse.external.httpPort | default 8123 }}
+{{ $protocol }}://{{ include "trigger-v4.urlencode" .Values.clickhouse.external.username }}:{{ include "trigger-v4.urlencode" .Values.clickhouse.external.password }}@{{ .Values.clickhouse.external.host }}:{{ .Values.clickhouse.external.httpPort | default 8123 }}
 {{- end -}}
 {{- end -}}
 {{- end }}