Skip to content

Fix fluentd lifecycle hook: wait for bucket existence, not just MinIO #841

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ricardobernardino2024 merged 2 commits into from
Sep 1, 2026
Merged

Fix fluentd lifecycle hook: wait for bucket existence, not just MinIO #841

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4aaf287
Fix fluentd lifecycle hook waiting for bucket existence, not just MinIO
circleci-app[bot] Sep 1, 2026
ca4d54e
Merge branch 'release-17.0.0' into chunk/fix-fluentd-hook-bucket-wait
ricardobernardino2024 Sep 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 20 additions & 9 deletions codacy/templates/fluentd/lifecycle-police-job.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ metadata:
"helm.sh/hook-weight": "10"
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
spec:
backoffLimit: 3
backoffLimit: 0

@codacy-production codacy-production Bot Sep 1, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 MEDIUM RISK

Suggestion: Setting backoffLimit to 0 prevents the Job from recovering from transient Pod-level failures (such as node pressure or eviction). While the script contains internal retries for the S3 operations, a value of 0 means the entire Helm hook fails immediately if the container is interrupted for environment-level reasons. Consider allowing at least one retry (backoffLimit: 1) to ensure the hook is resilient to infrastructure glitches.

template:
metadata:
name: "{{ .Release.Name }}"
Expand All @@ -35,21 +35,32 @@ spec:
- |
ENDPOINT="http://{{ .Values.global.minio.location }}:{{ .Values.global.minio.port }}"
BUCKET="{{ .Values.fluentdoperator.bucketName }}"
LIFECYCLE='{"Rules":[{"Expiration":{"Days": {{ .Values.fluentdoperator.expirationDays }} },"Filter":{},"ID":"Delete old logs","Status":"Enabled"}]}'
MAX_WAIT=600
LIFECYCLE='{"Rules":[{"Expiration":{"Days": {{ .Values.fluentdoperator.expirationDays }} },"ID":"Delete old logs","Status":"Enabled"}]}'
MAX_WAIT=1200
ELAPSED=0
echo "Waiting for MinIO at ${ENDPOINT}..."
until aws s3 ls --endpoint-url "${ENDPOINT}" > /dev/null 2>&1; do
echo "Waiting for MinIO bucket ${BUCKET} at ${ENDPOINT}..."
until aws s3api head-bucket --bucket "${BUCKET}" --endpoint-url "${ENDPOINT}" > /dev/null 2>&1; do

@codacy-production codacy-production Bot Sep 1, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚪ LOW RISK

Suggestion: Redirecting stderr to /dev/null hides potential configuration, network, or authentication errors, making it difficult to debug why the bucket check is failing during the 20-minute wait window. Removing '2>&1' would allow error messages to appear in the pod logs for easier troubleshooting.

if [ "${ELAPSED}" -ge "${MAX_WAIT}" ]; then
echo "Timed out waiting for MinIO after ${MAX_WAIT}s"
echo "Timed out waiting for MinIO bucket after ${MAX_WAIT}s"
exit 1
fi
echo "Bucket not ready yet (${ELAPSED}s elapsed), retrying in 10s..."
sleep 10
ELAPSED=$((ELAPSED + 10))
done
echo "MinIO is ready. Setting lifecycle configuration on bucket ${BUCKET}..."
aws s3api put-bucket-lifecycle-configuration \
echo "MinIO bucket ${BUCKET} is ready. Setting lifecycle configuration..."
RETRY=0
until aws s3api put-bucket-lifecycle-configuration \
--bucket "${BUCKET}" \
--endpoint-url "${ENDPOINT}" \
--lifecycle-configuration "${LIFECYCLE}"
--lifecycle-configuration "${LIFECYCLE}"; do
RETRY=$((RETRY + 1))
if [ "${RETRY}" -ge 5 ]; then
echo "Failed to set lifecycle configuration after ${RETRY} attempts"
exit 1
fi
echo "Lifecycle config failed, retrying in 10s (attempt ${RETRY}/5)..."
sleep 10
done
echo "Lifecycle configuration set successfully."
{{ end }}