Fix fluentd lifecycle hook: add Filter field and pre-push delete - #840
Conversation
… pre-push delete
Two fixes for the persistent deploy_to_doks_release failure (job 31035):
1. codacy/templates/fluentd/lifecycle-police-job.yaml: Add "Filter":{} to
the lifecycle configuration JSON. Newer MinIO versions strictly follow the
AWS S3 spec which requires a Filter element in each lifecycle rule, even
when empty. Without it, put-bucket-lifecycle-configuration returns a
MalformedXML error immediately, causing 3 fast pod failures and
BackoffLimitExceeded (the ~15 min total time is main-deployment rollout;
the hook itself fails in seconds).
2. .circleci/config.yml: Delete the existing chart version from chart museum
before pushing. Chart museum rejects re-uploads of the same version with
409, which failed the first pipeline run (job 31005). Adding a pre-push
DELETE via the chart museum REST API (with || true to tolerate missing
charts) ensures subsequent CI runs can always push the updated chart.
AI-Generated: true
Up to standards ✅🟢 Issues
|
![codacy-production[bot]](https://avatars.githubusercontent.com/in/56611?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull Request Overview
This PR addresses deployment failures by updating the fluentd lifecycle configuration for S3/MinIO compatibility and adding a pre-push cleanup step in the CI pipeline. Although the analysis shows the PR is up to standards according to Codacy, a critical risk was identified in the .circleci/config.yml. The DELETE request used to clear existing chart versions lacks validation for the version string; an empty value could result in the deletion of all chart versions. Additionally, while the template changes correctly implement the 'Filter' field, the manual JSON construction is fragile and should be hardened with default values.
Test suggestions
- Verify the presence of 'Filter':{} in the lifecycle policy JSON within the fluentd-lifecycle-policy-creator template.
- Verify that 'curl -X DELETE' is called with correct credentials and URL before 'helm push' in .circleci/config.yml.
- Ensure the 'helm_push' job handles cases where the chart version does not exist on ChartMuseum without failing the pipeline step.
TIP Improve review quality by adding custom instructions
TIP How was this review? Give us feedback
| ENDPOINT="http://{{ .Values.global.minio.location }}:{{ .Values.global.minio.port }}" | ||
| BUCKET="{{ .Values.fluentdoperator.bucketName }}" | ||
| LIFECYCLE='{"Rules":[{"Expiration":{"Days": {{ .Values.fluentdoperator.expirationDays }} },"ID":"Delete old logs","Status":"Enabled"}]}' | ||
| LIFECYCLE='{"Rules":[{"Expiration":{"Days": {{ .Values.fluentdoperator.expirationDays }} },"Filter":{},"ID":"Delete old logs","Status":"Enabled"}]}' |
There was a problem hiding this comment.
⚪ LOW RISK
Suggestion: The inclusion of '"Filter": {}' correctly implements S3 Lifecycle V2 requirements. Note that this manually constructed JSON string will be malformed if '.Values.fluentdoperator.expirationDays' is missing or non-numeric, which would cause the job to fail.
Try running the following prompt in your coding agent:
Update the Helm template to use a default value for expirationDays and ensure it is treated as a numeric value to prevent generating malformed JSON.
PR #841 accidentally removed the `"Filter":{}` field that PR #840 had added to the S3 lifecycle rule JSON. Newer MinIO versions strictly follow the AWS S3 spec and require a Filter element in each lifecycle rule (even when empty). Without it, put-bucket-lifecycle-configuration returns MalformedXML immediately, causing the in-shell retry loop to exhaust its 5 attempts and exit with failure, which Kubernetes surfaces as BackoffLimitExceeded on the post-upgrade hook. AI-Generated: true Co-authored-by: circleci-app[bot] <127350680+circleci-app[bot]@users.noreply.github.com>
1 participant
Summary
Fixes the recurring
BackoffLimitExceededfailure indeploy_to_doks_release(job 31035, step "Install Codacy").Root cause analysis:
The 15-minute timing in the failed job is the main Codacy deployment rollout. After pods are ready, the post-upgrade hook (
fluentd-lifecycle-policy-creatorjob) runs and fails immediately because the lifecycle configuration JSON is missing theFilterfield now required by the MinIO version in use. WithbackoffLimit: 3(set by PR #839), three quick failures triggerBackoffLimitExceeded.A secondary issue: chart museum returns
409: already existswhen a CI run tries to pushcodacy-17.0.0-RC-1.tgzagain (seen in job 31005). This caused the first pipeline run to fail athelm_push_incubator, and the fix from PR #839 only reached chart museum when the chart was cleared between runs.Changes:
codacy/templates/fluentd/lifecycle-police-job.yaml: Add"Filter":{}to the lifecycle rule JSON. The AWS S3 spec (and newer MinIO) requires aFilterelement in every lifecycle rule. Without it,put-bucket-lifecycle-configurationreturnsMalformedXMLand the hook pod exits non-zero on the first attempt..circleci/config.yml: Beforehelm push, call the ChartMuseum REST API to delete any existing version (DELETE /api/charts/codacy/<version>). Uses|| trueso it is a no-op when the version doesn't exist yet. This ensures each CI run can cleanly push the updated chart without 409 conflicts.Test plan
helm_push_incubatorsucceeds without 409 on a re-push of the same RC versiondeploy_to_doks_releasecompletes withoutBackoffLimitExceededon the next pipeline runhttps://app.circleci.com/agents/gh/codacy/chat/3b6f1bd6-1439-46f0-a4f9-9a1c30717a7d