Clean up stale hook jobs before helm upgrade to fix BackoffLimitExceeded

[circleci-app]

Problem

deploy_to_doks_release (job 30795) fails at the Install Codacy step with:

Error: UPGRADE FAILED: release codacy-release failed, and has been rolled back due to atomic being set:
post-upgrade hooks failed: job failed: BackoffLimitExceeded

The root cause: a previous failed deployment (with --atomic) rolls back but leaves post-upgrade hook Jobs (e.g. DB migrations) in the codacy-release namespace. These Jobs have a hook-succeeded delete policy, so they survive the rollback. On the next attempt, Helm finds the pre-existing failed Job with the same name and immediately reports BackoffLimitExceeded without even running the hook body.

Fix

Add a clean_jobs target in .do/Makefile that deletes all Jobs in the namespace before the upgrade:

.PHONY: clean_jobs
clean_jobs:
    -kubectl delete jobs --all -n ${NAMESPACE} --ignore-not-found=true

Wire it into deploy_to_doks_from_chartmuseum between setup_helm_repos and helm_install_from_chart_museum. The - prefix suppresses Make errors if no jobs exist, and --ignore-not-found=true makes kubectl itself non-fatal when the namespace is clean.

This is safe because codacy-release is a dedicated Helm release namespace — any jobs present before an upgrade are stale hook artifacts from prior failed runs.

https://app.circleci.com/agents/gh/codacy/chat/6104887e-b55b-423f-8544-1876b003d7f1

[codacy-production]

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

AI Reviewer: first review requested successfully. AI can make mistakes. Always validate suggestions.

_{TIP This summary will be updated as you push new changes.}

[codacy-production]

Pull Request Overview

The PR successfully addresses the 'BackoffLimitExceeded' blocker for Helm upgrades by introducing a job cleanup step. While the technical implementation aligns with the stated intent, it introduces a significant operational risk by using kubectl delete jobs --all. This command is destructive and will terminate unrelated workloads if the namespace is shared.

Furthermore, there is a consistency gap between deployment paths: the cleanup logic is integrated into the ChartMuseum workflow but omitted from the local directory deployment flow. While Codacy grades the PR as being up to standards, these logic and safety concerns should be addressed to ensure deployment stability across all environments.

About this PR

• The deployment logic is inconsistent. If stale jobs cause 'BackoffLimitExceeded' errors, this will still occur when developers or CI systems use the 'deploy_to_doks_from_local_dir' target. Ensure the cleanup is applied to all deployment workflows.

Test suggestions

• Verify 'clean_jobs' successfully deletes all existing Jobs in a populated namespace.
• Verify 'clean_jobs' does not fail when no Jobs exist in the namespace (idempotency).
• Verify 'deploy_to_doks_from_chartmuseum' executes 'clean_jobs' specifically after context setup but before 'helm_install_from_chart_museum'.

Prompt proposal for missing tests

Consider implementing these tests if applicable:
1. Verify 'clean_jobs' successfully deletes all existing Jobs in a populated namespace.
2. Verify 'clean_jobs' does not fail when no Jobs exist in the namespace (idempotency).
3. Verify 'deploy_to_doks_from_chartmuseum' executes 'clean_jobs' specifically after context setup but before 'helm_install_from_chart_museum'.

Low confidence findings

• The PR lacks automated verification scripts to ensure the 'kubectl' command behaves correctly within the CI/CD pipeline environment without failing the build.

_{TIP Improve review quality by adding custom instructions
TIP How was this review? Give us feedback}