Add retry logic for helm install hook failures in release deploy

[circleci-app]

Summary

• Job 30935 (deploy_to_doks_release) failed at "Install Codacy" with: UPGRADE FAILED: post-upgrade hooks failed: job failed: BackoffLimitExceeded
• The stale job cleanup added by Clean up stale hook jobs before helm upgrade to fix BackoffLimitExceeded #836/Fix clean_jobs missing --namespace flag causing BackoffLimitExceeded #837 is working (the old delete-rabbitmq-queues job was successfully deleted before the upgrade), but the newly-created hook job itself is failing at runtime — indicating a transient issue (e.g. RabbitMQ not yet ready when the post-upgrade hook runs)
• This PR converts deploy_to_doks_from_chartmuseum from a prerequisite chain to a recipe with retry: if the first helm install attempt fails, it cleans up the failed hook job and retries once

How it works

-$(MAKE) clean_jobs                        # pre-clean stale jobs (soft fail if none)
$(MAKE) helm_install_from_chart_museum ||  # first attempt
  ($(MAKE) clean_jobs &&                   # on failure: clean hook jobs helm left behind
   $(MAKE) helm_install_from_chart_museum) # retry once

The VERSION variable is propagated through recursive $(MAKE) calls automatically.

https://app.circleci.com/agents/gh/codacy/chat/00bf1479-ab69-4124-be64-d1b83cdaf69f

[codacy-production]

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

AI Reviewer: first review requested successfully. AI can make mistakes. Always validate suggestions.

_{TIP This summary will be updated as you push new changes.}

[codacy-production]

Pull Request Overview

The PR successfully implements the requested retry logic for Helm hook failures within the deployment pipeline. No critical logic flaws or security vulnerabilities were identified. The implementation is up to standards according to Codacy analysis. However, there are no automated tests provided to verify the retry permutations (e.g., success on first try, success on retry, or total failure). While the logic is sound, adding explicit logging is highly recommended to ensure transient failures are visible in CI logs.

Test suggestions

• Verify that a successful first attempt does not trigger the retry logic or the second cleanup.\n- [ ] Verify that a failed first attempt followed by a successful retry results in a successful job completion.\n- [ ] Verify that if both attempts fail, the make target exits with a non-zero status.

Prompt proposal for missing tests

Consider implementing these tests if applicable:
1. Verify that a successful first attempt does not trigger the retry logic or the second cleanup.\n- [ ] Verify that a failed first attempt followed by a successful retry results in a successful job completion.\n- [ ] Verify that if both attempts fail, the make target exits with a non-zero status.

Low confidence findings

• The retry logic is implemented directly in the Makefile using shell-based operators (|| and &&). This pattern can hide transient failures in standard CI/CD dashboard views unless the logs are examined manually. Consider if your CI tool has a more native way to handle retries for specific steps for better visibility.

_{TIP Improve review quality by adding custom instructions
TIP How was this review? Give us feedback}

[codacy-production]

_{⚪ LOW RISK}

Suggestion: Add a log message to signal the retry. This improves observability in CI logs when the initial install fails due to hook jobs.\n\nsuggestion\n\t$(MAKE) helm_install_from_chart_museum || (echo "Helm install failed, retrying after cleaning jobs..." && $(MAKE) clean_jobs && $(MAKE) helm_install_from_chart_museum)\n

[codacy-production]

_{⚪ LOW RISK}

Nitpick: The hyphen prefix is redundant as the 'clean_jobs' target already handles potential errors internally.\n\nsuggestion\n\t$(MAKE) clean_jobs\n