Skip to content

Increase helm deploy timeout to fix release job timeout - #834

Merged
ricardobernardino2024 merged 1 commit into
release-17.0.0from
chunk/increase-helm-deploy-timeout
Aug 31, 2026
Merged

Increase helm deploy timeout to fix release job timeout#834
ricardobernardino2024 merged 1 commit into
release-17.0.0from
chunk/increase-helm-deploy-timeout

Conversation

@circleci-app

@circleci-app circleci-app Bot commented Aug 31, 2026

Copy link
Copy Markdown

Summary

  • Job 30732 (deploy_to_doks_release) timed out at the "Install Codacy" step with make: *** Terminated
  • Root cause: CircleCI no_output_timeout: 15m and helm --timeout=900s (also 15 min) were identical — helm produces no output while waiting for pods to roll out, so CircleCI killed the process at the same instant helm would have timed out
  • Fix: raise no_output_timeout to 30m and helm --timeout to 1500s (25 min), giving helm a clean window to complete or fail before CircleCI intervenes

Test plan

  • Trigger a new run of the release_pipeline workflow on a release branch and confirm the deploy_to_doks_release job completes without timing out

https://app.circleci.com/agents/gh/codacy/chat/26acbca2-3399-4fb6-bfc5-41f9ce1f1539

The deploy_to_doks_release job (job 30732) timed out because the
CircleCI no_output_timeout (15m) matched the helm --timeout (900s).
Helm produces no output while waiting for pods to roll out, so CircleCI
killed the process just as helm was about to hit its own timeout.

Fix: raise no_output_timeout to 30m and helm --timeout to 1500s (25m)
so helm has room to complete or fail cleanly before CircleCI intervenes.

AI-Generated: true
@codacy-production

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

AI Reviewer: first review requested successfully. AI can make mistakes. Always validate suggestions.

Run reviewer

TIP This summary will be updated as you push new changes.

@codacy-production codacy-production Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

The PR successfully increases deployment timeouts to mitigate release job failures. Codacy analysis indicates the changes are up to standards with no new quality issues.

A critical observation is the discrepancy in scope: while the Helm timeout is increased globally via the Makefile, the CircleCI 'no_output_timeout' is only increased for a specific job anchor. This creates a risk where other deployment jobs (such as 'deploy_to_doks_infrastructure') remain vulnerable to premature termination by CircleCI before Helm reaches its own timeout. This inconsistency should be addressed to ensure all deployment paths benefit from the increased duration.

About this PR

  • The increase to the Helm --timeout (1500s) in the Makefile applies to all commands using that target. However, the CircleCI no_output_timeout (30m) was only updated for the 'deploy_to_cluster_from_chartmuseum' anchor. Ensure other jobs that use the Helm upgrade command, such as 'deploy_to_doks_infrastructure', also have their CircleCI timeouts adjusted to prevent the runner from killing the job before Helm reaches its internal timeout.

Test suggestions

  • Verify CircleCI no_output_timeout is 30m in .circleci/config.yml for the deployment step
  • Verify Helm --timeout is 1500s in .do/Makefile for the upgrade command
  • Ensure Helm timeout (25m) is less than CircleCI no_output_timeout (30m) to allow Helm to report errors before process termination

TIP Improve review quality by adding custom instructions
TIP How was this review? Give us feedback

@ricardobernardino2024
ricardobernardino2024 merged commit d8293ea into release-17.0.0 Aug 31, 2026
9 checks passed
@ricardobernardino2024
ricardobernardino2024 deleted the chunk/increase-helm-deploy-timeout branch August 31, 2026 18:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant