Increase helm deploy timeout to fix release job timeout - #834
Conversation
The deploy_to_doks_release job (job 30732) timed out because the CircleCI no_output_timeout (15m) matched the helm --timeout (900s). Helm produces no output while waiting for pods to roll out, so CircleCI killed the process just as helm was about to hit its own timeout. Fix: raise no_output_timeout to 30m and helm --timeout to 1500s (25m) so helm has room to complete or fail cleanly before CircleCI intervenes. AI-Generated: true
Up to standards ✅🟢 Issues
|
There was a problem hiding this comment.
Pull Request Overview
The PR successfully increases deployment timeouts to mitigate release job failures. Codacy analysis indicates the changes are up to standards with no new quality issues.
A critical observation is the discrepancy in scope: while the Helm timeout is increased globally via the Makefile, the CircleCI 'no_output_timeout' is only increased for a specific job anchor. This creates a risk where other deployment jobs (such as 'deploy_to_doks_infrastructure') remain vulnerable to premature termination by CircleCI before Helm reaches its own timeout. This inconsistency should be addressed to ensure all deployment paths benefit from the increased duration.
About this PR
- The increase to the Helm --timeout (1500s) in the Makefile applies to all commands using that target. However, the CircleCI no_output_timeout (30m) was only updated for the 'deploy_to_cluster_from_chartmuseum' anchor. Ensure other jobs that use the Helm upgrade command, such as 'deploy_to_doks_infrastructure', also have their CircleCI timeouts adjusted to prevent the runner from killing the job before Helm reaches its internal timeout.
Test suggestions
- Verify CircleCI no_output_timeout is 30m in .circleci/config.yml for the deployment step
- Verify Helm --timeout is 1500s in .do/Makefile for the upgrade command
- Ensure Helm timeout (25m) is less than CircleCI no_output_timeout (30m) to allow Helm to report errors before process termination
TIP Improve review quality by adding custom instructions
TIP How was this review? Give us feedback
Summary
deploy_to_doks_release) timed out at the "Install Codacy" step withmake: *** Terminatedno_output_timeout: 15mand helm--timeout=900s(also 15 min) were identical — helm produces no output while waiting for pods to roll out, so CircleCI killed the process at the same instant helm would have timed outno_output_timeoutto30mand helm--timeoutto1500s(25 min), giving helm a clean window to complete or fail before CircleCI intervenesTest plan
release_pipelineworkflow on a release branch and confirm thedeploy_to_doks_releasejob completes without timing outhttps://app.circleci.com/agents/gh/codacy/chat/26acbca2-3399-4fb6-bfc5-41f9ce1f1539