Add logic for pending pods timeout - #482
Conversation
Recently a way to distinguish pending pods from running pods was added. This brings new opportunities for improvement in test-operator, including early exist for stuck (pending) pods. This PR introduces parameter PendingTimeout, which will allow users to set the maximum time they are willing to wait for a pod in a pending state until it is marked as stuck. Once the limit is exceeded we can mark it as stuck and move onto the next pod. This feature will also help in times, when the pods are stuck and therefore exceed the full testing time limit, resulting in no log collection. With this change the job should end early and have logs reporting the pending state. One thing to note is that it is important to make sure pods that exceeded the pending timeout should be considered as failed by jobs. There should be no false positives by introducing this change!
|
Skipping CI for Draft Pull Request. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: kstrenkova The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 3h 05m 55s |
1 participant
Recently a way to distinguish pending pods from running pods was added. This brings new opportunities for improvement in test-operator, including early exist for stuck (pending) pods.
This PR introduces parameter PendingTimeout, which will allow users to set the maximum time they are willing to wait for a pod in a pending state until it is marked as stuck. Once the limit is exceeded we can mark it as stuck and move onto the next pod.
This feature will also help in times, when the pods are stuck and therefore exceed the full testing time limit, resulting in no log collection. With this change the job should end early and have logs reporting the pending state.
One thing to note is that it is important to make sure pods that exceeded the pending timeout should be considered as failed by jobs. There should be no false positives by introducing this change!