test: Update the QA flow for SH

[nfcodacy]

This PR:

1. Allows for manual QA approval even if the tests are failing. This way we can check manually some individual tests and then decide if the release should go on, instead of repeating tests over and over again until they finally pass.

2. Separates the auto tests on two more suites: web_permissions and quarantine.

• The first one stores some heavy tests that are the usual ones that have recurring problems with test data disappearing, so they usually fail on the first tries. This way we can rerun them faster and more easily.

• The second one allows to store tests that are failing constantly, so that we can rerun them quicker than having to rerun entire suites.

[codacy-production]

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

AI Reviewer: first review requested successfully. AI can make mistakes. Always validate suggestions.

_{TIP This summary will be updated as you push new changes.}

[codacy-production]

Pull Request Overview

This PR updates the self-hosted QA workflow by introducing 'web permissions' and 'quarantine' test jobs and decoupling manual approval holds from automated test results. While the automated quality check is 'Up to Standards', the empty PR description provides no context for these significant process changes.

A major point of concern is the removal of automated test dependencies for manual deployment holds (manual_qa_hold and manual_solutions_eng_hold). This change allows builds to be promoted even if critical tests fail, shifting the entire burden of verification to the human reviewer. Ensure this risk is acknowledged and that reviewers are instructed to manually verify test outcomes before approving the holds.

About this PR

• The removal of automated test requirements for manual deployment holds significantly increases the risk of promoting unstable builds to production if human review of test results is bypassed.
• The PR description is empty, providing no context for the changes other than the title and code comments.

Test suggestions

• Execute 'test_web_permissions' job in the release workflow after cookie checks.
• Execute 'test_quarantine' job in the release workflow after cookie checks.
• Allow manual QA hold approval to proceed based only on installation job success, regardless of test job outcomes.

_{TIP Improve review quality by adding custom instructions
TIP How was this review? Give us feedback}