Repository navigation
Run the badssl.com tests in CI without failing the build - #39
Merged
Merged
Conversation
The tests that connect to badssl.com fail in most CI runs with TLS handshake errors or timeouts, and different tests fail each time, because the badssl.com servers are unreliable. Tag those tests with "Requires", "badssl", so the test runner skips them unless it is given /badssl, and register httpclient_test_badssl and websocketsclient_test_badssl, labelled badssl, to run only them. In CI, run the other tests with -LE badssl, then run the badssl.com tests in a separate step that does not fail the job. Signed-off-by: Gareth Sylvester-Bradley <garethsb@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The tests that connect to badssl.com fail in most CI runs with TLS handshake errors or timeouts, and a different set fails each time, because the badssl.com servers are unreliable. That keeps every pull request red, and makes it impossible to require status checks.
This PR keeps running those tests, but it no longer lets them fail the job:
"Requires", "badssl". The test runner skips a test with that tag unless it gets/badssl. There are 12 inoutside_tests.cpp, the two HTTPS-to-HTTP redirect tests inredirect_tests.cpp, and 3 websocket client tests inauthentication_tests.cpp.httpclient_test_badsslandwebsocketsclient_test_badssl, labelledbadssl. They runtest_runner /badssl /select:@Requires=badssl, so they run only the tagged tests.ctest -LE badssl, and then run the badssl.com tests withctest -L badsslin a separate step withcontinue-on-error. The step still shows in each job's log when it fails.Running plain
ctestlocally still runs everything, including the badssl.com tests.#38 tracks replacing badssl.com with local TLS servers, so that these tests can go back into the main test step.
Testing
I tested a Release build with Clang 21 on Ubuntu 26.04, with WebSockets enabled.
ctest -LE badsslruns the 8 normal tests, and none of the badssl.com tests are in their listing.ctest -L badsslruns the 2 new tests, which run exactly the 17 tagged tests. Both pass.#37 adds a Clang job that runs plain
ctest. Whichever of the two pull requests merges second needs to give that job the same two test steps.Part of #38.
Co-authored-by: Gareth's 🤖 buddy