RSMetaCheck action is not making job fail

[vuillaut]

Testing RSMetaCheck on one of my repo in gitlab CI, it reveals one issue and one pitfall but the CI does not fail:

https://gitlab.in2p3.fr/gammalearn/gammalearn/-/jobs/2053885

[Anas-Elhounsri]

Testing RSMetaCheck on one of my repo in gitlab CI, it reveals one issue and one pitfall but the CI does not fail:

https://gitlab.in2p3.fr/gammalearn/gammalearn/-/jobs/2053885

Yeah the GitHub action that we created does not fail whether we detect the pitfalls or not, we thought that having the action fail might not suit some users since it is more akin to software failure, but we can still discuss this and see what suits our use case the best.

[vuillaut]

Honestly, as a developer, I don't see the point of having a CI check if it cannot tell me if the check passes or not automatically and I manually have to go see the pipeline outputs.
This can be configurable (since we now have a config file).

[Anas-Elhounsri]

Honestly, as a developer, I don't see the point of having a CI check if it cannot tell me if the check passes or not automatically and I manually have to go see the pipeline outputs.
This can be configurable (since we now have a config file).

I agree, we can have it this way. I will work on an update to the GitHub Actions to have it fail whenever pitfall is detected by default, and let he usee configure it if they wish to.

[vuillaut]

Thank you @Anas-Elhounsri !
I think rsmetacheck should be modified, not the action.

AnalysisConfig should include these new entries:

fail_on_pitfalls: bool = True
fail_on_warnings: bool = True

with the subsequent changes in conf.py to read these

Then in cli.py

# ADDED
def _exit_on_findings(analysis_output: str, analysis_config: AnalysisConfig) -> None:
    try:
        with open(analysis_output) as f:
            data = json.load(f)
    except Exception as e:
        print(f"Warning: Could not read analysis output to evaluate exit code: {e}")
        return

    summary = data.get("summary", {})
    pitfalls = summary.get("total_pitfalls_detected", 0)
    warnings = summary.get("total_warnings_detected", 0)

    should_fail = False
    if analysis_config.fail_on_pitfalls and pitfalls > 0:
        print(f"CI gate: {pitfalls} pitfall(s) detected (fail_on_pitfalls=true).")
        should_fail = True
    if analysis_config.fail_on_warnings and warnings > 0:
        print(f"CI gate: {warnings} warning(s) detected (fail_on_warnings=true).")
        should_fail = True

    if should_fail:
        sys.exit(1)

This way it's configurable and we don't have to modify the action or the gitlab snippet. If the script fails (sys.exit(1)), the CI will fail automatically