Skip to content

ci: require setup-matrix (except maven/nuget) + run live-API e2e on PR#103

Closed
Mikola Lysenko (mikolalysenko) wants to merge 1 commit into
mainfrom
ci/enforce-setup-matrix-and-live-api-e2e
Closed

ci: require setup-matrix (except maven/nuget) + run live-API e2e on PR#103
Mikola Lysenko (mikolalysenko) wants to merge 1 commit into
mainfrom
ci/enforce-setup-matrix-and-live-api-e2e

Conversation

@mikolalysenko
Copy link
Copy Markdown
Collaborator

Removes the known_gap bypasses so the setup-flow matrix fails CI when an
ecosystem's setup integration is broken, and enables the live-API e2e
smoke suites as blocking PR checks. maven and nuget stay experimental.

tests/setup_matrix/matrix.json

  • Flip baseline_supported: false → true for poetry, pdm, cargo, bundler,
    go, composer, deno
    (single) and pip, uv (workspace). An unmet ideal
    now classifies as a blocking regression instead of a non-blocking
    known_gap.
  • maven/mvn and nuget/dotnet remain baseline_supported: false.
  • Updated the matrix comments: baseline_supported now marks "required to
    support setup" (true everywhere except maven/nuget).

.github/workflows/ci.yml

  • Drop continue-on-error: true on the setup-matrix job so the
    orchestrator's non-zero exit (on regression) fails the job.
  • Add e2e_npm / e2e_pypi / e2e_gem / e2e_scan to the e2e PR matrix.
    They keep #[ignore] and are selected via -- --ignored; the existing
    Node/Python/Ruby setup steps already key off these suite names.

Expected effects / caveats

  • These legs will go red until setup actually wires the hooks for the
    now-required ecosystems (cargo, gem, golang, composer, deno, and the pypi
    poetry/pdm + pip/uv workspace cases). That red is the intended "CI fails
    if broken" signal — no underlying setup logic was changed here.
  • The live-API suites hit the real public proxy (patches-api.socket.dev)
    and can go red on upstream 503s rather than a true regression; the hermetic
    e2e-docker matrix remains the always-on counterpart.
  • For these jobs to block merges, they must be added to the repo's
    required status checks (branch-protection UI — not configurable from the
    workflow file).

Remove the known_gap bypasses so the setup-flow matrix fails CI when an
ecosystem's `setup` integration is broken, and enable the live-API e2e
smoke suites as blocking PR checks. maven and nuget stay experimental.

matrix.json:
- Flip baseline_supported false -> true for poetry, pdm, cargo, bundler,
  go, composer, deno (single) and pip, uv (workspace). An unmet ideal now
  classifies as a blocking *regression* instead of a non-blocking
  known_gap. maven/mvn and nuget/dotnet remain baseline_supported:false.
- Update the matrix comments to reflect that baseline_supported now marks
  "required to support setup" (true everywhere except maven/nuget).

ci.yml:
- Drop `continue-on-error: true` on the setup-matrix job so the
  orchestrator's non-zero exit (on regression) fails the job.
- Add e2e_npm/pypi/gem/scan to the e2e PR matrix. They keep #[ignore]
  and are selected via `-- --ignored`; the existing Node/Python/Ruby
  setup steps already key off these suite names. These hit the real
  public proxy and may go red on upstream 503s rather than a regression;
  the hermetic e2e-docker matrix remains the always-on counterpart.

Note: for these jobs to block merges they must be listed in the repo's
required status checks (branch-protection UI). Underlying setup support
for the now-required ecosystems is unchanged, so those legs are expected
to go red until `setup` wires their install hooks.

Assisted-by: Claude Code:opus-4-8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant