feat(e2e): use xtask for host Linux workflows#2254
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
0a09431 to
93c0246
Compare
|
Label |
93c0246 to
f19d8df
Compare
Agent reviewRecommendation: request changes. The architecture is sound and closely follows the canonical xtask pattern, but I found four functional/security issues plus required documentation drift. No Critical findings. Warnings
Suggestions
cargo-xtask reference comparisonThe implementation aligns strongly with matklad's cargo-xtask specification:
The deliberate departure is that these tasks require Lima, mise, Bash, package managers, systemd, and Podman. That is reasonable for Linux environment orchestration, but it makes explicit prerequisite/version documentation and safe host mutation more important. Validation
The infrastructure audit also found one pre-existing, unrelated drift item: |
There was a problem hiding this comment.
is it correct to say that we are both building and testing OpenShell inside these vms?
i think the goal of bringing in these vms is to recreate arbitrary environments so that we can reproduce issues or run e2e tests.
instead of building the artifacts into the vm, can we instead load artifacts (binary or install packages) into the prepared vm? this lets us keep build a test separate concerns.
the flow in my mind is to
- build the artifacts using existing flows
- load the artifacts into the vm and start openshell
- run e2e/conformance tests against the openshell gateway running in the vm
There was a problem hiding this comment.
I would second that. In our e2e tests I merged #2311 to use prebuilt binaries for all our e2e tests. The goal state that I would envision is that we build the binaries ONCE for a given git SHA and store these in the ghcr or something similar. The testing pipelines would consume these binaries and test them where applicable. Once tests pass, the release pipelines would consume these binaries to package them.
One could even change the ordering of this so that the artifacts that are tested in the e2e tests are the PACKAGES themselves. This would mean that we produce deb or rpm packages (or snaps) and use these to install the artifacts under test in our test infrastructure. This would have the advantage of covering a lot of what our "canary" tests are currently covering.
There was a problem hiding this comment.
instead of building the artifacts into the vm, can we instead load artifacts (binary or install packages) into the prepared vm? this lets us keep build a test separate concerns.
Codex points out that building in a VM may be useful to test the development flow and could ensure that we're able to build cross-distro binaries locally. So we may want to consider keeping it:
So: the reviewer is correct about separating build from test, and the reply proposes the right end state. The only part I would avoid is eliminating build-in-guest entirely—it has value, but as a deliberate diagnostic mode rather than the primary test pipeline.
There was a problem hiding this comment.
We already have infra to build cross-distro binaries, so I think we punt on this for now to keep the PR simple and focused on one problem.
There was a problem hiding this comment.
Assessment from my agent:
- Building in a VM can diagnose whether a source checkout builds in a particular environment.
- It should not be part of the normal E2E or release-validation path.
- “Building cross-distro binaries” is imprecise. OpenShell defines artifact portability through static musl binaries and GNU binaries with a glibc 2.28 floor—not by producing separate binaries on every distro. See architecture/build.md:44.
- Runtime compatibility should test the same binary or package across Ubuntu, CentOS, and RHEL. Otherwise each environment tests a different artifact.
- Merged PR ci(e2e): reuse prebuilt CLI and gateway artifacts #2311 (ci(e2e): reuse prebuilt CLI and gateway artifacts #2311) already established build-once artifact reuse and reduced E2E wall time by 39%.
- Testing the resulting .deb and .rpm packages is the strongest end state because the exact artifacts that pass can subsequently be promoted.
If cross-distro development builds are a real supported contract, add them later as an explicitly separate diagnostic workflow
There was a problem hiding this comment.
is it correct to say that we are both building and testing OpenShell inside these vms?
Yes and no. In the Github Action we use the prebuilt CLI and gateway binaries via the environment variables OPENSHELL_BIN and OPENSHELL_GATEWAY_BIN that are set by the setup actions setup-e2e-cli/setup-e2e-gateway and honored by e2e_build_gateway_binaries. If that's not set, it builds them as usual in the environment (e.g. the VM or, if we weren't using prebuilt binaries, the Github Action machine). The xtask is here just to set up the environment and run a mise task no different than it ran before.
can we instead load artifacts (binary or install packages) into the prepared vm?
The Debian package test requires a provided artifact, so, yes.
|
|
||
| fn driver_available(driver: &str) -> Result<bool, String> { | ||
| let output = Command::new("limactl") | ||
| .args(["start", "--list-drivers"]) |
There was a problem hiding this comment.
pointed out by agent:
--list-drivers only confirms that Lima registered the QEMU driver; it does not confirm that the
required qemu-system-* executable is installed. For example, Lima 2.0.3 can list qemu on a
Homebrew installation with no QEMU executable available. In that case --snapshot selects
Backend::Qemu and fails during limactl start instead of taking the fallback below. Could we verify
that QEMU is operational, or fail early with an actionable installation message?
I reproed it locally on my mac:
• $ limactl start --list-drivers
qemu
vz
krunkit
$ limactl --tty=false start \
--name openshell-qemu-probe \
--arch aarch64 \
--cpus 1 \
--memory 1 \
--disk 1 \
--vm-type qemu \
--plain \
template:ubuntu-26.04
time="2026-07-22T17:52:04-07:00" level=info msg="Terminal is not available, proceeding without
opening an editor"
time="2026-07-22T17:52:04-07:00" level=info msg="Starting the instance `openshell-qemu-probe` with
internal VM driver `qemu`"
time="2026-07-22T17:52:04-07:00" level=fatal msg="failed to find the QEMU binary for the architecture
`aarch64`: exec: \"qemu-system-aarch64\": executable file not found in $PATH"
``
There was a problem hiding this comment.
I split the local VM/lima stuff out of this PR to reduce the scope. This will be resolved in a separate PR that restores that functionality.
f19d8df to
3dd56a9
Compare
krishicks
changed the title
krishicks
changed the title
0062365 to
30aaf83
Compare
krishicks
changed the title
30aaf83 to
13318fb
Compare
This adds a cargo xtask entry point for running Podman end-to-end tests locally or on a prepared Linux host, using the scripts that used to be in GitHub Actions as the basis. The --setup mode validates an explicit OS selection and composes focused setup scripts for Ubuntu 24.04 and Ubuntu 26.04. Signed-off-by: Kris Hicks <khicks@nvidia.com>
13318fb to
57d7be3
Compare
4 participants
Summary
This adds a cargo xtask entry point for running Podman end-to-end tests locally or on a prepared Linux host, using the scripts that used to be in Github Actions as the basis. The host provider validates typed OS selection and composes focused setup scripts for Ubuntu 24.04 and Ubuntu 26.04.
Related Issue
Changes
Testing
mise run pre-commitpassesChecklist