test(e2e): verify serverless SSH remote window and file transfers

[anton-107]

Changes

Add a real-workspace serverless SSH end-to-end test. It invokes the extension's tunnel command, switches to the new Remote SSH window, dismisses onboarding, opens a file from Explorer, and checks the editor content. A small test extension verifies three complete 8 MiB remote file round trips with gaps between them and removes the test files afterward.

Only the SSH spec uses a current VS Code release for Remote SSH compatibility; existing specs retain the extension's minimum supported editor version.

The .ucws.e2e.ts suffix selects the existing serverless CI workspace credentials. Running this spec alone does not start a dedicated cluster. Remote SSH installs through the existing vendored-VSIX mechanism; its optional extension pack is excluded.

Include an isolated local runner with explicit OAuth-profile support and WebdriverIO 9 executable resolution, generic single-spec documentation, and a packaging fix that preserves local package.json edits. The view-title lookup handles missing and empty title attributes in newer VS Code versions.

Tests

• Latest local serverless run passed: 1 test in 2m18s, including Explorer/editor assertions and all three large transfers, using the original Remote SSH 0.128.0 VSIX.
• Verified the original Remote SSH package installs offline without its optional extension pack.
• Lint, formatting, and diff checks passed for the changed test sources.
• Validated full-suite, single-spec, mixed-spec, and glob routing with WebdriverIO 9.29.0: every selected spec runs once with the intended editor version.
• CI validation is in progress. The serverless workspace rejects SSH port 7772 with PERMISSION_DENIED: Traffic on this port is not permitted, before the remote window opens. CI also needs Remote SSH available in its existing EXTENSION_VSIX_DIR.

[rugpanov]

🤖 Integration tests ❌ failed for bf961853.
View run

[rugpanov]

🤖 Integration tests ❌ failed for b2748475.
View run

[rugpanov]

🤖 Integration tests ✅ passed for 1aac0530.
View run

[rugpanov]

Review notes

All test/tooling changes — no shipped extension code. CI is green (Integration Tests + unit). The findings below are about test-suite integrity and reliability, not the extension itself.

Blocking / please address

1. The SSH skip is keyed on Remote-SSH install availability, which both hides coverage and lets the spec leak into the default run.
wdio.conf.ts (beforeSession, catch → TEST_SSH_SKIP_REASON) · ssh_connection.ucws.e2e.ts:18

• Any Remote-SSH install failure (no vendored VSIX / no marketplace) silently becomes a skip, so the job can pass with zero SSH coverage.
• Conversely, the default specs glob **/*.e2e.ts matches this spec and test:integ/test:integ:extension run with no --spec. If Remote SSH is installed but the workspace isn't the serverless-SSH target, the spec runs and findQuickPick("Serverless") times out (fail) rather than skipping.
• Suggest: make skip an explicit opt-in and distinguish "dependency missing → fail the SSH job" from "not an SSH job → skip". Could you confirm how the e2e job invokes the runner (--spec vs full glob)?

2. dismissRemoteWelcome can throw inside the 8-minute waitUntil and abort a passing run.
ssh_connection.ucws.e2e.ts:197-212 (called from the waits at ~453 and ~492)

• action.click() on a stale element, or waitForDisplayed({reverse, timeout:10_000}) when the button doesn't vanish, throws — and a throw in a waitUntil condition rejects immediately, failing the spec on a transient onboarding-UI race.
• Suggest: wrap the click + reverse-wait in try/catch so dismissal is best-effort.

3. The 12-minute test timeout is shorter than the worst-case sum of the internal waits.
ssh_connection.ucws.e2e.ts:12

• login 120s + new-window 180s + ui-ready 480s + picker/explorer/editor/cleanup 4×30s ≈ 15+ min. A slow-but-progressing run is killed by the generic mocha timeout before the targeted timeoutMsg diagnostics and the finally cleanup run.
• Suggest: raise the outer timeout above the aggregate, or drive the flow from one shared deadline.

Should-fix

4. Local runs accumulate credential-bearing temp roots that are never cleaned.
run-e2e.mjs:28 (mkdtempSync per run) · wdio.conf.ts:463 + writeDatabricksConfig (writes token/client_secret into .databrickscfg)

• Each local invocation mints a fresh temp root and leaves behind the .databrickscfg plus hundreds of MiB of VS Code/extension data. onPrepare's start-of-run fs.rm only clears the new (empty) root. CI is unaffected (it leaves TEST_E2E_ROOT unset).
• Suggest: remove the credential-bearing workspace/extension dirs on success, failure, and signals; keep the redacted logs.

5. The probe has no per-operation timeout, so a hung remote FS op leaves the remote window/tunnel open.
ssh-test-probe/extension.js:51 (never reaches closeWindow at line 123)

• The spec still fails via its own 8-min wait, but nothing force-closes the remote window/tunnel on that path.
• Suggest: add op-level watchdog timeouts in the probe and explicitly close the remote window/tunnel from the test teardown.

6. beforeSession rewrites the settings.json that wdio-vscode-service also writes, relying on hook ordering.
wdio.conf.ts (the "The service has already written settings before this user hook runs" block)

• If that ordering ever flips, the read hits ENOENT/partial JSON, or the service overwrites the SSH settings afterward (losing remote.SSH.configFile, trust-disable, terminal profile).
• Suggest: confirm the order holds, or supply these through the service config instead of rewriting the file.

7. run-e2e.mjs --profile runs bare databricks and its error message is misleading on non-auth failures.
run-e2e.mjs:34 (cliJson), catch at ~37

• A fresh clone only has the package-local bin/databricks; bare databricks may be missing or a different version. The catch also always says "Cannot resolve profile… run databricks auth login" even for ENOENT/timeout. (Token correctly kept out of the message.)
• Suggest: invoke the package-local CLI explicitly and distinguish "CLI missing/timed out" from an auth failure.

Nits

• win32 code.cmd shim is unexercised (sshE2eUtils.ts:47-56): no Windows default terminal profile is configured, so this path is effectively posix-only — either validate on Windows CI or document it.
• Spec identity relies on globSync normalizing to exactly ssh_connection.ucws.e2e.ts in two places (capabilities getter + onPrepare); a divergence (case-insensitive FS, symlinked tmp root) could route the spec to the wrong capability. Matches on posix.

Confirmed clean

package-vsix.sh mktemp+EXIT-trap (preserves uncommitted edits — improvement over git checkout --) · commonUtils.ts !title fallback (safe superset of === null) · capability wdio:exclude/splice routing · transfer hashing + atomic .tmp-rename reporting (deadlines resolve to failures, not hangs) · assert→with {type:"json"} update · token scrubbing in run-e2e.mjs.

Two items (#1 runner invocation, #6 hook ordering) depend on the external e2e setup, which isn't visible from this repo — worth a note from you.

[github-actions]

If integration tests don't run automatically, an authorized user can run them manually by following the instructions below:

Trigger:
go/deco-tests-run/vscode

Inputs:

• PR number: 2194
• Commit SHA: 5f1b500c2766f78952bc6f8dbcc2a85b34f3ab3d

Checks will be approved automatically on success.

[anton-107]

Thanks — all seven findings and both nits are addressed in 5f1b500. Taking your two open questions first, since they shaped the fixes.

How the e2e job invokes the runner (#1). From this repo: test:integ → test:integ:extension → run-e2e.mjs with no --spec, i.e. the full glob. The job itself runs in the external workflow that integration-tests.yml triggers, which isn't visible here, so I can't confirm its exact command — and your read of the consequence was right. The opt-in below makes it moot: without the flag the spec is excluded whatever selects it.

Hook ordering (#6). I couldn't verify it, so instead of confirming the order I removed the dependency on it.

Blocking

1. Skip keyed on Remote-SSH availability. Replaced with a TEST_SSH_E2E opt-in, which splits the two cases the skip conflated. Without the flag the spec is in the top-level exclude, so no invocation runs it — the default glob, a directory glob or an explicit --spec. With the flag nothing skips: a failed Remote SSH install now throws out of beforeSession instead of setting a skip reason, so the job can't report success having exercised no part of the tunnel. TEST_SSH_SKIP_REASON is gone from both files. A --spec for the spec without the flag exits naming the flag, rather than leaving wdio to report an empty selection.

2. dismissRemoteWelcome throwing inside waitUntil. Per-label try/catch now covers $$, isDisplayed, click and the reverse-wait, so the function can't reject and the next poll retries. A failure on one label still tries the other.

3. Test timeout below the aggregate. Raised to 18 min, with the budget written out in the comment (login 120 + input 10 + picker 30 + window 180 + probe 480 + explorer 30 + editor 30 + close 30 + probe cleanup 30 = 940s) so a future edit to any wait notices the ceiling.

Should-fix

4. Credential-bearing temp roots. The runner now removes test-root, user-data-dir and extension test from a root it minted — never a caller-supplied TEST_E2E_ROOT, and never in CI — keeping logs/. It hangs off process.on("exit"), so it covers a clean run, a throw from the credential check, and a forwarded signal. Verified on the throw and SIGINT paths: the .databrickscfg and editor data go, logs/ and its contents stay.

5. No per-operation timeout in the probe. Every workspace.fs call and the revealInExplorer command now has a 60s watchdog, so a stuck remote call still reaches its error report and the closeWindow at the end. On top of that the spec's finally waits 30s for the probe to delete its files and close the window itself, then force-closes the window if the probe is wedged past its own watchdogs. The tunnel is a child of the local window's extension host and goes with the session.

6. settings.json rewrite vs. the service. The SSH settings now go into the service capability's userSettings and into the file, and the file read tolerates a missing or unparsable file instead of throwing. Whichever hook writes first, all the keys survive: the capability copy covers a service write after this hook, the file copy covers one that already happened.

7. Bare databricks in the runner. It now prefers the package-local bin/databricks (bin/databricks.exe on Windows, matching CliWrapper.cliPath) and falls back to PATH. The catch separates a missing CLI, a 30s timeout, a non-zero exit and non-JSON output into distinct messages; none of them include captured output. Confirmed execFileSync reports ENOENT, ETIMEDOUT/SIGTERM and a bare status respectively, so each branch routes as intended.

Nits

• win32 shim: documented as unexercised at the branch, noting that no Windows terminal profile is configured either.
• Spec identity: now one isSshSpec helper matching on base name, used by the capability getter, onPrepare and beforeSession, so the three can't drift apart on a symlinked temp root or a case-insensitive filesystem.

Documented the opt-in in CONTRIBUTING.md as well.

One caveat worth flagging: I couldn't run eslint/prettier/tsc or the e2e suite against this change locally — the sandbox has no npm registry access, so yarn install fails. That's also why #6 is order-independent rather than order-confirmed: wdio-vscode-service's source wasn't available to check. I'll rely on CI for lint and formatting.

[rugpanov]

🤖 Integration tests running for 5f1b500c — ⏳.
View run