Skip to content

Fix TestUpdateNPMInstall timing out on a cold npm cache - #485

Open
joe4dev wants to merge 1 commit into
mainfrom
devx-1108-npm-install-test-timeout
Open

Fix TestUpdateNPMInstall timing out on a cold npm cache#485
joe4dev wants to merge 1 commit into
mainfrom
devx-1108-npm-install-test-timeout

Conversation

@joe4dev

@joe4dev joe4dev commented Sep 4, 2026

Copy link
Copy Markdown
Member

Motivation

TestUpdateNPMInstall is failing on main, on both the Windows and the ubuntu runners. The two report it very differently, which is what made it look like a registry problem.

Windows — npm claims success, process exits 1:

=== FAIL: . TestUpdateNPMInstall (168.29s)
    Messages: npm install failed:
              added 2 packages, and audited 3 packages in 2m
              found 0 vulnerabilities

Ubuntu — the smoking gun, killed at exactly the deadline with no output:

=== FAIL: . TestUpdateNPMInstall (120.01s)
    Error: signal: killed
    Messages: npm install failed:

It is a timeout, not an npm error. The test runs on the shared testContext(t)2 minutes — and nearly all of that goes into the npm install from the registry, a cost set by the registry rather than by the test.

How long it takes when it passes (windows-latest, last six green main runs):

run duration
33663720068 19.18s
33661716590 81.30s
33659332647 21.38s
33636690396 21.62s
33608951873 35.86s
33382161264 62.11s

So normally 19-81s — but on a slow day the install alone stretches past two minutes (one failing run's npm reported in 3m), and the 2-minute budget turns that into a red build. Ubuntu's 120.01s duration is exactly the context deadline, which is the clearest confirmation available.

The Windows signature is misleading for a specific reason: npm there is a .cmd wrapper around node, so exec.CommandContext kills the wrapper while the node grandchild keeps going, finishes, and writes its success summary into the still-open pipe. On Linux npm is a real exec, so the kill surfaces honestly as signal: killed.

Solution

  • Give the test its own 5-minute context instead of testContext, following the existing precedent in sam_e2e_test.go and license_test.go (both opt out with a comment for the same class of reason). That is ~3.7x the slowest observed passing run and still clears the ~3m slow-registry outlier, so the budget stops being the thing that fails without letting a genuinely hung install sit for ages.
  • --no-audit --no-fund on the local install — small next to the download, but free.
  • Set cmd.WaitDelay on both npm-spawning execs, so a deadline kill can't leave CombinedOutput blocked on a grandchild holding the pipe (the DEVX-846 lesson, already documented in CLAUDE.md).
  • Include ctx.Err() in both failure messages, so the next timeout says "context deadline exceeded" instead of impersonating an npm failure. This is what made the original diagnosis slow.

Test-only change; no production code touched.

Manual testing

The test skips when lstk is on PATH (npm install -g would collide), and forcing it to run installs @localstack/lstk globally, so I did not run it end to end locally. Sizing came from the CI durations above instead. Locally I timed the individual phases in throwaway dirs (global install into a temp --prefix) to confirm which one dominates:

npm install @localstack/lstk                             # slow path: the download
npm install --no-audit --no-fund @localstack/lstk        # 1s once cached
npm install -g --prefix "$(mktemp -d)" @localstack/lstk  # 1s — reuses that cache

The last line is the useful one: the npm install -g inside lstk update is not a second slow install, it reuses the cache the first one filled. Only one phase is expensive.

CI on this PR is the real verification — the Integration Tests (windows-latest) and Integration Tests (ubuntu-latest) jobs, both of which fail on main today.

Docs

Nothing to document — test-infrastructure only. No user-facing command, flag, env var, or behavior changes.

Review

Human review advised — the diagnosis is worth a second pair of eyes even though the change is test-only, since the 5-minute budget is a judgement call.

Closes DEVX-1108

@joe4dev joe4dev added semver: patch docs: skip Pull request does not require documentation changes labels Sep 4, 2026
@joe4dev
joe4dev force-pushed the devx-1108-npm-install-test-timeout branch 2 times, most recently from e1a696e to eb9ebbd Compare September 4, 2026 11:56
Co-Authored-By: Claude <noreply@anthropic.com>
@joe4dev
joe4dev force-pushed the devx-1108-npm-install-test-timeout branch from eb9ebbd to f6a3d4f Compare September 4, 2026 12:03
// grandchild to finish and write its success summary into the still-open
// pipe, and the failure reads "exit status 1" beside output claiming the
// install succeeded.
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

main change: increasing timeout from 2 min to 5 min

// deadline leaves the node grandchild holding the output pipe, and
// CombinedOutput would block for that grandchild's full lifetime (the
// DEVX-846 lesson).
npmInstall := exec.CommandContext(ctx, "npm", "install", "--no-audit", "--no-fund", "@localstack/lstk")

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

supporting change: skipping audit and fund is recommended in CI https://www.lowlydba.com/cicd-npm-flags/

@joe4dev
joe4dev marked this pull request as ready for review September 4, 2026 12:21
@joe4dev
joe4dev requested review from a team and peter-smith-phd as code owners September 4, 2026 12:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs: skip Pull request does not require documentation changes semver: patch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant