Skip to content

Fix the two remaining sample CI failures - #273

Closed
peter-smith-phd wants to merge 1 commit into
masterfrom
fix-emr-build-and-terraform-provider-pin
Closed

Fix the two remaining sample CI failures#273
peter-smith-phd wants to merge 1 commit into
masterfrom
fix-emr-build-and-terraform-provider-pin

Conversation

@peter-smith-phd

Copy link
Copy Markdown
Contributor

Fixes the last two failing sample jobs. Neither is caused by the sample code — both are drift in something the sample pulls in at build time.

Stacked on top of #272 conceptually but branched from master; the two are independent and can land in either order.

emr-serverless-python-dependencies — venv build failure

process "/bin/bash -c python3 -m venv $VIRTUAL_ENV" did not complete successfully: exit code: 1
  apt install python3.14-venv

localstack/localstack:latest has moved to Debian forky/sid with Python 3.14, and Debian splits ensurepip out of the stdlib into a separate pythonX.Y-venv package, so python3 -m venv can't bootstrap pip.

The base image can't simply be swapped. This build produces a venv that docker-compose mounts into the LocalStack container as PYSPARK_PYTHON at /tmp/environment, so its interpreter must match the container's. So instead: install the missing package, deriving the version from the image's own interpreter rather than hardcoding 3.14, so it keeps working as the base image moves on.

RUN apt-get update \
    && apt-get install -y --no-install-recommends \
        "python$(python3 -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")')-venv" \
    && rm -rf /var/lib/apt/lists/*

This also covers a second latent break: poetry bundle venv creates its own venv and would have hit the same wall.

terraform-resources — AWS provider state waiter

Error: waiting for API Gateway REST API (p7gdc8clzx) to become available:
unexpected state '', wanted target 'AVAILABLE'

The sample declares no required_providers block, so terraform init resolves the latest AWS provider on every run — CI got 6.60.0. Bisected against LocalStack:

Provider Result
6.0.0 – 6.56.0 PASS
6.57.0+ FAIL

6.57.0 added a state waiter for aws_api_gateway_rest_api that polls a field LocalStack's CreateRestApi/GetRestApi response doesn't return (id, name, createdDate, apiKeySource, endpointConfiguration, disableExecuteApiEndpoint, rootResourceId — no state), so it reads '' and waits until timeout.

Pinned to ~> 6.56.0 (>= 6.56.0, < 6.57.0), matching the style already used by lambda-function-urls-javascript and emr-serverless-python-dependencies.

⚠️ This is a stopgap that hides the gap rather than closing it. Users on provider ≥ 6.57.0 still hit this against LocalStack outside this repo — the API Gateway response is worth fixing upstream, after which the pin can be relaxed.

Verification

Both verified locally against a live LocalStack:

  • EMR — reproduced the failure, then the image builds through poetry bundling and export. The resulting bundle, mounted at /tmp/environment inside localstack/localstack:latest, runs and imports both typer and the jobs package.
  • terraform-resources — reproduced with the latest provider, then the sample's own make run applies all 10 resources, GetRestApis confirms auth-demo exists, and destroy removes all 10.

Note on appsync-graphql-api

appsync-graphql-api was failing on master but passes on #272 — its except websockets.exceptions.ConnectionClosedOK clause (broken by websockets ≥ 14, which dropped exceptions from the lazy top-level __getattr__) is only evaluated when an exception propagates, and none does once awslocal works. Deliberately left alone here as it isn't blocking CI.

Draft so CI can exercise the full matrix.

🤖 Generated with Claude Code

Both samples fail for unrelated reasons, neither caused by the sample code
itself; both are drift in something the sample pulls in at build time.

emr-serverless-python-dependencies

  The build now aborts with "process /bin/bash -c python3 -m venv
  $VIRTUAL_ENV did not complete successfully: exit code 1". The
  localstack/localstack:latest base image has moved to Debian forky/sid with
  Python 3.14, and Debian splits ensurepip out of the stdlib into a separate
  pythonX.Y-venv package, so python3 -m venv cannot bootstrap pip.

  The base image cannot simply be swapped: this build produces a venv that
  docker-compose mounts into the LocalStack container as PYSPARK_PYTHON at
  /tmp/environment, so its interpreter has to match the container's. Instead
  install the missing package, deriving the version from the image's own
  interpreter rather than hardcoding 3.14 so this keeps working as the base
  image moves on. This also covers a second latent break: poetry-plugin-bundle
  creates its own venv and would have hit the same wall.

terraform-resources

  Creation of aws_api_gateway_rest_api fails with "unexpected state '',
  wanted target 'AVAILABLE'". The sample declares no required_providers
  block, so terraform init resolves the latest AWS provider on every run.
  Bisecting against LocalStack puts the break exactly at 6.57.0, which added
  a state waiter polling a field LocalStack's CreateRestApi/GetRestApi
  response does not return; 6.56.0 and everything before it are fine.

  Pin to ~> 6.56.0, matching the style already used by
  lambda-function-urls-javascript and emr-serverless-python-dependencies.

  This is a stopgap that hides the gap rather than closing it. Users on
  provider >= 6.57.0 still hit this against LocalStack outside this repo, so
  the API Gateway response is worth fixing upstream.

Verified locally against a live LocalStack. The EMR image builds through
poetry bundling and export, and the resulting bundle mounted at
/tmp/environment inside localstack/localstack:latest runs and imports both
typer and the jobs package. terraform-resources applies all 10 resources via
its own make run, GetRestApis confirms auth-demo exists, and destroy removes
all 10.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@peter-smith-phd

Copy link
Copy Markdown
Contributor Author

Folded into #272 so all sample CI fixes land as a single PR. The commit here is cherry-picked onto that branch unchanged; no content lost.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant