Skip to content

feat: support native SSM expiration policies for runner token parameters - #5265

Merged
guicaulada merged 6 commits into
github-aws-runners:mainfrom
jvrplmlmn:jvrplmlmn/ssm-token-ttl-expiration-policy
Sep 23, 2026
Merged

feat: support native SSM expiration policies for runner token parameters#5265
guicaulada merged 6 commits into
github-aws-runners:mainfrom
jvrplmlmn:jvrplmlmn/ssm-token-ttl-expiration-policy

Conversation

@jvrplmlmn

@jvrplmlmn jvrplmlmn commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Description

Registration tokens and JIT configs written to SSM are orphaned when an instance fails to boot or is terminated before registering. The only cleanup today is the scheduled SSM housekeeper lambda, which cannot delete parameters younger than a day, so orphaned parameters (billed hourly when on the Advanced tier) can live for up to ~2 days.

The new opt-in ssm_token_ttl_seconds attaches a native SSM Expiration policy to the token / JIT config parameters so SSM deletes leftovers itself once the TTL passes. Parameter policies require the Advanced tier, which incurs additional cost per parameter, hence disabled by default. Expiration is enforced asynchronously by SSM and is a cleanup mechanism rather than a security boundary; the housekeeper lambda remains as a backstop.

Test Plan

Related Issues

@jvrplmlmn jvrplmlmn mentioned this pull request Aug 11, 2026
Open
@edersonbrilhante

edersonbrilhante commented Aug 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Nice idea. Just a small note, I am working ro refactor the module to make easier to maintain and add new features.
I have some idea to nest some variables to organize. It is a bit hard to keep track the features. I don't mind adding this new variable, but I would propose to deprecate in following major version and move it to a nested ssm variable.

@Brend-Smits @npalm @guicaulada what do you think?

@guicaulada

guicaulada commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

@jvrplmlmn would mind rebasing and fixing conflicts on this one :)

@guicaulada
guicaulada force-pushed the jvrplmlmn/ssm-token-ttl-expiration-policy branch from 1dd5618 to c99bf0a Compare September 8, 2026 13:11
@guicaulada
guicaulada marked this pull request as ready for review September 8, 2026 13:36
@guicaulada
guicaulada requested review from a team as code owners September 8, 2026 13:36
@guicaulada
guicaulada force-pushed the jvrplmlmn/ssm-token-ttl-expiration-policy branch 4 times, most recently from 1b78804 to 3ecca00 Compare September 9, 2026 16:20
edersonbrilhante
edersonbrilhante reviewed Sep 9, 2026
View reviewed changes
Comment thread modules/multi-runner/variables.tf Resolved
@guicaulada
guicaulada force-pushed the jvrplmlmn/ssm-token-ttl-expiration-policy branch from 527aabc to b0f983a Compare September 11, 2026 01:15
@guicaulada
guicaulada force-pushed the jvrplmlmn/ssm-token-ttl-expiration-policy branch 2 times, most recently from 54cbd09 to ac36954 Compare September 22, 2026 12:47
@edersonbrilhante

edersonbrilhante commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

@guicaulada Here I would like a second opinion. How about add support for this feature in global level(v2) too. So can be configure in global level and allow override per runner entry.

https://github.com/github-aws-runners/terraform-aws-github-runner/blob/main/modules/multi-runner/variables.experimental.storage-provider.tf#L31

Also @guicaulada the last commit not signed

@guicaulada

guicaulada commented Sep 23, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Weird.. why is the docs commit not signed? I thought we had fixed that already.

EDIT: Apparently we didn't fix it for forks.

jvrplmlmn and others added 6 commits September 23, 2026 09:50
@jvrplmlmn @guicaulada
feat: support native SSM expiration policies for runner token parameters
61ef158 
Registration tokens and JIT configs written to SSM are orphaned when an
instance fails to boot or is terminated before registering. The only
cleanup today is the scheduled SSM housekeeper lambda, which cannot
delete parameters younger than a day, so orphaned parameters (billed
hourly when on the Advanced tier) can live for up to ~2 days.

The new opt-in `ssm_token_ttl_seconds` attaches a native SSM Expiration
policy to the token / JIT config parameters so SSM deletes leftovers
itself once the TTL passes. Parameter policies require the Advanced
tier, which incurs additional cost per parameter, hence disabled by
default. Expiration is enforced asynchronously by SSM and is a cleanup
mechanism rather than a security boundary; the housekeeper lambda
remains as a backstop.
@github-actions @guicaulada
docs: auto update terraform docs
4feb1d7
@guicaulada
refactor(ssm): nest token expiration configuration
6e927b1
@guicaulada
fix(ssm): omit token TTL environment variable when disabled
3e80786
@guicaulada
test(ssm): align token TTL coverage with storage providers
18b02ff
@github-actions @guicaulada
docs: auto update terraform docs
52a0ad5
@guicaulada
guicaulada force-pushed the jvrplmlmn/ssm-token-ttl-expiration-policy branch from 3ca164f to 52a0ad5 Compare September 23, 2026 12:51
@guicaulada guicaulada mentioned this pull request Sep 23, 2026
Merged
@NickAnge NickAnge mentioned this pull request Sep 23, 2026
Merged
@edersonbrilhante
edersonbrilhante self-requested a review September 23, 2026 15:25
edersonbrilhante
edersonbrilhante approved these changes Sep 23, 2026
View reviewed changes

@edersonbrilhante edersonbrilhante left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's just create an issue to add support in v2 as well

@guicaulada
guicaulada merged commit 2dca7f9 into github-aws-runners:main Sep 23, 2026
82 of 83 checks passed
@runners-releaser runners-releaser Bot mentioned this pull request Sep 23, 2026
Merged
guicaulada added a commit that referenced this pull request Sep 24, 2026
@guicaulada
fix(ssm): delete expired runner parameters as pages are listed (#5464)
d815b76 
The SSM housekeeper accumulates every parameter page before deleting
anything. A later listing failure prevents all cleanup, and an omitted
`Parameters` array on the first page silently discards later candidates.

Delete eligible parameters as each page arrives, including pages after
empty responses. Isolate deletion failures per parameter, preserve the
existing age and dry-run protections, and stop starting new work with
ten seconds remaining. A later listing failure leaves earlier deletions
completed.

Cleanup is stateless: every invocation starts a fresh scan of current
parameters. Successfully deleted parameters disappear from subsequent
listings, leaving the remaining work for later scheduled runs.
Pagination tokens exist only within an invocation.

Validation: all 115 storage-provider tests pass, including empty pages,
deletion before later-page failure, fresh scans after partial cleanup,
and continuing past a failed deletion. Runtime TypeScript, ESLint,
Prettier, Lambda bundle build, and diff checks pass. No AWS deployment
or live parameter deletion was performed.

Independent of native SSM expiration-policy work in #5265. Companion
PRs: #5463 and #5465.
Brend-Smits added a commit that referenced this pull request Oct 22, 2026
@runners-releaser @Brend-Smits
chore(main): release 8.0.0 (#5276)
38d0cb5 
🤖 I have created a release *beep* *boop*
---


##
[8.0.0](v7.11.0...v8.0.0)
(2026-10-21)


### ⚠ BREAKING CHANGES

* **terraform:** require Terraform 1.5.6
([#5261](#5261))

### Features

* **compute-providers:** add AWS EC2 provider modules
([#5343](#5343))
([ffa299f](ffa299f))
* **dynamic-labels:** add a capacity-type label to choose spot or
on-demand per job
([#5485](#5485))
([edecd36](edecd36))
* **dynamic-labels:** add allow-list mode to the AWS dynamic labels
policy
([#5411](#5411))
([05984f6](05984f6))
* **examples:** add multi-runner-v2 example
([#5367](#5367))
([f2779cc](f2779cc))
* **github-app:** select app by rate limit budget
([#5391](#5391))
([668d8da](668d8da))
* **github-app:** use ssm manifest for extra apps
([#5282](#5282))
([33c3332](33c3332))
* **multi-runner:** add experimental config translation
([#5284](#5284))
([dd24d33](dd24d33))
* **multi-runner:** consume resolved experimental config
([#5340](#5340))
([ff8bfc2](ff8bfc2))
* **multi-runner:** move v1 state into v2 modules
([#5415](#5415))
([e8d7956](e8d7956))
* **orchestration-providers:** add webhook provider modules
([#5344](#5344))
([c05c9b3](c05c9b3))
* **runner-config:** add provider-aware runner configuration
([#5345](#5345))
([42b1261](42b1261))
* **runners:** add network_interfaces for advanced EC2 launch template
configuration
([#5427](#5427))
([1e72c14](1e72c14))
* **scale-down:** opt-in idle confirmation window before terminating
not-busy runners
([#5397](#5397))
([595f3e5](595f3e5))
* **scale-set:** add scale set orchestration in Terraform
([#5299](#5299))
([937fe15](937fe15))
* **storage:** add runner config consumer
([#5358](#5358))
([138309c](138309c))
* support native SSM expiration policies for runner token parameters
([#5265](#5265))
([2dca7f9](2dca7f9))
* **terraform:** add SSM storage provider boundary
([#5454](#5454))
([d04c5da](d04c5da))
* **terraform:** require Terraform 1.5.6
([#5261](#5261))
([f335b32](f335b32))


### Bug Fixes

* allow null amd_sev_snp/nested_virtualization in cpu_options vali…
([#5430](#5430))
([0b5ec3d](0b5ec3d))
* **control-plane:** clarify partial runner creation test name
([#5339](#5339))
([b1dd396](b1dd396))
* **deps:** bump actions/deploy-pages from 5.0.0 to 5.0.1 in the github
group
([#5418](#5418))
([1a691ea](1a691ea))
* **deps:** bump github/codeql-action/upload-sarif from 4.36.3 to 4.37.8
([#5317](#5317))
([fdb4385](fdb4385))
* **deps:** bump
google/osv-scanner-action/.github/workflows/osv-scanner-reusable-pr.yml
from 2.5.0 to 2.5.1
([#5331](#5331))
([1523a66](1523a66))
* **deps:** bump ossf/scorecard-action from 2.4.3 to 2.4.4
([#5319](#5319))
([d798070](d798070))
* **deps:** bump step-security/harden-runner from 2.19.4 to 2.21.0
([#5318](#5318))
([c81927a](c81927a))
* **deps:** bump step-security/harden-runner from 2.19.4 to 2.21.1
([#5386](#5386))
([48b0c42](48b0c42))
* **deps:** bump step-security/harden-runner from 2.21.1 to 2.22.1
([#5532](#5532))
([5ccf8b3](5ccf8b3))
* **deps:** bump terraform-linters/setup-tflint from 6.3.0 to 6.3.1
([#5419](#5419))
([5f0ecef](5f0ecef))
* **deps:** bump terraform-linters/setup-tflint from 6.3.1 to 6.3.2
([#5515](#5515))
([4c4191f](4c4191f))
* **deps:** bump the github group with 3 updates
([#5330](#5330))
([ab6edf5](ab6edf5))
* **deps:** bump the github group with 3 updates
([#5446](#5446))
([7bb9280](7bb9280))
* **deps:** bump the github group with 3 updates
([#5504](#5504))
([aa08de3](aa08de3))
* **deps:** bump the github group with 5 updates
([#5531](#5531))
([c5b9a47](c5b9a47))
* **deps:** bump zizmorcore/zizmor-action from 0.6.2 to 0.6.3
([#5387](#5387))
([62aef5d](62aef5d))
* **deps:** bump zizmorcore/zizmor-action from 0.6.3 to 0.6.4
([#5447](#5447))
([aa24dc0](aa24dc0))
* **ec2:** avoid launch template tag conflict for on-demand
capacity-type overrides
([#5522](#5522))
([dc9e820](dc9e820))
* **ec2:** improve failure observability
([#5335](#5335))
([91751ef](91751ef))
* fail over rate-limited GitHub App requests
([#5523](#5523))
([7453956](7453956))
* fix typo in variable
([#5462](#5462))
([0786125](0786125))
* **github-runner:** record the rate-limit metric even when the call
fails
([#5426](#5426))
([5026cd9](5026cd9))
* **lambda:** bump @babel/core from 7.23.9 to 7.29.7 in /lambdas
([#5181](#5181))
([3729cf7](3729cf7))
* **lambda:** bump @humanfs/node from 0.16.6 to 0.16.8 in /lambdas
([#5336](#5336))
([f7dd7cb](f7dd7cb))
* **lambda:** bump axios from 1.18.0 to 1.18.1 in /lambdas
([#5223](#5223))
([21d27f1](21d27f1))
* **lambda:** bump axios from 1.18.1 to 1.20.0 in /lambdas
([#5502](#5502))
([e2f9e5e](e2f9e5e))
* **lambda:** bump browserslist from 4.22.3 to 4.28.8 in /lambdas
([#5329](#5329))
([27e7d53](27e7d53))
* **lambda:** bump fast-uri from 3.1.2 to 3.1.5 in /lambdas
([#5242](#5242))
([9c3e28c](9c3e28c))
* **lambda:** bump fast-uri from 3.1.5 to 3.1.6 in /lambdas
([#5337](#5337))
([07bbc87](07bbc87))
* **lambda:** bump fast-uri from 3.1.6 to 3.1.8 in /lambdas
([#5488](#5488))
([3062702](3062702))
* **lambda:** bump follow-redirects from 1.16.0 to 1.16.1 in /lambdas
([#5540](#5540))
([a7b6dea](a7b6dea))
* **lambda:** bump js-yaml from 3.15.0 to 3.15.1 in /lambdas
([#5264](#5264))
([a3116bd](a3116bd))
* **lambda:** bump js-yaml from 3.15.1 to 3.15.2 in /lambdas
([#5398](#5398))
([3db123f](3db123f))
* **lambda:** bump proxy-addr from 2.0.7 to 2.0.8 in /lambdas
([#5510](#5510))
([faf57cd](faf57cd))
* **lambda:** bump source-map-js from 1.2.1 to 1.2.2 in /lambdas
([#5511](#5511))
([e058e05](e058e05))
* **lambda:** bump undici from 6.28.0 to 6.28.1 in /lambdas
([#5489](#5489))
([3ff5877](3ff5877))
* **lambda:** create log groups before functions
([#5324](#5324))
([2f844ab](2f844ab)),
closes
[#5323](#5323)
* **migration-test:** preserve unreserved job retry concurrency
([#5516](#5516))
([feba645](feba645))
* **multi-runner:** gate v2 routing with experimental feature
([#5404](#5404))
([b448845](b448845))
* **multi-runner:** make v2 inputs independent
([#5368](#5368))
([45d3d0e](45d3d0e))
* **multi-runner:** route v1 and v2 runners separately
([#5346](#5346))
([cfac174](cfac174))
* **pages:** configure GitHub Pages deployment environment
([#5334](#5334))
([f081004](f081004))
* **pool-lambda:** grant permission to terminate failed EC2 runners
([#5538](#5538))
([0043890](0043890))
* re-indent commands in CONTRIBUTING.md guide
([#5305](#5305))
([547a98b](547a98b))
* retain arm64 build coverage in lambda workflow
([#5535](#5535))
([80078d7](80078d7))
* **runners:** guard SEGMENT against set -u in start-runner.sh
([#5233](#5233))
([182b37d](182b37d))
* **runners:** support instance requirements in EC2 Fleet overrides
([#5314](#5314))
([4f80654](4f80654))
* **scale-set:** implement review followups
([#5477](#5477))
([445269c](445269c))
* **ssm:** batch housekeeper parameter deletions
([#5469](#5469))
([28a5135](28a5135))
* **ssm:** delete expired runner parameters as pages are listed
([#5464](#5464))
([d815b76](d815b76))
* **ssm:** restore incremental housekeeper cleanup
([#5508](#5508))
([22b9310](22b9310))
* **storage:** add SSM adapter diagnostics
([#5385](#5385))
([d08be31](d08be31))
* **webhook:** cache only a successful config load
([#5519](#5519))
([a2fea96](a2fea96))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

---------

Signed-off-by: Brend Smits <brend.smits@philips.com>
Co-authored-by: runners-releaser[bot] <194412594+runners-releaser[bot]@users.noreply.github.com>
Co-authored-by: Brend Smits <brend.smits@philips.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@edersonbrilhante edersonbrilhante edersonbrilhante approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@jvrplmlmn @edersonbrilhante @guicaulada