ssm-housekeeper throttled by DeleteParameter rate limit

[wadherv]

Description

The ssm-housekeeper Lambda deletes stale runner token/JIT-config parameters from SSM Parameter Store one at a time, with a fixed 50ms sleep between calls. AWS enforces a hard, non-adjustable account/region-wide quota of 5 requests/second for DeleteParameter, so once the backlog of stale parameters grows past a few thousand, the housekeeper throttles on nearly every delete and times out before finishing. The backlog never shrinks — it can grow faster than it's cleaned up — until the account hits the 10,000 standard-parameter limit. At that point, the scale-up Lambda's SQS batch handler starts failing every batch with You have reached the maximum number of standard parameters for this AWS account and Region (10000), preventing runners from scaling up until the parameter count is brought back under the limit.

Root Cause

In ssm-housekeeper.ts, cleanSSMTokens() deletes parameters one-by-one via DeleteParameterCommand, sleeping 50ms between each call:

for (const parameter of parameters.Parameters ?? []) {
  if (parameter.LastModifiedDate && new Date(parameter.LastModifiedDate) < minimumDate) {
    logger.info(`Deleting parameter ${parameter.Name} with last modified date ${parameter.LastModifiedDate}`);
    try {
      if (!options.dryRun) {
        // ❌ sleep 50ms to avoid rait limit -- targets ~20 req/sec, but the real
        //    AWS quota for DeleteParameter is a fixed 5 req/sec (non-adjustable,
        //    account+region wide) -- this throttles on almost every call once
        //    there's any real backlog
        await new Promise((resolve) => setTimeout(resolve, 50));
        await client.send(new DeleteParameterCommand({ Name: parameter.Name }));
      }
    } catch (e) {
      // ❌ throttled deletes are logged and skipped, never retried this run
      logger.warn(`Failed to delete parameter ${parameter.Name} with error ${(e as Error).message}`);
    }
  }
}

Confirmed via aws service-quotas list-service-quotas --service-code ssm:

Rate of DeleteParameter requests   | Value: 5.0  | Adjustable: False
Rate of DeleteParameters requests  | Value: 5.0  | Adjustable: False
Standard parameters                | Value: 10000.0 | Adjustable: False

The 50ms sleep is ~4x faster than the real ceiling allows, so DeleteParameter calls throttle (Rate exceeded) almost continuously once there's a nontrivial backlog. Deletes are also issued one name per API call instead of being batch processed, which would give up to 10x the deletion throughput at the same quota cost.

Separately, GetParametersByPathCommand re-lists the entire token path from scratch on every invocation — there's no cursor/checkpoint persisted between runs, so a run that times out partway through makes the next scheduled run start the scan over rather than resume.

This is unrelated to the high-throughput-enabled SSM service setting or the Standard/Advanced parameter tier — neither affects DeleteParameter/DeleteParameters throughput; that quota is fixed regardless of tier or the high-throughput flag. (The AWS error text — Ensure you have the high-throughput setting enabled for higher limits — is generic and misleading for this specific operation.)

Impact

• Runners stop scaling up: once the account hits the 10,000 standard-parameter cap, scale-up's SQS batch handler fails every batch with the parameter-limit error, so queued CI jobs get no runner until the parameter count is brought back under the limit.

How to Trigger

1. Deploy with a runner pool whose churn accumulates SSM token/JIT-config parameters (via scale-up/pool) faster than ~5/sec can clean up — e.g. any pool with enough sustained job throughput that the daily stale-parameter backlog exceeds roughly 1,500 (5 req/sec × 300s timeout ≈ max deletable per run at the default sleep).
2. Observe ssm-housekeeper CloudWatch logs: repeated WARN lines:
   Failed to delete parameter <path> with error Rate exceeded. Ensure you have the high-throughput setting enabled for higher limits...
3. Observe the invocation's REPORT log line ends with Status: timeout, having processed only a subset of the listed parameters.
4. Over consecutive days the backlog doesn't shrink (and can grow if churn outpaces ~5/sec), eventually approaching the account's 10,000 standard-parameter limit and triggering the scale-up batch-drop failure described above.

Environment

• Module version: v7.6.0 (current main as of this report)
• ssm_housekeeper left at defaults: schedule_expression = "rate(1 day)", lambda_timeout = 60, config.minimumDaysOld = 1
• Runner pool with sustained SQS-driven scale-up throughput (ghec-style EventSourceMapping on the queued-builds queue)

Suggested Fixes

Fix 1: Batch process the deletes instead of one-at-a-time

Use DeleteParametersCommand to delete up to 10 names per call under the same 5 req/sec quota — up to 10x the deletion throughput of the current code:

import { DeleteParametersCommand, GetParametersByPathCommand, SSMClient } from '@aws-sdk/client-ssm';

const BATCH_SIZE = 10; // max allowed by DeleteParameters
const DELAY_MS = 220; // ~5 req/sec quota for DeleteParameters, with margin

const staleNames = (parameters.Parameters ?? [])
  .filter((p) => p.LastModifiedDate && new Date(p.LastModifiedDate) < minimumDate)
  .map((p) => p.Name as string);

for (let i = 0; i < staleNames.length; i += BATCH_SIZE) {
  const batch = staleNames.slice(i, i + BATCH_SIZE);
  logger.info(`Deleting batch of ${batch.length} parameters`, { batch });
  try {
    if (!options.dryRun) {
      await new Promise((resolve) => setTimeout(resolve, DELAY_MS));
      const result = await client.send(new DeleteParametersCommand({ Names: batch }));
      if (result.InvalidParameters?.length) {
        logger.warn(`Some parameters could not be deleted`, { invalid: result.InvalidParameters });
      }
    }
  } catch (e) {
    logger.warn(`Failed to delete batch with error ${(e as Error).message}`, { batch });
  }
}

Fix 2: Correct the pacing and add throttle-aware retry

Even with batching, the sleep must target the real 5 req/sec ceiling (~200–220ms between calls, not 50ms). Additionally, catch ThrottlingException specifically and retry with backoff instead of logging a warning and moving on — currently a throttled batch is dropped for the rest of that run, not retried, so the backlog can persist indefinitely under sustained load.

Our Workaround

We temporarily raised lambda_timeout on ssm-housekeeper (60s → 300s) and updated the parameter default tier from Standard tier to Advanced tier, increasing the max parameter count from 10,000 to 100,000 parameters.

[jvrplmlmn]

We recently faced a similar scenario in which we hit the quota of SSM parameters and it was a combination of a surge of scale up events and deletion not being able to keep up.

In our specific scenario, we ended up tuning the frequency in which we ran the ssm-housekeeper function but that was not sufficient for it to catch up.

An alternative here, particularly if you are already using SSM Advanced Parameters, is to use the built-in TTL mechanism, which could even remove the need of running the ssm-houskeeper lambda (of course this might vary based on your setup).

I have implemented the idea above in this PR:

• feat: support native SSM expiration policies for runner token parameters #5265

[wadherv]

@jvrplmlmn using built-in TTL is a good addition.

[wadherv]

I have started working on a new enhancement to use DynamoDB for storing runner configuration instead of SSM Parameter Store. We are already approaching the hard limit of 10 writes per second in SSM Parameter Store(docs), and usage continues to grow rapidly. Moving to DynamoDB will help us scale more effectively and remove this bottleneck.

Ref: #5266