ExpiredTokenException error on credential file refresh on on-premise (/root/.aws/credentials)

[byoungsoo]

Env

• On-premise
• Use AWS Systems Manager(SSM) Hybrid Activation
• SSM agent updates /root/.aws/credentials every 30 minutes for new credentials

Configuration

/etc/codedeploy-agent/conf/codedeploy.onpremises.yml

iam_session_arn: arn:aws:sts::<ACCOUNT>:assumed-role/AWSOnpremServerRole/AWSOnpremServerRole_CDP
aws_credentials_file: /root/.aws/credentials
region: ap-northeast-2

Issue

At the first time, when codedeploy-agent start no credential issue, but if the first credential updated by SSM agent expired then codedeploy-agent get error even though credentials file is updated.

2026-02-11T07:01:28 ERROR [codedeploy-agent(5470)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Cannot reach InstanceService: Aws::CodeDeployCommand::Errors::ExpiredTokenException - The security token included in the request is expired
2026-02-11T07:01:28 DEBUG [codedeploy-agent(5470)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Sleeping 7 seconds.
.......
2026-02-11T07:01:38 INFO  [codedeploy-agent(5470)]: [Aws::CodeDeployCommand::Client 400 2.126136 3 retries] poll_host_command(host_identifier:"arn:aws:sts::558846430793:assumed-role/AWSOnpremServerRole/mi-0c9c2ec582198650e") Aws::CodeDeployCommand::Errors::ExpiredTokenException The security token included in the request is expired
2026-02-11T07:01:38 ERROR [codedeploy-agent(5470)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Error polling for host commands: Aws::CodeDeployCommand::Errors::ExpiredTokenException - The security token included in the request is expired - /usr/local/share/gems/gems/aws-sdk-core-3.242.0/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call'

However, if the aws_credentials_file path is not /root/.aws/credentials, credential is updated and no error.

Reason

file_credentials.rb

def refresh
      @credentials = Aws::SharedCredentials.new(path: @path).credentials
      raise "Failed to load credentials from path #{@path}" if @credentials.nil?
      @expiration = Time.new + 1800
    rescue Aws::Errors::NoSuchProfileError
      raise "Failed to load credentials from path #{@path}"
    end

aws-sdk-core/shared_credentials.rb

def initialize(options = {})
      shared_config = Aws.shared_config
      @path = options[:path]
      @path ||= shared_config.credentials_path
      @profile_name = options[:profile_name]
      @profile_name ||= ENV['AWS_PROFILE']
      @profile_name ||= shared_config.profile_name
      if @path && @path == shared_config.credentials_path
        @credentials = shared_config.credentials(profile: @profile_name)
      else
        config = SharedConfig.new(
          credentials_path: @path,
          profile_name: @profile_name
        )
        @credentials = config.credentials(profile: @profile_name)
      end
      @metrics = ['CREDENTIALS_CODE']
    end

I think when the credential path is /root/.aws/credentials, the code uses the cached singleton shared_config object, which prevents credentials from being refreshed even when the file is updated.

Solution

1. Update document
   In this document, It should be explicitly stated that it is recommended to use a path other than /root/.aws/credentials for the credentials-file path.

2. Update code
   Use credentials as usual when not on the default path. However, it is true that the credentials for that path do not generally change, and since the current situation is an exception, it would be better to modify the document as in 1.

else
    config = SharedConfig.new(
      credentials_path: @path,
      profile_name: @profile_name
    )
    @credentials = config.credentials(profile: @profile_name)
  end

---

If there is anything I misunderstood, please let me know.