3 Ways to Reduce the Risk from Misused AWS IAM User Access Keys

Used incorrectly, AWS IAM user access keys can pose high risk; the good news is that great alternatives, explored here, exist.

When misused or otherwise not used securely, AWS IAM user access keys have long been one of the most effective, lowest hanging fruits for attackers seeking a foothold in a cloud environment. Such credentials are one of the main culprits in some highly damaging breaches reported in the past few years. Christophe Tafani-Dereeper’s review of major 2021 cloud breaches reinforces the fact that access keys remain a major initial attack vector.

The irony about this common configuration is that a secure, accessible and easy-to-implement alternative to using access keys usually exists. This fact is truer than ever with AWS’s recent release of IAM Roles Anywhere. Given the significant risk, why is misuse of access keys so prevalent and how can it be avoided? Do the benefits of IAM user access keys outweigh the misuse risk?

In this post, we review the risk of using access keys, provide key examples from past breaches and offer practical advice on how to reduce the misuse that leads to risk.

To be clear, access keys themselves are not a security issue or a fault on Amazon’s part. Security issues arise from their misuse, and the responsibility for such misuse lies solely with the customer as part of the shared responsibility model. So use them with caution!

What are IAM access keys?

Access keys are a type of credential that can be generated to authenticate an IAM user. They are a combination of an access key ID (a string that starts with AKIA) and an access key secret, which is presented only when the access key is created. At any time, an IAM user can have up to two access keys, which may be deactivated (that is, they still exist, and can be reactivated) and eventually deleted.

Unless prevented by a proper control, examples of which we will present later, an active access key allows absolutely anyone to authenticate as the IAM user and access any permission granted to it. Access keys are essentially permanent credentials; if leaked (e.g. by being placed in a public resource such as a code repository or public S3 bucket), they create risk of exposure to the AWS environment as they can be exploited using all the permissions of the user associated with the keys.

Unfortunately, as we show in the following section, access key leaks happen often and can cause substantial damage.

3 notable AWS breaches

Rami McCarthy's Github archive lists AWS breaches and is a great resource for staying abreast of and getting details on notable breaches of AWS environments – in short, for learning from the errors of others and avoiding a similar fate. If you pay close attention, you will see that many of the breaches originated from an access key falling into the hands of the wrong person.

Let’s look at a few examples that illustrate the different ways in which these credentials can leak. Since the details are at the Github link above, we will discuss each example only briefly.

Nature’s Basket (2020)

The analysis of this breach in MGaurav Chib’s Medium blog reported that the access keys of the root user (no less!) were found on a file hosted on a public S3 bucket. This kind of exposure doesn’t just allow initial access to the AWS account; it allows full access and complete control over the account. This is a good place to point out that, for this very reason, you should never generate or maintain access keys for root users and, if you do, you should follow AWS’s recommendation to lock access keys away properly.

Hosting credentials on exposed resources such as public S3 buckets is quite a common channel for leakage.

Credentials leak source: Exposed resources

Animal Jam (2020)

The gaming company Animal Jam had a breach which, as reported on The Register, resulted in exposing 46 million users' records due to access keys shared on a compromised Slack channel.

Slack channels, for the same reason as public S3 buckets, are a location where you should never store or communicate about sensitive information in plaintext. The Animal Jam breach is another example of how loose access key credentials can get. Even without a breach, the chance of credentials ending up in the wrong hands when on a Slack channel is significant.

Credentials leak source: Slack channels

Datadog (2016)

As Datadog reported, in 2016 an attacker used an access key and SSH private key in use by their automated provisioning and release systems – part of their CI/CD pipeline – to gain unauthorized access to their resources on AWS. Specifically, the malicious actor was able to obtain credentials for the Datadog service, service metadata and credentials shared with Datadog for third-party integrations.

This breach highlights two interesting points; first, if not maintained correctly, provisioning platforms can be a source from which credentials may leak; second, data leaks from a third-party vendor to which you supply credentials for integrations may also create additional risk or a threat vector that you should be concerned about. Keep this latter point in mind as we discuss third parties in the next section.

Credentials leak source: Provisioning platforms and third parties

What are the benefits of using access keys?

To understand how to mitigate the risk caused by access keys, we first need to understand their utility: why they are even generated and how they are used. The best way to do so is by looking at certain common use cases – by no means best practices -- and where they harbor potential risk. Further below, we present more secure alternatives to these use cases or preferred practices when applying them.

Programmatic access to AWS

Probably the most common utility for access keys is that of developers or DevOps professionals seeking to interact with an AWS account by using the AWS command line interface (CLI) or via another programmatic access tool (e.g. Terraform).

As per AWS documentation, configuring the CLI saves the credentials to specific local locations. As described in our analysis of the TeamTNT malware, this reality was not overlooked by cyber criminals, who created malware to target these exact locations and harvest the credentials stored there. Note that for the risk to be realized, the malware would need to execute on the victim’s system so it can harvest the AWS credentials -- this would require a failure of the endpoint security controls (which are not administered or managed by AWS, but rather are the purview of the AWS customer).

Similarly, a user may store these credentials in environment variables, which are also known to attackers. We’ve seen reports in the wild of the weaponization of vulnerabilities, such as the Log4j vulnerability and a vulnerability in the Python ctx library, for the purpose of harvesting environment variables – with the assumed intent of exfiltrating the credentials stored there.

Workloads

Another use case for such credentials is that of allowing workloads to access resources in AWS. You may think of this as a private case of programmatic access yet we bring it to your attention because, since humans are not involved in using the access keys, mitigation will be different.

Third parties

Another access-keys use case that, fortunately for an organization’s cloud security, is becoming less common is third-party vendors requiring access keys so that an IAM user can access their customers’ AWS accounts.

From a security perspective this use case is the worst of all because, by definition, the access keys are placed where you have far fewer technical and legal controls to make sure they are not misused.

Three ways to reduce risk by retiring or curbing use of access keys

The current reality is that IAM users have become almost obsolete for most of their use cases. IAM and security practitioners have at their disposal newer, more secure alternatives that are the recommended best practice. As mentioned, the recent release of AWS IAM Roles Anywhere makes this even truer. Organizations anywhere, and especially cloud native companies, should consider if they need IAM users at all and IAM user access keys in particular.

This section reviews three simple ways to reassess and, where possible, purge the use of access keys in your environment. We will also discuss what to do with legacy instances or in other cases in which you find it too difficult to go “cold turkey” on access key removal.

1. Use replacements for access keys

The most secure credentials are those you never generate at all. For this reason, it’s far better to replace any use of IAM user access keys with a more secure method (and, of course, to apply the new method going forward).

Let’s review the most common and straightforward use cases of how to do so.

Generating temporary credentials for human programmatic access

As a general note, the best practice for managing users in AWS – rather than use of IAM users – is to use an identity provider (IdP) to centrally manage your users and provide them with access to different AWS accounts. When the IdP is external you may create a “federation” (or, in plain language, trust) between the IdP and AWS. You can then use an authentication protocol such as SAML to authenticate the user to the IdP and receive a token from it for use with AWS in gaining access. In the case of SAML, this token is called a “SAML Assertion.”

AWS provides a service called AWS IAM Identity Center that allows one-time sign-in to an identity provider to serve in accessing multiple services. AWS IAM Identity Center centralizes management of single sign-on access, making it easier to manage identities in external IdPs, Microsoft Active Directory or identities created in AWS IAM Identity Center itself. You can then, as an alternative mechanism for storing permanent credentials, configure profiles that use AWS IAM Identity Center to authenticate to AWS’s CLI. AWS caches the temporary credentials locally (for example, on a Mac, in the ~/.aws/sso/cache folder), and you can make CLI calls by specifying that profile using an appropriate flag.

If you don’t use AWS IAM Identity Center, other tools let you use your IdP to authenticate to the AWS CLI, such as saml2aws. In addition, the new IAM Roles Anywhere feature allows the use of certificates issued by a certificate authority (CA) for creating temporary credentials. This approach is another great means, in lieu of IAM users, for users to obtain permissions to AWS resources. Roles Anywhere is mostly meant to address the need of external workloads to access AWS resources but if you aren’t using AWS IAM Identity Center you can indeed use it as a replacement.

Using IAM roles for workloads

The secure way to provide workloads with access to AWS resources is by using IAM roles. For this purpose, we should look differently at AWS workloads and non-AWS workloads.

When you need to provide access to AWS workloads (for example, EC2s or Lambda functions), there really is no reason to load them manually with access to resources using static credentials such as IAM user access keys. It is extremely simple to attach an IAM role to the workload and simply provide the permissions to that IAM role. Lambda functions, for example, come with a role already attached. This is also specified as an AWS best practice.

Security-wise, this approach is really a no-brainer as it involves no credentials to manage and nothing that may leak – and IAM roles are much easier to use. This mechanism is more secure on orders of magnitude than manually managing IAM user access keys for such access.

Non-AWS workloads may also use IAM Roles to generate temporary credentials by utilizing workload identity federation; an example for an implementation of this approach is spiffe-aws-assume-role.

In addition, for on-prem workloads, the new Roles Anywhere is an excellent replacement for user access keys. As mentioned, the feature allows the creation of trust between a CA and an AWS account, after which private certificates issued by the CA are trusted by the AWS account. These certificates may be used for signing requests for temporary credentials from STS assuming specific IAM roles. Certificates are a more secure alternative as their maintenance and use are far less error-prone than permanent plaintext access keys and, by using Roles Anywhere, you use short-lived credentials for IAM roles instead of permanent credentials. While still very new, this feature seems like a very promising mechanism for allowing on-prem workloads to access AWS resources.

Using IAM roles for third party access

Finally, as mentioned, there really is no reason for a third party to demand IAM user access keys as a method of obtaining cross-account access. This approach violates the common AWS best practice of using an IAM role with external ID. It also creates an unsettling situation in which you are willingly sharing permanent credentials and allowing access to your environment to an entity outside your organization.

If you are compelled, such as for business reasons, to use a third party vendor that demands this kind of configuration, you should seriously consider the relevant compensating controls we describe in the following section and apply them as possible.

2. Prevent unwarranted IAM users and access key creation

As a preventive approach, you can limit the creation of access keys and even the IAM users themselves in your environment. By using a service control policy (SCP) that denies access to the iam:CreateAccessKey and iam:CreateUser permissions, you can prevent all but specific identities from generating access keys or even creating new IAM users. Such a policy won’t address risk from existing IAM users and their access keys – but it’s a powerful way to prevent unwarranted IAM users and access keys from ever being created.

An SCP that prevents creation of access keys from all principals within an account other than a specific admin role (for the extreme case in which it is needed) would look something like this:

{   
  “Version”: “2012-10-17”,
  “Statement”: [
    {
      “Sid”: “DenyCreateAccessKeyExceptForAdminRole”,
      “Effect”: “Deny”,
      “Action”: [
          “iam:CreateAccessKey”,
          “iam:CreateUser”
      ],
      “Resource”: “*”,
      “Condition”: {
        “StringNotLike”: {
          “aws:PrincipalARN”:”arn:aws:iam::*:role/AdminRole”
        }
      }
    }
  ]
}

3. Monitor Your environment

Once you’ve strictly limited use of access keys in your environment, you should track your current inventory and any creation of new ones to make sure they aren’t somehow still abused.

In addition, if you indeed still have active access keys in your environment, simple monitoring controls such as those described below are available to maintain your environment’s posture.

More advanced tools and methods exist, but the point is that you can monitor your environment effectively and with ease by applying very simple tools.

Trigger notifications upon event creation

AWS allows you to easily configure EventBridge to trigger a notification, such as to an SNS topic, when an access key is created.

In just a few minutes we were able to set up an EventBridge rule that sends an email to all subscribers of an SNS topic that alerts on the creation of a new access key (following the CloudTrail event CreateAccessKey).

Figure 1 - Screenshots showing creation of an EventBridge rule to trigger an SNS topic that sends an email upon creation of an access key

This powerful setup helps you keep tabs on access keys created so you can closely review them in a timely manner and detect if they were created in a way that doesn’t adhere to your security policy.

Auditing credentials reports

An effective tool to get you started in reducing access key occurrences and that you can also use routinely to track their use is AWS credential reports, which you can easily generate from the IAM service. These reports create a sorted list of all the IAM users in your environment that have active access keys, as well as when the access keys were generated and even when they were most recently used.

Figures 2 and 3 demonstrate their use:

Figure 2 - Creation blade and button on the AWS console for a credential report

Figure 3 - Credential report showing information about IAM users and their access keys

Use AWS config to watch for stale access keys

In some situations you will need active access keys in your environment. One method crucial to securing their use is proper rotation at a pre-determined frequency. AWS config has a simple out-of-the-box rule that finds IAM users with access keys older than a specified number of days (the default is 90).

When you meet up with reality

As we all know, reality doesn’t always play by our rules. You may have situations that compel you to maintain the use of access keys – even if not optimal security-wise.

For example, some third party vendors still require use of access keys rather than an IAM role as their best practice, and your organization may be using them for business reasons, compelling you to oblige with the vendor’s preference.

Or your organization may be using access keys for certain workloads, so replacing them with a secure alternative may take time for operational reasons, compelling you to work with this reality in the meantime.

For these situations, we describe below certain controls you can use to reduce the associated risk.

Secure storage

The first action is to ensure that the access key is stored securely. We’ll look at some such use cases – easily available and common solutions that are a dramatic improvement over storing an access key in environment variables or in plaintext form. (Depending on your requirements you may find more advanced solutions, but such solutions are beyond the scope of this article.)

Keep in mind that policy scrutiny and rotation – two approaches to reducing risk described below – are also applicable to scenarios in which you are compelled to use an access key.

If you’re running a workload on prem, cannot utilize Roles Anywhere and cannot provide it with another kind of identity that can be federated to an AWS account, you have a few solutions for secure storage, probably the best known of which is Hashicorp’s Vault.

While AWS IAM Identity Center is the recommended approach, if your organization doesn’t use an IdP and you can’t get around for some reason to using AWS IAM Identity Center, you can still use a lightweight open source solution such as aws-vault for safer keys storage. This tool allows you to store access keys on your OS’s keystore (which is much more secure than local plaintext or environment variables). When triggered to perform API calls, aws-vault will generate and use temporary session tokens for subsequent calls during the session. (You can also use a flag to instruct it to simply make each call using the permanent access key.)

Policy scrutiny

AWS has robust and flexible mechanisms for controlling the permissions of identities. You should be familiar with them and apply them wherever applicable but especially for identities prone to security posture issues. By carrying out proper policy scrutiny you can reduce the risk from a breach of the identity (though, of course, not eliminate the risk completely). Let’s briefly explore a few preventive things you can do.

Maintain the principle of least privilege

The principle of least privilege (POLP) involves granting each identity only the permissions it needs to perform a given business function. Applying least privilege is a complicated task not covered here end-to-end, but do take note of some least-privilege best practices:

• Always specify IAM user permissions at the action level, and on specific resources or resources of a certain pattern, and avoid using “*” as much as possible
• Be cautious when using AWS managed policies and, preferably, avoid using them in a permanent manner as they tend to be extremely over-permissive
• Review the Access Analyzer’s policy validator’s security recommendations for the IAM policies attached to the user

Use conditional statements to limit access

Finally, to further limit the extent to which the IAM user can operate, you can apply an IAM policy (either inline or customer managed), which denies the IAM user access to resources and/or actions that are sensitive. You can also use a permissions boundary to achieve this.

On top of that, you can apply limitations that set conditions regarding when and how an IAM user can access resources. AWS has a wide variety of condition keys for use in such situations. Let’s look, for example, at the following IAM policy:

{
  "Version":"2012-10-17",
  "Id":"ExampleIpConditionStatement",
  "Statement":[
     {
        "Sid":"networkdataperimeter",
        "Effect":"Deny",
        "Action": "*",
        "Resource":"*",
        "Condition":{
           "NotIpAddress":{
              "aws:SourceIp":[
                 "<WORKLOAD_IP>"
              ]
           },
           "Bool":{
              "aws:ViaAWSService":"false"
           }
        }
     }
  ]
}

(If you’re wondering about the aws:ViaAWSService key that appears here, we invite you to read our blog post on AWS condition keys.)

Applying this IAM policy will limit the IAM user (and thus the access keys) to be allowed to send calls from a specific IP only. If you generate this access key for a workload, you can specify the IP from which you know the workflow is supposed to operate. Doing so should make it much harder for malicious actors to utilize the access keys even when they leak since they would have to make the calls from the network from which the designated workflow makes calls.

Rotation

Rotation in the case of an IAM user access key means to replace the access key (by disabling and later deleting it) and create and use a new one instead.

The security benefit of this process is clear: even if it leaks, the access key will only be effective and usable for breaching the environment up until the rotation occurs. The recommended frequency is 90 days but the more frequently you perform rotation, the lower your exposure will be.

Note that this process is error prone and could very well result in leakage of the access keys or disruption of the service using them. The advantage of IAM roles is that using them may be seen as a similar process (creating credentials that are short lived and very quickly become moot) but you get the procedure out-of-the-box with little risk of failure. This makes IAM roles the superior option to implement unless you are compelled not to.

When the rotation applies to a workload, consider investing the time to automate the rotation process. If you are already using an access key to grant a workload access to the environment, it can (given the proper permissions) perform its own key rotation of creating and storing the new access key in the secure storage location, and deactivating and deleting the former access key.

Automating the process of rotation allows you to:

• Perform the rotation more frequently
• Be confident the rotation has taken place since automation is more reliable than a person charged with doing it manually
• Minimize the exposure of humans to the credentials in plaintext (as long as you make sure the workload doesn’t do anything careless such as logging the credentials)
• Guard with care the permissions for generating and deleting access keys. When automating the process, be sure to pay great attention to securing the workload.

Final words

IAM users are becoming obsolete. In most cases, AWS accounts don’t need them at all. If you run into cases in which business demands supersede this security choice, and you are compelled to use access keys, keep in mind that you can take steps to secure them and reduce the risk arising from their use. However your top priority should be to eliminate use of IAM user access keys, and perhaps IAM users in general - as much as possible.

Remember: The most secure access keys are those never created at all!

As reviewed here, for most use cases in which someone wants to use an IAM user and access key, you have reliable and robust alternatives that are far superior from a security standpoint. These alternatives leverage IAM roles to create temporary credentials instead of long-lived ones like user access keys. This has become especially true with the release of AWS IAM Roles Anywhere. Familiarize yourself with these alternatives and have them made available to the developers in your organization.

If your AWS identity account is like the many accounts we run into, access keys are -- for legacy reasons or lack of best practices -- still widely in use. It’s essential that you have a plan for “responsibly” retiring existing keys. For this, you will need to be able to query CloudTrail logs to understand how and when access keys have been used and where calls were made from so that you can detect workloads that may store such keys locally.

Understand that pursuing “access key hygiene” is a project. On top of the insights offered in this blog post, to have a fruitful conversation with your development teams, you will need to have impeccable control over the IAM user activity in your environment -- which is easier said than done. This depth of visibility and control is crucial for the process to be effective -- and for you to not only secure the trust you currently have with developers in your organization but to enhance it.