Wayward Sheriffs and Confused Deputies: Risks in GCP Third Party Access

Most GCP third-party vendors ask for permanent service account keys for access -- increasing credential leakage risk. Used correctly, short-lived credentials offer a secure alternative.

Third-party access in the cloud

Third-party access features extensively in almost every organization in the cloud. Companies use third-party services in their cloud environments, and those third-party services have to access the customer’s environment to access the APIs of cloud service providers (CSPs) to do their job. However, a third party represents an element in an organization’s security that is inherently outside its control. Secure access mechanisms are therefore key in limiting risk exposure through third parties.

The key points we’ll review in this post:

• Third-party access is most commonly performed in Google Cloud Platform (GCP) by using service account keys.
• This exposes organizations to credential leakage risk.
• The other possible method, service account impersonation, is vulnerable to confused deputy attacks.
• We propose – and invite you to join in – that Google adopt a suggested new alternative method for third-party access that protects from both these risks.

Basic terminology in Google Cloud Platform

What is a service account?

A service account is a special type of Google account used in GCP to represent a non-human user that needs to access Google APIs. When it wants to call Google APIs, your application assumes the identity of the service account to do so.

How third parties act in customer environments

Third-party vendors represent applications, rather than humans, calling Google APIs in your environment. Therefore, the appropriate way for them to act in your environment is through service accounts, in two primary ways:

• Service Account Keys: A key shared with the third party that allows it to act as the service account.
• Service Account Impersonation: Allowing a specific service account or user already controlled by the third party to impersonate, or act as, the service account in your account by creating short-lived credentials for it.

GCP third-party access in the wild

Each vendor determines which method it uses to access its customers’ environments. Almost every vendor we’ve seen accesses GCP customer environments in the same way: by requesting that the customer create a service account with certain GCP permissions on the project and then requesting the permanent service account key.

GCP’s own documentation acknowledges the dangers of using permanent service account keys and advises in large bold letters: Use service account keys only if there is no viable alternative.

In the GCP console’s page where such keys are created, similar advice is dispensed:

Credential leakage

The reason all these resources suggest limiting usage of access keys is simple. Every service account key you release into the wild exposes your environment to additional credential leakage risk. Not only is this risk often very significant, it is entirely out of your control. In short, a compromised service account key is more of a “wayward” sheriff than a reliable protector.

GCP’s own page on Best practices for managing service account keys explains the unique risks well. It states: “To bad actors, service account keys can be even more valuable than a leaked password.”

Bad actors might look for service account keys in a variety of places, including:

• Source code repositories of open-source projects
• By leveraging SSRF attacks
• Public Cloud Storage buckets
• Public data dumps of breached services
• Unprotected metadata and configuration data

In addition to public locations, bad actors might look for service account keys in private locations they've compromised. Examples include:

• Email inboxes
• File shares
• Backup storage
• Temporary file system directories
• Personal machines of the employees of the third-party vendor

The Log4Shell vulnerability also illustrated this painful point, with hackers looking for access keys to cloud environments in compromised systems.

When permanent credentials are distributed to third-party vendors, credential leakage risks become both likelier and more severe: Likelier because of the myriad ways credential leakage can happen, and more severe because credential leakage gives the attacker full legitimate access to a service account in your cloud environment that they can utilize from their own machine to act as they wish.

Unfortunately, permanent credentials are the most common method used today to grant access to third-party vendors, primarily because Google has not yet offered or supported a viable alternative. As an aside, using permanent credentials to access customer environments in AWS and Azure is considered unacceptable practice in terms of security.

As a result, the blast radius of any potential breach, partial or full, of a third-party vendor in GCP environments is immense.

The other way: service account impersonation

What is service account impersonation

GCP does offer an alternative: Service account impersonation, the method used to allow principals and resources to impersonate, or act as, a service account. The key difference compared to permanent keys is that service account impersonation creates short-lived credentials, so most leakage vectors cannot occur, and those which can occur have limited impact (due to the short lifespan of the token).

The risk: confused deputy attacks

We’ve written previously about confused deputy risks in AWS. Simply put, a confused deputy attack is when a malicious entity takes advantage of the privileges granted to a third-party identity to gain information or perform actions in your cloud environment. These can happen in GCP when service account impersonation is used. For example, if all that’s needed to perform such actions is the service account email address, a malicious actor can fairly easily set up a user profile with the third-party vendor and have it perform actions and/or retrieve sensitive information from your GCP environment.

The attack would go as follows:

1. Before the attack, your company registers with ExampleVendor. To enable ExampleVendor to act in your production environment, you create the service account example_vendor@client_prod.iam.gserviceaccount.com, allowing example_vendor_prod@example_vendor_prod.iam.gserviceaccount.com to impersonate it.
2. An attacker sets up a new profile with the ExampleVendor and inputs example_vendor@client_prod.iam.gserviceaccount.com as the service account that ExampleVendor can impersonate. The attacker can do this by guessing the service account or examining publicly available information, such as onboarding docs for the vendor, public assets of the client, and the like).
3. The attacker can now log into their profile with the ExampleVendor and, posing as the victim, use the third-party service to access any available information and perform whatever actions are possible.

To establish unique trust that can’t be exploited by a confused deputy attack, client-vendor trust must include a non-public component.

A new way forward: Safely using short-lived credentials in GCP

So if service account keys expose us to significant credential leakage risk, and service account impersonation exposes us to confused deputy attacks, how do we get the best of both worlds and avoid both risks altogether?

Unlike AWS, GCP does not provide a built-in way to avoid the confused deputy problem when using short-lived credentials for third-party access, so we have to rely on manual solutions.

Method #1: Randomly generated client-side service account names

1. The customer registers with ExampleVendor.
2. During the onboarding process, ExampleVendor asks the company to create a service account for ExampleVendor to impersonate, with a specific, unique service account name, e.g.: example_vendor-ac70d702@client_prod.iam.gserviceaccount.com. (We can’t use a full UUID as GCP limits service account names to 30 characters before the “@”.)
3. The vendor impersonates this service account.

Then, when the attack is attempted:

1. The attacker registers with ExampleVendor.
2. ExampleVendor asks the attacker to create the service account: example_vendor-b4df00d5@client_prod.iam.gserviceaccount.com.
3. Since the attacker cannot create this service account, they cannot access the system.

The downside of this method is that you force clients to use somewhat unfriendly service account names.

Method #2: One service account per customer

This method reflects the approach described here by Andrew Ayer of SSLMate:

1. The customer registers with ExampleVendor.
2. ExampleVendor creates a new service account: example_vendor_prod_f18d3fcf@example_vendor_prod.iam.gserviceaccount.com .
3. ExampleVendor asks the customer to create a service account that the newly generated service account is allowed to impersonate in ExampleVendor’s production environment.

Behind the scenes, to avoid handling large numbers of service account keys, all of ExampleVendor’s per-customer service accounts trust a main service account that is allowed to impersonate them.

The downside of this method is that it increases vendor-side complexity because the vendor needs to create (and delete) service accounts for every client. Since the default service-account limit per project is 100 (though you can ask to expand the quota), the vendor would even need to create new projects to accommodate more customers.

Method #3: Label verification

1. When registering with ExampleVendor, ExampleVendor asks the company during the onboarding process to:
   1. Create a service account for ExampleVendor to impersonate.
   2. Attach a label: ExternalId to the project the service account is in, with the value of the label randomly generated by the vendor: UUID_1. This ensures that the label is unique to each customer.
2. When impersonating the service account, the vendor must verify that the value of the label is the same one generated during the onboarding process.

Then, when the attack is attempted:

1. The attacker tries to register to the service, giving example_vendor@client_prod.iam.gserviceaccount.com as the service account to impersonate.
2. The attacker is given a different ExternalId value: UUID_2
3. When the attacker tries to use the service, the ExternalId value is checked and, since it is not UUID_2 (rather, UUID_1), the user is denied access to the system.

The downside of this method is that trust relies on the third party validating correct behavior – but that is true to an extent for all methods.

Method #4: What we’re asking Google to implement

This method does not yet exist but we are asking Google to support it. We want Google to follow suit after AWS and facilitate external ID checks while impersonating service accounts. After Google implements this, the desired flow would be:

1. The customer registers with ExampleVendor.
2. ExampleVendor asks the customer to register a service account with an external ID created by the vendor.
3. When ExampleVendor impersonates the service account, GCP checks that the external ID of the connection request is identical to that set by the customer in its service account.

What can you do about it?

First, you can add a star to our feature request for a built-in mechanism for secure third-party access with short-lived credentials.

As a customer

1. Ask your vendors about moving away from permanent credentials and using short-lived credentials with service account impersonation instead.
2. Ensure they use one of the aforementioned solutions (or a different one) to avoid the confused deputy problem.

As a vendor

Seek to move away from permanent credentials and service account keys, and implement one of the methods in this article (or another one) to use short-lived credentials to ensure secure access to customer environments.

Considering moving to a safer method of third-party access? Using a different secure method? I’d love to hear from you.

Contact me!
Twitter: @NoamDahan