EKS pod identity: The next step beyond OIDC and IRSA

Overview

The objective of this blog is to introduce the concept of EKS Pod identity which is being used to grant IAM permissions to a service account, and only Pods that use that service account have access to those permissions following the principle of least privileges and credential isolation.

IAM Role for Service Accounts (IRSA) is commonly used in EKS to provide temporary access to containers running in a particular pod, which is being mapped by service account annotations to the respective workload. This involved configuring a trust relationship between the AWS account and cluster control plane using OpenID Connect(OIDC).

This has limitations due to the trust relationship requirement, such as the need for administrator permissions to create OIDC providers or update the IAM trust policy. Moreover, you need to update the trust policy for every new cluster, and as the number of clusters increases, you will start hitting the IAM trust policy size limit. To overcome this, you may need to duplicate IAM roles.

Recently, AWS introduced “EKS Pod Identities” as a new mechanism for associating AWS IAM roles with k8s pods (selected via k8s service account associations).

EKS Pod identity will coexist with IRSA, which offers an additional solution to obtain IAM permissions for the pod. IRSA has the flexibility to work across different Kubernetes deployment options, including EKS in the cloud, EKS Anywhere, self-managed Kubernetes cluster on Amazon EC2, and ROSA. EKS Pod identity is purposely designed for EKS in the cloud.

Pod Identity uses a new EKS service principal, “pods.eks.amazonaws.com,” that can be used to establish trust between IAM roles and the EKS service, and the new APIs on EKS that enable you to set up permissions without the need to execute privileged IAM operations like the setup of an OIDC identity provider, the EKS service, and the new APIs on EKS that enable you to setup permissions without the need to execute privileged IAM operations like the setup of an OIDC identity provider.

Pod identity also supports IAM role session tags. Role session tags enable IAM administrators to author a single permission policy that can work across roles by allowing access to AWS resources based on matching tags.

At Clearwater, we are leveraging the benefits of three different IAM roles independently available for EKS through traditional IRSA, new “pod identities,” and the host node’s instance role, depending upon the use case.

Zoom image will be displayed

How it works

Below is a diagram showing the flow of how this system works using EKS Pod Identity:

Zoom image will be displayed

EKS pod identity being used to access s3

1. The IAM role access s3 bucket with a trust can be assumed by EKS Service principal pods.eks.amazonaws.com. The trust policy for the IAM role would look like the policy shown below. Optionally, you can restrict the use of the role to certain EKS clusters by using a conditional string as shown below: trust policy for the IAM role would look like the policy shown below. Optionally, you can restrict the use of the role to certain EKS clusters by using a conditional string as shown below:

{
  "Version": "2012–10–17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
      "Service": "pods.eks.amazonaws.com"
      },
      "Action": [
        "sts:AssumeRole",
        "sts:TagSession"
      ],
      "Condition": {
        "StringEquals": {
          "aws:SourceAccount": "aws-account-number"
        },
        "ArnEquals": {
          "aws:SourceArn": "cluster-arn"
        }
      }
    }
  ]
}

2. PodIdentityAssociation, which creates an association between the IAM role and kubernetes service account.

3. After the AWS IAM role is associated with the service account, any newly created pods using that service account will be intercepted by the EKS Pod Identity webhook.

4. AWS SDK/CLI calls the Amazon EKS Pod Identity Agent endpoint to retrieve the temporary IAM credentials.

5. AWS SDKs will use the vended temporary credentials to access other AWS resources.

Pre-requisites:

· Eks-auth service endpoint should be enabled on vpc for private eks-clusters.

· Nodes should be able to download images from Amazon[SN1] ECR.

· Default node role should have permissions “eks-auth:AssumeRoleForPodIdentity” for the agent to do the AssumeRoleForPodIdentity action in the EKS Auth API.

Getting Started

To activate the add-on, add these lines in eks_addons block:

eks_addons = {
  ...
  eks-pod-identity-agent = {
    most_recent = true
  }
  ...
}

You should have something similar to this:

module "eks" {
  source = "terraform-aws-modules/eks/aws"
  version = "~> 20.0"
  cluster_name = var.cluster_name
  cluster_version = lookup(local.versions, "kubernetes", "1.27")
  vpc_id = var.vpc_id
  subnet_ids = local.node_subnet_ids
  cluster_enabled_log_types = var.logging_types
  cluster_endpoint_private_access = true
  cluster_addons = {
    coredns = {
      most_recent = true
    }
    vpc-cni = {
      most_recent = true
    }
    kube-proxy = {
      most_recent = true
    }
    eks-pod-identity-agent = {
      most_recent = true
    }
    ...
    tags = {
      Environment = "dev"
    }
  }
}

Note: An “eks-auth:AssumeRoleForPodIdentity” permission must be added to the node’s role.

Associate role and service account

Example to access s3, creating an association is simple here and includes below steps:

Creating an IAM Role, that needs to be assumed to access s3:

data "aws_iam_policy_document" "assume_role" {
  statement {
    effect = "Allow"
    principals {
    type = "Service"
    identifiers = ["pods.eks.amazonaws.com"]
    }
    actions = [
      "sts:AssumeRole",
      "sts:TagSession"
    ]
  }
}
resource "aws_iam_role" "s3_role" {
    name = "eks-pod-identity-s3"
    assume_role_policy = data.aws_iam_policy_document.assume_role.json
}
resource "aws_iam_role_policy_attachment" "example_s3" {
    policy_arn = "arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess"
    role = aws_iam_role.s3_role.name
}

At this point, we can create an aws_eks_pod_indentity_association resource that associates the role to a specific service account. There are no requirements that a service account already exists on the cluster, a behaviour that can help a lot in automation.

resource "aws_eks_pod_identity_association" "association" {
  cluster_name = var.cluster_name
  namespace = var.namespace
  service_account = var.service_account_name
  role_arn = aws_iam_role.s3_role.arn
}

That’s it. When a pod that uses the service account with that name in the appropriate namespace is identified, the pod identity agent will add some environment variables to the pods that AWS SDK will use to retrieve the credentials.

Session tags

During the assumeRole operation, the EKS pod identity agent will attach a set of tags like service account name, cluster name, and namespace that can be used to grant access to AWS resources in the role’s policies in a more granular way, allowing the reuse of the same role with multiple service accounts and reducing the duplicity.

In the same way, every tag associated with the IAM role can be accessed in this format ${aws:PrincipalTag/tagname}

An example policy that limits the access to some parameters in system manager service based on the cluster name may appear like that

data "aws_iam_policy_document" "read_parameters_store" {
  statement {
    effect = "Allow"
    condition {
      test = "StringEquals"
      variable = "ssm:ResourceTag/eks-cluster-name"
      values = ["${aws:PrincipalTag/eks-cluster-name}"]
    }
    resources = [
      "parameterarn", …
    ]
    actions = [
      "ssm:GetParameter",
      "ssm:GetParameters"
    ]
  }
}

Migration from IRSA to Pod Identity

You need to be at EKS cluster version 1.24 and above. Have add-ons and IAM roles in place as explained above. To ensure a seamless migration, you can update your existing IAM roles to trust both the new EKS service principal as well as the OIDC provider endpoint supported by IRSA as shown:

Zoom image will be displayed

The EKS Pod Identity webhook gives preference to EKS Pod Identity over IRSA, when both trusted entities are present. After that, you can create an aws_eks_pod_indentity_association resource that associates the role to a specific service account. Once the association is in place, all new/restarted pods will have their pod spec mutated with the new environment variables supported by EKS Pod Identity.

Conclusion

EKS pod Identity is easy to set up without much of a dependency from an administrative access point of view. This feature makes managing IAM permissions for EKS a bit easier and ensures that apps are deployed more securely. This feature can be complimented with IRSA or easily migrated from IRSA, and in the end, it is important to decide which method best suits your use case.

---

About the author

Chetan Sharma is a Senior Cloud Engineer at Clearwater Analytics, bringing a versatile background in Cloud architecting and modernisation. With experience spanning across the domains, Chetan is pretty helpful in adapting new challenges related to app modernisation and cloud native stuff.