Hello Beautiful People of Internet! 👋

Hope you guys are doing well tinkering with your clusters :)

---

The Question

Let me ask you folks a question:

"How do you provide IAM access to your pods in your EKS Cluster?"

Possible Answers

1. Hardcoding AWS Access and Secret Access Key

for real? is someone really doing this in EKS in 2024? It can be the worst way to do this. Not Recommended at all. Please don't even think about it. You can imagine all the security risks involved.

2. Using Open Source Tools Like kube2iam or kiam

While these tools are better than previous method, but they are not without their drawbacks. This approach intercepts your AWS requests and injects temporary STS credentials using a dedicated agent which generally runs as a deamonset. However, it requires granting your worker nodes access to the roles intended for your pods. This setup potentially grants nodes more permissions than they typically need, which can introduce security concerns. So let's throw this approach as well right out of the window!

3. Leveraging IRSA (Good Option)

You can use IAM Roles for Service Accounts(IRSA) which natively connect IAM roles from AWS to Service Accounts of Kubernetes.

For this to work, you have to follow a certain process.

You have to mention all the details of target service account name, namespace and cluster details in the IAM Role's trust policy. for example :

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "$PROVIDER_ARN"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${ISSUER_HOSTPATH}:sub": "system:serviceaccount:default:my-serviceaccount"
        }
      }
    }
  ]
}

It's definitely one of the best option but I didn't like how we needed to pollute the IAM role trust policy principals for all the namespace and service account in which we want to use that role. For example, if you have 5 service account in different namespaces but they have to use the same IAM role then in the trust policy, you have to repeat above written code 5 times for every principal (namespace + Service Account).

Also, you have to annotate your service account to let it know about the role it have to assume. Too many manual steps and misery of adding principals in Trust Policy :')

Also, A while back (> 2 years), I already discussed IRSA in one of the blogs, do check that out here or image below for more understanding :

---

4. New! - EKS Pod Identity

   Enter Nov 2023, AWS released a feature called EKS Pod Identity which made it more easier to delegate IAM access to pods running in your EKS Cluster.

   This blog is also about this specific feature. Let's dive into it.

   

What is EKS Pod Identity

In the ever-evolving world of cloud computing, one of the most critical challenges cluster administrators face is securely managing access to AWS services from within Kubernetes pods. Traditionally, this has been a daunting task, often involving complex configurations and potential security risks.

However, with AWS's introduction of EKS Pod Identities, cluster administrators now have a robust and simplified way to handle IAM access for their pods.

How to Create and Use EKS Pod Identity

1. Make sure you have installed EKS Pod Identity EKS Agent Add-on.

2. Now, Create your desired IAM Role.

3. In the trust policy, add the following code :

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Principal": {
                    "Service": "pods.eks.amazonaws.com"
                },
                "Action": [
                    "sts:TagSession",
                    "sts:AssumeRole"
                ]
            }
        ]
    }
   

   (it will remain same for all EKS Pod Identities and for a role which is being used by any number of EKS Pod Identities, you don't have to change it ever. That's one of the advantages of using this approach.)

4. In your EKS console, as shown below, Create a new Pod Identity Association

4. Enter all the details, Select your newly created role with pod identity, your namespace and then name of ServiceAccount.

Observation*- you can enter name of namespace and ServiceAccount which are not yet created. It will be apply automatically once they are created. Shown in below screenshot.*

5. Now, Test above configuration :

   1. We created EKS Pod Identity Association as shown in above screenshot.

   2. Created a pod, which run aws-cli & checks IAM access using aws sts get-caller-identity command.

   3. Make sure it uses ServiceAccount what we have configured in EKS Pod Identity.

       apiVersion: v1
       kind: Pod
       metadata:
         name: aws-cli-pod
         namespace: kube-system # as per Pod Identity Association
       spec:
         serviceAccountName: metrics-server # as per Pod Identity Association
         containers:
           - name: aws-cli-container
             image: amazon/aws-cli:latest
             command:
               - /bin/sh
               - -c
               - |
                 # Command to check IAM identity
                 aws sts get-caller-identity
             env:
               - name: AWS_REGION
                 value: ap-south-1
      

   4. Create this pod and check logs.

      

      Nice! We can see our pod have assumed the IAM role which we created earlier and mapped in EKS Pod Identity.

      

      note - This was manual creation approach, if you want, you can use your favourite IaC tool like Terraform, CloudFormation, etc.

But the question is...

from the docs "If a pod uses a service account that has an pod identity association, Amazon EKS sets environment variables in the containers of the pod. The environment variables configure the AWS SDKs, including the AWS CLI, to use the EKS Pod Identity credentials"

  - env:
    - name: AWS_STS_REGIONAL_ENDPOINTS
      value: regional
    - name: AWS_DEFAULT_REGION
      value: ap-south-1
    - name: AWS_REGION
      value: ap-south-1
    - name: AWS_CONTAINER_CREDENTIALS_FULL_URI
      value: http://169.254.170.23/v1/credentials
    - name: AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE
      value: /var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token

in simple terms, When you configure EKS Pod Identity for a workload, the pods get injected some useful env vars and eks-pod-identity-token volume by the EKS Pod Identity Agent. And all supported SDKs and AWS CLI versions already have support of using these variable for AWS authentication automatically.

I hope clears that doubt! (read this for more info : https://docs.aws.amazon.com/eks/latest/userguide/pod-id-how-it-works.html)

Also, Keep in mind that AWS has a priority order to pick credentials if it's available at multiple places. For example - In JavaScript SDKs, it follows as attached below :

---

Advantages

1. Least Privilege - EKS Pod Identities enable scoping IAM permissions to a specific service account, ensuring that only pods using that account can access the associated permissions.

2. Credential Isolation - Pods can only access credentials for the IAM role linked to their service account, ensuring containers cannot access credentials from other pods.

3. Audit - AWS CloudTrail provides access and event logging which makes it easier to audit operations related operations in the cluster.

4. No Need of creating OIDC Provider unlike in IRSA setup.

5. Non-polluting IAM Role Trust Policy - single IAM principal instead of the separate principals for each cluster/namespace/ServiceAccount which we had to do in IRSA setup.

6. Scalability - The EKS Auth service in EKS Pod Identity assumes temporary credentials once per node, rather than for each pod's AWS SDK. The Amazon EKS Pod Identity Agent on each node then distributes these credentials to the SDKs, reducing the load by preventing duplication across pods.

   

---

You know what When I was working with some JavaScript microservice at Kutumb which was using JavaScript SDK v2, We found that this new feature is not yet supported by latest v2 SDK!

We were doing a POC migrating our services from kube2iam to EKS Pod Identity, and with JavaScript v2 AWS SDK, it was failing to get credentials.

So felt like digging into it, with help of awesome genius colleagues I was able to find the root cause and fixed that :)

also, raised the PR of the fix, check here : https://github.com/aws/aws-sdk-js/pull/4565

---

Thank you for reading this piece of information, wanted to share it from past couple of month, finally got the time to wrote and publish it :)

If you have any suggestions and/or feedback, please let me know.

Until we meet next time 👋

---

References

1. https://maturitymodel.security.aws.dev/en/2.-foundational/dont-store-secrets-in-code/

2. https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-eks-pod-identity/

3. https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html

4. https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/

5. https://www.reddit.com/r/aws/comments/1853wbh/eks_pod_identity_vs_irsa/