Search for a command to run...
Effective Techniques for Providing IAM Access to EKS Pods
Hey, Y'all 👋 Kratik Here :)
Hailing from The City of Lakes, India.
I love to learn and share stuff about DevOps, SRE, Security, Cost Optimization, etc.
Join me on this journey where we will learn together with the help bunch of interesting blogs.
Amazing article, thank you for spreading good practices! was looking into this for a long time.
DNS Search Domains, ndots, and Kubernetes Resolver Behavior
Understand how IAM Access Entries can replace aws-auth for managing EKS IAM authentication.
Ever wondered how can you create your own custom Github Action and publish it to the GitHub Marketplace? Let's learn it step by step in this blog.
Let's setup Vault with External-secrets to manage secrets in Kubernetes like never before!
Hello Beautiful People of Internet! 👋
Hope you guys are doing well tinkering with your clusters :)
Let me ask you folks a question:
"How do you provide IAM access to your pods in your EKS Cluster?"
for real? is someone really doing this in EKS in 2024? It can be the worst way to do this. Not Recommended at all. Please don't even think about it. You can imagine all the security risks involved.
While these tools are better than previous method, but they are not without their drawbacks. This approach intercepts your AWS requests and injects temporary STS credentials using a dedicated agent which generally runs as a deamonset. However, it requires granting your worker nodes access to the roles intended for your pods. This setup potentially grants nodes more permissions than they typically need, which can introduce security concerns. So let's throw this approach as well right out of the window!
You can use IAM Roles for Service Accounts(IRSA) which natively connect IAM roles from AWS to Service Accounts of Kubernetes.
For this to work, you have to follow a certain process.
You have to mention all the details of target service account name, namespace and cluster details in the IAM Role's trust policy. for example :
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "$PROVIDER_ARN"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"${ISSUER_HOSTPATH}:sub": "system:serviceaccount:default:my-serviceaccount"
}
}
}
]
}
It's definitely one of the best option but I didn't like how we needed to pollute the IAM role trust policy principals for all the namespace and service account in which we want to use that role. For example, if you have 5 service account in different namespaces but they have to use the same IAM role then in the trust policy, you have to repeat above written code 5 times for every principal (namespace + Service Account).
Also, you have to annotate your service account to let it know about the role it have to assume. Too many manual steps and misery of adding principals in Trust Policy :')
Also, A while back (> 2 years), I already discussed IRSA in one of the blogs, do check that out here or image below for more understanding :
Enter Nov 2023, AWS released a feature called EKS Pod Identity which made it more easier to delegate IAM access to pods running in your EKS Cluster.
This blog is also about this specific feature. Let's dive into it.
In the ever-evolving world of cloud computing, one of the most critical challenges cluster administrators face is securely managing access to AWS services from within Kubernetes pods. Traditionally, this has been a daunting task, often involving complex configurations and potential security risks.
However, with AWS's introduction of EKS Pod Identities, cluster administrators now have a robust and simplified way to handle IAM access for their pods.
Make sure you have installed EKS Pod Identity EKS Agent Add-on.
Now, Create your desired IAM Role.
In the trust policy, add the following code :
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "pods.eks.amazonaws.com"
},
"Action": [
"sts:TagSession",
"sts:AssumeRole"
]
}
]
}
(it will remain same for all EKS Pod Identities and for a role which is being used by any number of EKS Pod Identities, you don't have to change it ever. That's one of the advantages of using this approach.)
In your EKS console, as shown below, Create a new Pod Identity Association
Observation*- you can enter name of namespace and ServiceAccount which are not yet created. It will be apply automatically once they are created. Shown in below screenshot.*
Now, Test above configuration :
We created EKS Pod Identity Association as shown in above screenshot.
Created a pod, which run aws-cli & checks IAM access using aws sts get-caller-identity command.
Make sure it uses ServiceAccount what we have configured in EKS Pod Identity.
apiVersion: v1
kind: Pod
metadata:
name: aws-cli-pod
namespace: kube-system # as per Pod Identity Association
spec:
serviceAccountName: metrics-server # as per Pod Identity Association
containers:
- name: aws-cli-container
image: amazon/aws-cli:latest
command:
- /bin/sh
- -c
- |
# Command to check IAM identity
aws sts get-caller-identity
env:
- name: AWS_REGION
value: ap-south-1
Create this pod and check logs.
Nice! We can see our pod have assumed the IAM role which we created earlier and mapped in EKS Pod Identity.
note - This was manual creation approach, if you want, you can use your favourite IaC tool like Terraform, CloudFormation, etc.
from the docs "If a pod uses a service account that has an pod identity association, Amazon EKS sets environment variables in the containers of the pod. The environment variables configure the AWS SDKs, including the AWS CLI, to use the EKS Pod Identity credentials"
- env:
- name: AWS_STS_REGIONAL_ENDPOINTS
value: regional
- name: AWS_DEFAULT_REGION
value: ap-south-1
- name: AWS_REGION
value: ap-south-1
- name: AWS_CONTAINER_CREDENTIALS_FULL_URI
value: http://169.254.170.23/v1/credentials
- name: AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE
value: /var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token
in simple terms, When you configure EKS Pod Identity for a workload, the pods get injected some useful env vars and eks-pod-identity-token volume by the EKS Pod Identity Agent. And all supported SDKs and AWS CLI versions already have support of using these variable for AWS authentication automatically.
I hope clears that doubt! (read this for more info : https://docs.aws.amazon.com/eks/latest/userguide/pod-id-how-it-works.html)
Also, Keep in mind that AWS has a priority order to pick credentials if it's available at multiple places. For example - In JavaScript SDKs, it follows as attached below :
Least Privilege - EKS Pod Identities enable scoping IAM permissions to a specific service account, ensuring that only pods using that account can access the associated permissions.
Credential Isolation - Pods can only access credentials for the IAM role linked to their service account, ensuring containers cannot access credentials from other pods.
Audit - AWS CloudTrail provides access and event logging which makes it easier to audit operations related operations in the cluster.
No Need of creating OIDC Provider unlike in IRSA setup.
Non-polluting IAM Role Trust Policy - single IAM principal instead of the separate principals for each cluster/namespace/ServiceAccount which we had to do in IRSA setup.
Scalability - The EKS Auth service in EKS Pod Identity assumes temporary credentials once per node, rather than for each pod's AWS SDK. The Amazon EKS Pod Identity Agent on each node then distributes these credentials to the SDKs, reducing the load by preventing duplication across pods.
You know what When I was working with some JavaScript microservice at Kutumb which was using JavaScript SDK v2, We found that this new feature is not yet supported by latest v2 SDK!
We were doing a POC migrating our services from kube2iam to EKS Pod Identity, and with JavaScript v2 AWS SDK, it was failing to get credentials.
So felt like digging into it, with help of awesome genius colleagues I was able to find the root cause and fixed that :)
also, raised the PR of the fix, check here : https://github.com/aws/aws-sdk-js/pull/4565
Thank you for reading this piece of information, wanted to share it from past couple of month, finally got the time to wrote and publish it :)
If you have any suggestions and/or feedback, please let me know.
Until we meet next time 👋
https://maturitymodel.security.aws.dev/en/2.-foundational/dont-store-secrets-in-code/
https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-eks-pod-identity/
https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html
https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/
https://www.reddit.com/r/aws/comments/1853wbh/eks_pod_identity_vs_irsa/