Before we dive deeper, let’s take a step back and understand how a basic DNS query and resolution actually work

Working of a DNS Query

so for example - let’s understand what happens when you enter blogs.kratik.dev in your browser

(I hope somewhere someone is actually entering this to read my blogs 🫠 )

Your browser tries to resolve IP of the domain in following order

1. check if domain exists in the /etc/hosts file

2. If the domain is not found there, the OS sends the request to the DNS servers configured on the device (from Wi-Fi, Ethernet, mobile data, or VPN).

On Linux, these DNS server settings are exposed through /etc/resolv.conf;
on other systems, they are managed internally, but the behaviour is the same.

what is /etc/hosts file ?

/etc/hosts is a local, static hostname-to-IP mapping file used by the operating system before DNS is consulted. It allows you to manually define how hostnames resolve without querying any DNS server.

in my machine, it looks something like this :

##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##
127.0.0.1    localhost
255.255.255.255    broadcasthost
::1             localhost

# Added by Docker Desktop
# To allow the same kube context to work on the host and the container:
127.0.0.1 kubernetes.docker.internal

127.0.0.1 kratik.servers
# End of section

💡- now you also know how localhost IP resolution works :)

so let’s test this with a domain we have defined kratik.awesome - it does not exist

let’s verify it first

curl

browser

okay, obviously it does not exist

(I wish It did )

Let’s add this domain in our /etc/hosts file

##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##
127.0.0.1    localhost
255.255.255.255    broadcasthost
::1             localhost

# Added by Docker Desktop
# To allow the same kube context to work on the host and the container:
127.0.0.1 kubernetes.docker.internal

127.0.0.1 kratik.servers

127.0.0.1 kratik.awesome
# End of section

now let’s run a test server in a python quickly 😎

(basically this commands runs a python server which serves files from the specified directory)

mkdir -p /tmp/sample

cat <<EOF >/tmp/sample/index.html
<h1>Hello from DNS Blog!</h1>
EOF

python3 -m http.server 80 --directory /tmp/sample

Ok, let’s check browser again

now you can see the above webpage, because :

1. browser requested the IP of domain kratik.awesome

2. /etc/hosts had an entry for above domain so replied with the IP instantly which was 127.0.0.1

3. we already running a Python HTTP server on port 80, so this server responded with our created HTML file.

I hope that clear the basics for /etc/hosts file, let’s move next to resolv.conf file

what is /etc/resolv.conf file

/etc/resolv.conf is a system configuration file that tells the operating system which DNS servers to use when resolving domain names.

it look something like this on a linux machine (this is an AWS EC2 machine btw)

# This is /run/systemd/resolve/resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 172.22.0.2
search ap-south-1.compute.internal

basic explanation of the fields in resolver files are:

1. nameserver - server to query for DNS lookups, you can define more servers by adding one more nameserver in same format in a new line.

2. search - this option tells the system which domain names to automatically append when you try to resolve a short (unqualified) hostname.

3. option - (it’s not present in the above file) - this field is used to configure the behaviour of the DNS lookups, for example - how long it waits, how many times it retries, and when it treats a name as fully qualified - we will understand it better further.

   available options are ndots & timeout & attempts

now let’s move deeper and check a resolver file in Kubernetes environment

/etc/resolv.conf file in Kubernetes Pods

If you see the same file in your pods in a Kubernetes cluster, it will look something like this

search kube-system.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5

This is where DNS resolution gets interesting: search domains and ndots

Let’s understand it one by one

search

search kube-system.svc.cluster.local svc.cluster.local cluster.local

What it does : If a hostname is not fully qualified, the resolver tries appending each of these domains in order until one resolves.

ndots

options ndots:5

so this actually counts the dots(.) in our DNS query, for example - api.github.com has 2 dots and my-service.my-namespace.svc.cluster.local has 4 dots

so basically it means, if a domain does not contain dots more than or equal to 5, it will not be considered as a fully qualified domain name and then all the domains listed in `search` will be appended in the DNS lookup one by one.

Let’s see this in action!

I will do a DNS query and also show you the logs of the dns server in k8s.

Setup of the k8s cluster for this test

We have a Kubernetes cluster with two namespaces:

• my-namespace

  • also create a nginx service here named my-service

• my-other-namespace

Each namespace has its own DNS search domain:

• my-namespace.svc.cluster.local

• my-other-namespace.svc.cluster.local

Kubernetes configures pods so that DNS lookups first try services in the same namespace, and then fall back to cluster-wide service discovery.

I will create netshoot pods to do DNS lookup in each of these namespace.

Case 1: Lookup from the same namespace (my-namespace)

cat /etc/resolv.conf file in 1st namespace

Command

CoreDNS logs

"A IN my-service.my-namespace.svc.cluster.local." NOERROR
"AAAA IN my-service.my-namespace.svc.cluster.local." NOERROR

What happened

• The pod’s resolver has this search list:

    search my-namespace.svc.cluster.local svc.cluster.local cluster.local
  

• Since my-service is a short name, the resolver appends the first search domain.

• The first attempt is:

    my-service.my-namespace.svc.cluster.local
  

• This service exists, so CoreDNS returns NOERROR.

• Resolution stops immediately.

✅ Result: my-service resolves successfully inside its own namespace.

Case 2: Lookup from another namespace (my-other-namespace)

cat /etc/resolv.conf file in 2nd namespace

Command

CoreDNS logs (in order)

"A IN my-service.my-other-namespace.svc.cluster.local." NXDOMAIN
"A IN my-service.svc.cluster.local." NXDOMAIN
"A IN my-service.cluster.local." NXDOMAIN
"A IN my-service." NXDOMAIN

What happened

From my-other-namespace, the resolver search list is:

search my-other-namespace.svc.cluster.local svc.cluster.local cluster.local

The resolver tries each option one by one:

1. my-service.my-other-namespace.svc.cluster.local
   Service does not exist → NXDOMAIN

2. my-service.svc.cluster.local
   No service with that name across namespaces → NXDOMAIN

3. my-service.cluster.local
   Not a valid service zone → NXDOMAIN

4. my-service (as-is)
   No global DNS record → NXDOMAIN

After all attempts fail, the lookup fails.

Result: my-service does not resolve from another namespace.

additionally, as soon as I add the namespace name in my lookup, it works! even in the other namespace. why? because now one of the search domain led to a valid fully qualified domain name.

CoreDNS logs

In this blog, we walked through how DNS resolution works at a basic level and then explored how Kubernetes builds on top of it using search domains and ndots.

By understanding how these pieces fit together, you can better debug DNS issues, avoid unnecessary lookups, and design more predictable service communication inside your cluster.

💡 - Default value of ndots in Linux (glibc) is 1 and in k8s environments it’s 5

Bonus Tip

How to skip adding search domains to your DNS query?

You can do this by adding a dot (.) at the end of the hostname.

for example :

nslookup my-service.
curl my-service.

What the trailing dot does

A trailing dot tells the resolver:
“This name is already fully qualified — do not append search domains.”

Where is the trailing dot useful?

Adding a trailing dot (.) is useful whenever you want full control over DNS resolution and want to avoid resolver-side surprises.

Hope you liked this knowledge byte.

Tune in for more blogs. Time to clear the year old drafts.

see you in the next one!

Hope you're happy managing your EKS clusters. There is this one thing we all do in our EKS related shenanigans - To provide an IAM entity access to your EKS Cluster. I personally find this annoying as I have to go and manually edit a ConfigMap called aws-auth

from the official docs:

One more thing is that whoever IAM entity (User or Role) creates the EKS cluster, automatically becomes the owner of that EKS Cluster. In some cases, we don't really want that, because it may become hard to track sometimes who created the cluster as this is not visible in any configuration in AWS.

In Dec 2023, AWS announced, new feature in Elastic Kubernetes Service (EKS), which helps cluster administrators to simplify mapping and configuration of AWS Identity and Access Management (IAM) users and roles with Kubernetes clusters.

But let's understand this deprecated approach first

What is aws-auth ConfigMap

This ConfigMap resides in the kube-system namespace and controls the access of IAM principals who are allowed to access the cluster.

Internally it uses AWS IAM Authenticator in Control plane which reads it's configuration from this aws-auth ConfigMap.

This ConfigMap looks something like below :

Limitations of aws-auth ConfigMap

This feature comes with a burden to manage complex ConfigMap for every IAM Principal. You may end up messing up this ConfigMap and hence, losing access to your cluster 🤷‍♂️

Also, It does not come with IAM audit logging.

What is EKS IAM Access Entry

This is a new set of APIs, which added another yet native way to setup authentication between your Kubernetes Cluster and IAM entities.

You can setup this during or after cluster creation. Also, At the time of writing this article, we have option to use either aws-auth ConfigMap or IAM Access Entries or both.

Create IAM Access Entry in EKS

You can either configure this at the time of cluster creation or afterwards for an existing EKS cluster as well.

Let's take example when creating a fresh cluster :

1. I have created a fresh EKS Cluster, It is important to note that, In the Cluster access section you have to keep config like below:

   

^ Here you can See I have selected Disallow cluster administrator access and In Cluster authentication mode, I have selected EKS API.

It ensures that whoever (IAM User or Role) creating this cluster does not get the admin/owner access by default. And when you select EKS API only in the cluster auth mode, you are opting here to utilize IAM Access Entries rather than aws-auth ConfigMap.

2. Once Cluster creation is done, try creating your first IAM access entry as show below.

3. We see a screen with bunch of inputs. Let's try to understand it :

   

   1. IAM Principal - select the IAM entity for which you want to grant access. (similar to mapUsers and mapRoles from aws-auth ConfigMap) In our case, we have selected an IAM user - test-user

   2. Type (optional) - as we are creating this for our test IAM user, we can keep this as Standard , Other Options are : EC2 Linux, EC2 Windows and FARGATE_LINUX
      note from the docs - If you create an access entry with typeEC2 LinuxorEC2 Windows, the IAM principal creating the access entry must have theiam:PassRolepermission.

   3. Username (optional) - username to be used while authenticating with K8s APIs.

   4. Groups (optional) - You can mention any group if you are using one in RoleBinding or ClusterRoleBinding subjects. I was wondering how to bind a user, and in this documentation, I found out (It does not apply to EKS but just in case) :

      

      also, FYI

      You don't need to create an access entry for an IAM role used by a managed node group or a Fargate profile. Amazon EKS automatically adds entries for these roles to the aws-auth ConfigMap, regardless of your cluster's platform version.

      For more info on these inputs, check the table below from AWS Docs.

      

4. Choose Access Policy. This will define your access level in your EKS Cluster.

   

   Here, you can select some of the pre-defined access policies.

   1. AmazonEKSAdminPolicy

   2. AmazonEKSClusterAdminPolicy

   3. AmazonEKSAdminViewPolicy

   4. AmazonEKSEditPolicy

   5. AmazonEKSViewPolicy

   6. AmazonEMRJobPolicy

Either you can grant the access using policies at cluster level or you can bind a namespace rather than whole cluster for more fine-grained controls. (as shown below)

Select your policy and choose scope and then you have to click on Add Policy to add all the policies one by one. (shown below)

5. Coming back to our example, For our test-user I am selecting AmazonEKSAdminPolicy policy.

6. To test this, I logged in as test-user in AWS CLI.

   

7. Also I downloaded the kubeconfig using following command.

    aws eks update-kubeconfig --name kratik-testing-iam-access-entries --alias kratik-testing-iam-access-entries
   

8. Before creating the IAM access Entry, I used to see this.

9. After creating the IAM Access Entry (as we did in Step #5)

woohoo! we were able to delegate IAM permissions to our test-user using IAM Access Entry.

Advantages

1. Visibility in Console about the access about the IAM Principals having the access.

2. Saves you from messing up the EKS Access by not managing complex entries in AWS ConfigMap.

and anyways, as aws-auth is deprecated you should migrate to EKS IAM Access Entries.

Thank you for reading this article 🌻

Let me know if you have any feedback or suggestions.

Peace out!

References

1. https://aws.github.io/aws-eks-best-practices/security/docs/iam/

2. https://docs.aws.amazon.com/eks/latest/userguide/creating-access-entries.html

3. https://docs.aws.amazon.com/eks/latest/userguide/access-entries.html

4. https://docs.aws.amazon.com/eks/latest/userguide/access-policies.html

5. https://docs.aws.amazon.com/eks/latest/userguide/migrating-access-entries.html

6. https://docs.aws.amazon.com/eks/latest/userguide/auth-configmap.html

Hope you guys are doing well tinkering with your clusters :)

The Question

Let me ask you folks a question:

"How do you provide IAM access to your pods in your EKS Cluster?"

Possible Answers

1. Hardcoding AWS Access and Secret Access Key

for real? is someone really doing this in EKS in 2024? It can be the worst way to do this. Not Recommended at all. Please don't even think about it. You can imagine all the security risks involved.

2. Using Open Source Tools Like kube2iam or kiam

While these tools are better than previous method, but they are not without their drawbacks. This approach intercepts your AWS requests and injects temporary STS credentials using a dedicated agent which generally runs as a deamonset. However, it requires granting your worker nodes access to the roles intended for your pods. This setup potentially grants nodes more permissions than they typically need, which can introduce security concerns. So let's throw this approach as well right out of the window!

3. Leveraging IRSA (Good Option)

You can use IAM Roles for Service Accounts(IRSA) which natively connect IAM roles from AWS to Service Accounts of Kubernetes.

For this to work, you have to follow a certain process.

You have to mention all the details of target service account name, namespace and cluster details in the IAM Role's trust policy. for example :

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "$PROVIDER_ARN"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${ISSUER_HOSTPATH}:sub": "system:serviceaccount:default:my-serviceaccount"
        }
      }
    }
  ]
}

It's definitely one of the best option but I didn't like how we needed to pollute the IAM role trust policy principals for all the namespace and service account in which we want to use that role. For example, if you have 5 service account in different namespaces but they have to use the same IAM role then in the trust policy, you have to repeat above written code 5 times for every principal (namespace + Service Account).

Also, you have to annotate your service account to let it know about the role it have to assume. Too many manual steps and misery of adding principals in Trust Policy :')

Also, A while back (> 2 years), I already discussed IRSA in one of the blogs, do check that out here or image below for more understanding :

4. New! - EKS Pod Identity

   Enter Nov 2023, AWS released a feature called EKS Pod Identity which made it more easier to delegate IAM access to pods running in your EKS Cluster.

   This blog is also about this specific feature. Let's dive into it.

   

What is EKS Pod Identity

In the ever-evolving world of cloud computing, one of the most critical challenges cluster administrators face is securely managing access to AWS services from within Kubernetes pods. Traditionally, this has been a daunting task, often involving complex configurations and potential security risks.

However, with AWS's introduction of EKS Pod Identities, cluster administrators now have a robust and simplified way to handle IAM access for their pods.

How to Create and Use EKS Pod Identity

1. Make sure you have installed EKS Pod Identity EKS Agent Add-on.

2. Now, Create your desired IAM Role.

3. In the trust policy, add the following code :

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Principal": {
                    "Service": "pods.eks.amazonaws.com"
                },
                "Action": [
                    "sts:TagSession",
                    "sts:AssumeRole"
                ]
            }
        ]
    }
   

   (it will remain same for all EKS Pod Identities and for a role which is being used by any number of EKS Pod Identities, you don't have to change it ever. That's one of the advantages of using this approach.)

4. In your EKS console, as shown below, Create a new Pod Identity Association

4. Enter all the details, Select your newly created role with pod identity, your namespace and then name of ServiceAccount.

Observation*- you can enter name of namespace and ServiceAccount which are not yet created. It will be apply automatically once they are created. Shown in below screenshot.*

5. Now, Test above configuration :

   1. We created EKS Pod Identity Association as shown in above screenshot.

   2. Created a pod, which run aws-cli & checks IAM access using aws sts get-caller-identity command.

   3. Make sure it uses ServiceAccount what we have configured in EKS Pod Identity.

       apiVersion: v1
       kind: Pod
       metadata:
         name: aws-cli-pod
         namespace: kube-system # as per Pod Identity Association
       spec:
         serviceAccountName: metrics-server # as per Pod Identity Association
         containers:
           - name: aws-cli-container
             image: amazon/aws-cli:latest
             command:
               - /bin/sh
               - -c
               - |
                 # Command to check IAM identity
                 aws sts get-caller-identity
             env:
               - name: AWS_REGION
                 value: ap-south-1
      

   4. Create this pod and check logs.

      

      Nice! We can see our pod have assumed the IAM role which we created earlier and mapped in EKS Pod Identity.

      

      note - This was manual creation approach, if you want, you can use your favourite IaC tool like Terraform, CloudFormation, etc.

But the question is...

from the docs "If a pod uses a service account that has an pod identity association, Amazon EKS sets environment variables in the containers of the pod. The environment variables configure the AWS SDKs, including the AWS CLI, to use the EKS Pod Identity credentials"

  - env:
    - name: AWS_STS_REGIONAL_ENDPOINTS
      value: regional
    - name: AWS_DEFAULT_REGION
      value: ap-south-1
    - name: AWS_REGION
      value: ap-south-1
    - name: AWS_CONTAINER_CREDENTIALS_FULL_URI
      value: http://169.254.170.23/v1/credentials
    - name: AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE
      value: /var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token

in simple terms, When you configure EKS Pod Identity for a workload, the pods get injected some useful env vars and eks-pod-identity-token volume by the EKS Pod Identity Agent. And all supported SDKs and AWS CLI versions already have support of using these variable for AWS authentication automatically.

I hope clears that doubt! (read this for more info : https://docs.aws.amazon.com/eks/latest/userguide/pod-id-how-it-works.html)

Also, Keep in mind that AWS has a priority order to pick credentials if it's available at multiple places. For example - In JavaScript SDKs, it follows as attached below :

Advantages

1. Least Privilege - EKS Pod Identities enable scoping IAM permissions to a specific service account, ensuring that only pods using that account can access the associated permissions.

2. Credential Isolation - Pods can only access credentials for the IAM role linked to their service account, ensuring containers cannot access credentials from other pods.

3. Audit - AWS CloudTrail provides access and event logging which makes it easier to audit operations related operations in the cluster.

4. No Need of creating OIDC Provider unlike in IRSA setup.

5. Non-polluting IAM Role Trust Policy - single IAM principal instead of the separate principals for each cluster/namespace/ServiceAccount which we had to do in IRSA setup.

6. Scalability - The EKS Auth service in EKS Pod Identity assumes temporary credentials once per node, rather than for each pod's AWS SDK. The Amazon EKS Pod Identity Agent on each node then distributes these credentials to the SDKs, reducing the load by preventing duplication across pods.

   

You know what When I was working with some JavaScript microservice at Kutumb which was using JavaScript SDK v2, We found that this new feature is not yet supported by latest v2 SDK!

We were doing a POC migrating our services from kube2iam to EKS Pod Identity, and with JavaScript v2 AWS SDK, it was failing to get credentials.

So felt like digging into it, with help of awesome genius colleagues I was able to find the root cause and fixed that :)

also, raised the PR of the fix, check here : https://github.com/aws/aws-sdk-js/pull/4565

Thank you for reading this piece of information, wanted to share it from past couple of month, finally got the time to wrote and publish it :)

If you have any suggestions and/or feedback, please let me know.

Until we meet next time 👋

References

1. https://maturitymodel.security.aws.dev/en/2.-foundational/dont-store-secrets-in-code/

2. https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-eks-pod-identity/

3. https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html

4. https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/

5. https://www.reddit.com/r/aws/comments/1853wbh/eks_pod_identity_vs_irsa/

Intro to GitHub Actions

I assume you already have some knowledge about GitHub actions, and in case if you don't : This is an automation tool provided by GitHub*(by Microsoft) -* as most of us have our code stored up in GitHub repos, many people use it for CI/CD pipelines.

Not just CI/CD, you can do a lot of thing with GitHub Actions, possibillites are limitless and depends on your creativity how you use it.

By Default, GitHub runs your workflow on their own servers aka runners. You can also host your CI/CD workers called self-hosted runners.

How does it work ?

Well, it's simple - you have to define a workflow file in YAML following the syntax and place this file in .github/workflows/ directory. You also have to define a trigger in this workflow and whenever that trigger event occurs, boom! your workflow is started. This is fun right ?

P.S. - I am using GitHub Actions from past couple of years and yes, and it still surprises me with some bugs (or features maybe?) but lately I have learned how to deal with it.

Types of Custom GitHub Action

From the docs https://docs.github.com/en/actions/creating-actions/about-custom-actions.

1. Docker - Docs - In this, you can containerize your action so it can run anywhere. We will be using this approach to create our custom action.

2. Javascript - Docs - GitHub runners already have node installed, so you can write your custom action in javascript to run it directly on the runner during workflow execution. It's fastest among other approaches.

3. Composite -Docs- In this, you combine multiple steps in a GitHub Action, and later you can use them as a composite workflow. I think it's superseded by Reusable Workflows and is a better alternative.

Enough with the types, let's dig into creating one.

Building a Custom GitHub Action from scratch

for the demo we will build a custom action whose job would be -

To detect if a Container image with given tag exists in a container registry (DockerHub or AWS ECR) or not, if exists, output some flag so build/push step can be skipped.

sometimes when we run the same CI/CD pipeline again (for whatever reason), we would like to skip the build/push stage so we can save some time as that image is already build and pushed, so why do it again ?

Cool, Assuming you understood the problem statement, So now we know what our custom GitHub Action will solve.

Let's Start.

Step 0 : Create and Init an empty git repo

mkdir container-image-check-custom-action
cd container-image-check-custom-action
git init

Step 1 : Write your script which solves your problem statement

We don't want to go deep into the development of below script but want to make sure it solves our problem statement.

let's write our script

script.sh

#!/bin/bash

CONTAINER_REPO_NAME="$2"
CONTAINER_IMAGE_TAG="$3"

# in case if it is a Docker image
DOCKER_HUB_USERNAME="$4"
DOCKER_HUB_PAT="$5"

function check_ecr_image_tag_if_exists() {

    echo "Started Image & Tag checking for repo $CONTAINER_REPO_NAME for tag $IMAGE_TAG"

    IMAGE_META="$(aws ecr describe-images --repository-name=$CONTAINER_REPO_NAME --image-ids=imageTag=$CONTAINER_IMAGE_TAG 2>/dev/null)"

    if [[ $? == 0 ]]; then
        IMAGE_TAGS="$(echo ${IMAGE_META} | jq '.imageDetails[0].imageTags[0]' -r)"
        echo $IMAGE_TAGS
        echo "image_exists=true" >>$GITHUB_OUTPUT
        echo "Image $1:$2 exists on ecr."
        echo $IMAGE_META
    else
        echo "$1:$2 does not exist on ecr."
        echo "image_exists=false" >>$GITHUB_OUTPUT
    fi

}

function check_docker_image_tag_if_exists() {

    echo "Started Image & Tag checking for repo $CONTAINER_REPO_NAME by user $DOCKER_HUB_USERNAME for tag $CONTAINER_IMAGE_TAG"

    TOKEN=$(curl -sSLd "username=${DOCKER_HUB_USERNAME}&password=${DOCKER_HUB_PAT}" https://hub.docker.com/v2/users/login | jq -r ".token")
    REPO_RESPONSE=$(curl -sH "Authorization: JWT $TOKEN" "https://hub.docker.com/v2/repositories/${DOCKER_HUB_USERNAME}/${CONTAINER_REPO_NAME}/tags/${CONTAINER_IMAGE_TAG}/")

    echo
    echo Response is:
    # echo $REPO_RESPONSE | jq .
    echo
    echo Tag status is:
    TAG_STATUS=$(echo $REPO_RESPONSE | jq .tag_status | tr -d '"')
    echo $TAG_STATUS
    echo

    if [[ $TAG_STATUS == *"active"* ]]; then
        echo Docker Image $CONTAINER_REPO_NAME exists with Tag $CONTAINER_IMAGE_TAG
        echo "image_exists=true" >>$GITHUB_OUTPUT
    else
        echo "Docker Image Does Not Exist"
        echo "image_exists=false" >>$GITHUB_OUTPUT
    fi
}

# -------------------------------------------------------------------------

if [[ $1 == "ecr" ]]; then
    echo "got ecr: $1"
    check_ecr_image_tag_if_exists
elif [[ $1 == "dockerhub" ]]; then
    echo "got dockerhub: $1"
    check_docker_image_tag_if_exists
else
    echo "Unsupported Registry: $1"
    exit 1
fi

Flow of the script

1. The script when executed, will determine if a Docker Hub or ECR image exists or not based on the given repo name and tag.

2. The first argument is used for setting the target registry type - ecr or dockerhub

3. The Second and third argument is for name of the container repo and the image tag, respectively.

4. For Docker Hub, you need to provide your Docker Hub username and Docker Hub Personal Access Token, that's what 4th and 5th args are for.

5. Based on the run, if image exists or not, it will update the GitHub Actions output variable by updating GITHUB_OUTPUT file. (Read - Setting Output variables)

Example - ECR

bash container-image-check-custom-action/script.sh ecr python-server v1

Example - Docker Hub

bash container-image-check-custom-action/script.sh dockerhub python-server v1 docker_user dockerhub_my_personal_access_token_1234

Step 2 : Test Locally

Using above commands, first of all check in your local env that is this script is working as expected or not.

ECR:
args = Repo and tag

DockerHub:
notice all the args

Step 3.1 : Build Docker Image to Test

As we are using docker type of our custom action, we should build the docker image and test is locally before pushing.

As this is going to run as a container, we should verify it locally before pushing, so we can avoid excuses like :

Dockerfile

FROM amazon/aws-cli:2.15.40
RUN yum update && yum install -y jq
WORKDIR /scripts
COPY . .
RUN chmod +x /scripts/docker-entrypoint.sh
ENTRYPOINT ["/scripts/docker-entrypoint.sh"]

now, build a docker image out of this

docker build . -t container-image-check-custom-action:v0

Step 3.2: Test/Run Using Docker

docker run -it container-image-check-custom-action:v0 dockerhub shorty latest k4kratik dckr_pat_1234

we can see it give us outputs as expected. (Yes, I know. I just tested it for DockerHub and not for AWS ECR.)

Step 4 : Define the Action definition in action.yaml

Now, we will create action.yaml so we can specify GitHub how to use our script in other workflows as a custom action.

name: Container Image Existence Checker

description: Check if a given ECR or Docker Hub image exists or not in the registry.

inputs:
  type:
    required: true
    default: "ecr"
    description: Type of the registry, allowed values are "ecr" and "dockerhub"
  container_repo_name:
    required: true
    description: Name of the Container Repository
  image_tag:
    required: true
    description: Image Tag for which you want to check.
  dockerhub_username:
    required: false
    description: Docker Hub username for authentication for private repositories.
  dockerhub_token:
    required: false
    description: Docker Hub Personal Access Token for authentication for private repositories.

outputs:
  image_exists:
    description: "A boolean value to indicate if the image exists"

runs:
  using: "docker"
  image: "Dockerfile"
  args:
    - ${{ inputs.type }}
    - ${{ inputs.container_repo_name }}
    - ${{ inputs.image_tag }}
    - ${{ inputs.dockerhub_username }}
    - ${{ inputs.dockerhub_token }}

Explaining action.yaml

This is the file which stays in root of our project dir and GitHub uses this to understand your custom action.

1. inputs : we have to define what inputs we want from user when he/she uses our custom action.

2. runs : we have to define the runtime behaviour of our custom action.

   1. using : we define what we want to use for running our code. you can use node versions like node20 , docker or composite

   2. image: either you can give direct image from docker hub e.g. docker://k4kratik/container-image-existence-checker:latest or Dockerfile path to build image on the go. For now we are using Dockerfile.

   3. args : all the arguments to our code.

Step 5 : Publish To GitHub Marketplace

Create a release - named v1 (You may have to accept agreement.), Check box which says Publish this Action to the GitHub Marketplace :

(FYI - you will only see Publish this Action to the GitHub Marketplace if your repo is public.)

We got it published here : https://github.com/marketplace/actions/container-image-existence-checker

Step 6 : Use this newly created GitHub Action in some other repository

I have created a sample repository to test our new custom action : https://github.com/k4kratik/workflow-testing/

Basically this is sample repository where some code is hosted, every time there is a commit on main branch it will trigger a GitHub Action workflow where code will be built and pushed to DockerHub.

Workflow for ECR :

name: ECR CI/CD Workflow for Service

env:
  SERVICE_NAME: my-backend-service

on:
  push:
    branches: [main]

jobs:
  ci:
    name: CI Job
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ap-south-1

      - name: Login to Amazon ECR
        id: login-ecr
        uses: aws-actions/amazon-ecr-login@v2

      - name: Check if the image exists
        uses: k4kratik/container-image-check-custom-action@v4
        id: check_image
        with:
          type: ecr
          container_repo_name: ${{env.SERVICE_NAME}}
          image_tag: ${{ github.sha }}

      - name: Docker build and push to ECR (Skip this step if image exists)
        if: steps.check_image.outputs.image_exists != 'true'
        run: |
          docker build -t ${{ secrets.ECR_REGISTRY }}/${{env.SERVICE_NAME}}:${{ github.sha }} .
          docker push ${{ secrets.ECR_REGISTRY }}/${{env.SERVICE_NAME}}:${{ github.sha }}

Notice that I am using some values as GitHub Secrets.

Workflow for DockerHub

name: Docker Hub CI/CD Workflow for Service

on:
  push:
    branches: [main]

jobs:
  ci:
    name: CI Job
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Check if the image exists
        id: check_image
        uses: k4kratik/container-image-check-custom-action@v4
        with:
          type: dockerhub
          container_repo_name: shorty
          image_tag: ${{ github.sha }}
          dockerhub_username: ${{ secrets.DOCKERHUB_USER }}
          dockerhub_token: ${{ secrets.DOCKERHUB_TOKEN }}

      - name: Login to Docker Hub
        if: steps.check_image.outputs.image_exists != 'true'
        uses: docker/login-action@v3
        with:
          username: ${{ secrets.DOCKERHUB_USER }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

      - name: Docker build and push (Skip this step if image exists)
        if: steps.check_image.outputs.image_exists != 'true'
        run: |
          docker build -t k4kratik/shorty:${{ github.sha }} .
          docker push k4kratik/shorty:${{ github.sha }}

Step 7 : Demo Time : Verify If the custom Action Work

We pushed some code in your sample repo called workflow-testing to start the workflow.

The first workflow run

we can see it executed all the steps.

Let's re-run it.

Re-Run : ECR

Re-Run: Docker Hub

We can see in these re-runs that how beautifully it skipped the build and push image as images were already existing in relevant registries.

That was it.

Thanks for reading. 🌻

Links:

1. GitHub Marketplace Link (to our custom action)

2. GitHub Repo Link (codebase of our custom action)

3. Workflow testing Repo (where we used this custom action)

Let me know if you have any suggestions.

-- Kratik

Today we are going to discuss how can we optimize the flow of storing secrets in Kubernetes and Also, learn to empower our developers to View/Modify secrets deployed in our Kubernetes cluster. We'll also delegate access to our devs based on the GitHub team they belong to.

So let me ask you a basic question, how do you store secrets in your K8s cluster?

If you answer that you are using plain base64 encoded manifest files and managing them manually, then :

Issues in the current approach

1. No easy visibility - No GUI to easily manage secrets

2. No Version Controlling

3. Need to expose K8s creds to devs if you want to share it with them. (You can create and set appropriate RBAC permissions for them though but it can not be used in some use cases)

4. No Support for any 3rd Party Authentication, for example - Using GitHub Team/Org to grant developers a predefined level of access.

Simple Solution - Use Vault with External Secrets

Yes, you heard it right, once this is set, delegating access to users would be a breeze :)

Action Items for us ⚡️

• Install External Secrets Operator

• Install Vault

• Configure Vault to Allow login using GitHub PAT

• Configure Vault auth for External Secrets

• Configure External Secrets to create secrets from Vault

• Create a sample secret in Vault, and see that being created in K8s

Install External Secrets Operator

https://external-secrets.io/latest/introduction/overview/

It has three components - External Secrets Controller, Webhook, Cert-Controller

You can install external secrets quickly with Helm!

https://external-secrets.io/latest/introduction/getting-started/

values.yaml

#? count of the controller pods for HA
replicaCount: 3

#? enable/disable the leader election - at any time, only one active controller is recommended
leaderElect: true

#? how many secrets to update concurrently
concurrent: 3

#? PDB configuration - helpful when doing maintenance tasks
podDisruptionBudget:
  enabled: true
  minAvailable: 1

#? Recommended - Exposes metrics!
serviceMonitor:
  enabled: true
webhook:
  serviceMonitor:
    enabled: true
certController:
  serviceMonitor:
    enabled: true

helm repo add external-secrets https://charts.external-secrets.io

helm upgrade --install external-secrets external-secrets/external-secrets -f values.yaml -n external-secrets --create-namespace

once deployed, It will look something like below :

Note - You can see three deployments of external-secrets but as we have enabled leaderElect only one of them will be active and hold the lease, other replicas will be just waiting if the primary one goes down and then one of the standby replicas will be the leader. Isn't it cool?

Bonus! - More on K8s leader election : https://carlosbecker.com/posts/k8s-leader-election/

Install Vault

Working

The internal working of Vault is quite interesting and a little complex to sum up in few words, we will need another blog to explain it in more detail. If you are completely clueless, then you can read more about this here:

1. https://developer.hashicorp.com/vault/docs/what-is-vault#what-is-vault-1

2. https://developer.hashicorp.com/vault/docs/internals/architecture

To install Vault, We will be using Banzaicloud's Vault Operator aka Bank Vaults!

To deploy anything in the Kubernetes Environment and to ensure its reliability and HA, Operators are becoming de-facto standards.

💡 Do you know : A Kubernetes operator is an application-specific controller that extends the functionality of the Kubernetes API to create, configure, and manage instances of complex applications on behalf of a Kubernetes user.
Check more here : https://operatorframework.io/what/

Install Vault-Operator Using Helm

helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com
helm repo update
helm upgrade --install vault-operator banzaicloud-stable/vault-operator -n vault-operator

Setup RBAC

rbac.yaml - need to apply this to give needed RBAC access to our setup.

 kubectl apply -f vault/rbac.yml -n vault-operator

💡 Explanation - Here, We are creating a Secret vault-sa-token-manual which will hold Kubernetes access token for ServiceAccount named vault (Before Kubernetes v1.24 we didn't have to create the secret manually, see - https://stackoverflow.com/a/72258300). This ServiceAccount has access to read/update pods and all access to secrets.
Note - You can also fine-tune the access by adding only required verbs/resource names/namespaces.

Also, did you notice we are binding a default ClusterRole named system:auth-delegator to Vault ServiceAccount? From the official docs - Allows delegated authentication and authorization checks. The system:auth-delegator ClusterRole has the permissions to call the Token Review API.

Deploying Vault with CRD

Now, as we have powered our cluster with Vault Operator, we can deploy Vault with a simple YAML file.

vault-deploy.yaml

Authentication & Authorization

Here we are authenticating our users using GitHub auth backend. And their access level is defined by the GitHub teams they belong to.

For Example, my account (k4kratik) is assigned the admin policy vault_admin. Similarly, I have created a GitHub Team DEVS_RW in my organization which will grant its members access defined in the policy rw_access_policy and Also created a team DEVS_RO for devs for which I only want to configure read-only access, this team will be assigned with the ro_access_policy.

The above is visualized below :

Config file Explanation :

From Line #1 to #102, I think it's pretty much self-explanatory. Let's Start with Line#106. (where externalConfig starts)

externalConfig.policies

Just like any other setup, we are defining IAM policies here, these will be assigned to users/teams.

• ro_access_policy : only read permissions.

• rw_access_policy : RW permissions. ["read", "list", "update"]

• vault_admin : Just defining * with verbs does not work to grant all permissions. So, we have defined all the admin-level access explicitly.

  Why ? : more info : https://discuss.hashicorp.com/t/vault-admin-policy/39803/2

  admin required access: https://developer.hashicorp.com/vault/tutorials/policies/policies#write-a-policy

externalConfig.auth:kubernetes

• type : As we are setting this up for external-secrets running on Kubernetes, its value should be kubernetes

• path : the identifier for this auth entry in the vault.

• config

  • token_reviewer_jwt : The JWT Token for vault ServiceAccount because it has the system:auth-delegator role, which can help us to use K8s' TokenReview API to validate other JWTs. To fetch its value, run the following command as in the earlier section of the blog, we have already set up the required RBAC.

    Command (copy this and replace with TOKEN_REVIEWER_JWT_TOKEN_HERE )

      kubectl get secret vault-sa-token-manual -n vault-operator -o go-template='{{.data.token}}' | base64 --decode
    

  • kubernetes_ca_cert : PEM encoded CA cert for use by the TLS client used to talk with the Kubernetes API.

    Command

      kubectl get secret vault-sa-token-manual -n vault-operator -o go-template='{{index .data "ca.crt"}}' | base64 --decode
    

  • kubernetes_host : Endpoint for Kubernetes API server. Make sure it's reachable from vault pods.

      kubectl config view --minify --output jsonpath="{.clusters[*].cluster.server}"
    

  • disable_issuer_verification : true - To disable the issuer verification.

• roles - define roles attached to this auth mechanism

  • name: name of the role

  • bound_service_account_names : ["external-secrets"] - We just want to allow external secrets SA, so adding that only.

  • bound_service_account_namespaces ["external-secrets"] - We just want to allow external secrets namespace, so adding that only.

  • token_policies : ro_access_policy - Important! - the name of the policy to attach, Our external-secrets just need read access to read the secrets.

  • token_max_ttl : token expiry time.

externalConfig.auth:github

• type : As we are setting this up for auth via GitHub, its value should be github

• path : the identifier for this auth entry in the vault.

• config

  • organization: name of your GitHub organization. (Replace GITHUB_ORGANIZATION_NAME_HERE with this)

    Note! - You must be part of this organization, or else you will not be able to log in, and you'll get the error : Configuration is not set!

  • token_ttl: lifetime for generated tokens.

map (mapping for GitHub)

• teams (mapping for GitHub Teams with Vault policies)

  • format:

    • TEAM_NAME : POLICY_NAME

  • e.g.

    • DEV_RW: rw_access_policy

    • DEV_RO: ro_access_policy

• users (mapping for GitHub Users with Vault policies)

  • e.g. k4kratik: vault_admin

externalConfig.secrets

• path : secret - path where the Vault will mount this secret engine.

• type : kv - Secret Engine type : Key/Value

• options :

  • version : 2 (recommended)

startupSecrets

• A test secret, which will be created at the startup.

audit

• Audit devices are the components in Vault that collectively keep a detailed log of all requests to Vault.

• The file audit device writes audit logs to a file. https://developer.hashicorp.com/vault/docs/audit/file#configuration

Deploy Vault

kubectl apply -f vault/vault-deploy.yml

it will deploy pods as shown below:

Login To Vault UI

• Let's expose the Vault UI for login

    kubectl port-forward svc/vault -n vault-operator 8200:8200
  

  

• You will be welcome with the below screen :

• Login with GitHub PAT (Personal Access Token)

  • Generate a GitHub PAT with at least read:org permission (guide here)

  • We will use that PAT to login into Vault

    

    Note - Remember the auth policy we created, according to our GitHub username, org or Team, We will be getting relevant access.

    

  • Check the secret engine, you will find our test secret : TEST_PROJECT_ONE/test

    

• Login with Root Token [Not Recommended]

  • Get the root token

      kubectl get secrets vault-unseal-keys -o jsonpath={.data.vault-root} | base64 --decode
    

  • Login with this token as shown below

    

  • Check What we configured :

    • Check the auth we defined - `github` and `k8s-one`

      

      • Let's check in k8s-one auth method, if our role k8s-one-external-secrets-role exists or not

        

This sums up the basic vault authentication and authorization setup + external secrets.

but hold your horses! we still need to create secrets from the Vault using External Secrets. Apologies if the blog seems too long, but trust me, it will be all worth it in the end.

Setup External Secrets

External Secrets have two important types of objects

1. SecretStore(or ClusterSecretStore) - It defines how to access the external API secret provider. (e.g. Vault, AWS Secrets Manager)

2. ExternalSecret (or ClusterExternalSecret) - It describes what data should be fetched, how the data should be transformed and saved as a K8s Secret.

• Let's create a Cluster-scoped Secret Store to connect it with the Vault.

• https://gist.github.com/k4kratik/9b8583d155423490cee0cd823d017da2

    kubectl apply -f external-secrets/secret-store.yml
  

• It will look something like below

  

• Before creating the ExternalSecret, let's create some test secrets in Vault

  

  

• Now, Let's create an ExternalSecret to use our ClusterSecretStore and create a k8s secret using it.

  sample-external-secret.yaml (GitHub)

• https://gist.github.com/k4kratik/1df82010dd5f7263608b03adaecd5069

    k apply -f external-secrets/sample-external-secret.yaml
  

  

• Let's Verify it by checking the secret

  

  

  Voila! we can see our secret is created and we can also see our values from Vault went through 🥳 🎉

  

  I had to decide between these two GIFs to place here, couldn't resist myself from using both 😛 😅

  

Setup Alternatives (or Future Improvements?)

1. Instead of Vault, you can also use AWS Secrets-manager. (Official Provider)

2. Instead of a self-signed cert, you can also use certs created by some reputed CA.

3. You can also use your domain and point the vault there with SSL. (ex. - AWS ALB Ingress w/ ACM SSL Cert)

4. You can create more fine-grained policies as per your use case.

References

• https://developer.hashicorp.com/vault/docs/auth/kubernetes

• https://developer.hashicorp.com/vault/docs/configuration

• https://kubernetes.io/docs/reference/access-authn-authz/authentication/

• https://kubernetes.io/docs/reference/kubernetes-api/authentication-resources/token-review-v1/

• https://learnk8s.io/microservices-authentication-kubernetes

• https://developer.hashicorp.com/vault/tutorials/policies/policies#write-a-policy

• https://developer.hashicorp.com/vault/docs/auth/github

• https://external-secrets.io/latest/api/externalsecret/

• https://external-secrets.io/latest/api/secretstore/

Let's learn how can we do that with AWS ECR.

Bonus! - Along with AWS ECR, we will also explore how we can use Docker Hub to host our helm charts.

Containers and OCI

Containers have transformed the landscape of modern software development and deployment, enabling faster and more reliable shipping of software. Organizations are running containers at a massive scale.

Back when containerization was new, evolving, and getting popular following Docker's release, a large community emerged and started developing new container runtimes to solve their needs.

To solve the issue of incompatibility between different runtimes and image formats, the community formed Open Container Initiative(OCI).

"The Open Container Initiative (OCI) is a lightweight, open governance structure (project), formed under the auspices of the Linux Foundation, for the express purpose of creating open industry standards around container formats and runtimes. The OCI was launched on June 22nd 2015 by Docker*,* CoreOS and other leaders in the container industry."

Currently, it defines three specifications:

OCI Runtime Specification

• A technical specification developed by the Open Container Initiative (OCI) that defines the configuration and lifecycle of containers at the runtime level.

• Includes various aspects like :

  • Container Filesystem

  • Container Lifecycle

  • Container Processes

  • Container Configuration

  • Container Security

• More info here: https://github.com/opencontainers/runtime-spec

OCI Image Specification

• A technical specification developed by the Open Container Initiative (OCI) that defines the format and structure of container images.

• It includes various aspects like :

  • Image Layout

  • Layers

  • Image Manifest

  • Image Configuration

  • Image References

• More info here: https://github.com/opencontainers/image-spec

OCI Distribution Specification

• It defines an API protocol to facilitate and standardize the distribution of content.

• More info here: https://github.com/opencontainers/distribution*-spec*

• "This is designed generically enough to be leveraged as a distribution mechanism for any type of content. The format of uploaded manifests, for example, need not necessarily adhere to the OCI Image Format Specification so long as it references the blobs which comprise a given artifact."

Adoption

• You can find various popular container runtimes that have adopted OCI standards, for example:

  • runC (BTW Docker donated its runtime engine called runc to OCI)

  • crun

  • Kata

  • gVisor

• One of the benefits of having a standard like OCI is that it ensures that containers created using different container runtimes and tools can be easily shared and executed across different environments.

Enough with the theories and blah blah blah...Let's jump right into the action!

Push your Helm chart to ECR

Prerequisites

• Helm CLI (v3.8.0+)

• AWS CLI

Creating a Helm Chart

1. Let's create a helm chart from scratch using helm cli

    helm create my-chart-for-ecr
   

   it's a standard command which will generate a basic helm chart (which basically deploys an NGINX server.)

2. Let's read the content of Chart.yaml

   cat my-chart-for-ecr/Chart.yaml

    apiVersion: v2
    name: my-chart-for-ecr
    description: A Helm chart for Kubernetes
   
    # A chart can be either an 'application' or a 'library' chart.
    #
    # Application charts are a collection of templates that can be packaged into versioned archives
    # to be deployed.
    #
    # Library charts provide useful utilities or functions for the chart developer. They're included as
    # a dependency of application charts to inject those utilities and functions into the rendering
    # pipeline. Library charts do not define any templates and therefore cannot be deployed.
    type: application
   
    # This is the chart version. This version number should be incremented each time you make changes
    # to the chart and its templates, including the app version.
    # Versions are expected to follow Semantic Versioning (https://semver.org/)
    version: 0.1.0
   
    # This is the version number of the application being deployed. This version number should be
    # incremented each time you make changes to the application. Versions are not expected to
    # follow Semantic Versioning. They should reflect the version the application is using.
    # It is recommended to use it with quotes.
    appVersion: "1.16.0"
   

   Note that version of this helm chart is 0.1.0

3. Let's package (packages a chart into a versioned chart archive file) the chart so we can push it.

    helm package ./my-chart-for-ecr/
   

Output :

Preparing ECR

1. Create a Repo in ECR for our use

    aws ecr create-repository \
         --repository-name my-chart-for-ecr \
         --region us-east-1 
    # change region as per your need
   

Getting Creds for ECR Authorization

1. Authenticate your Helm client for our ECR

    aws ecr get-login-password \
         --region us-east-1 | helm registry login \
         --username AWS \
         --password-stdin AWS_ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com
   

Pushing Helm Chart to the ECR

1. Push the chart using helm CLI.

helm push my-chart-for-ecr-0.1.0.tgz oci://AWS_ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/

1. Describe your Helm chart.

aws ecr describe-images \
     --repository-name my-chart-for-ecr \
     --region us-east-1

Output :

let's observe it in AWS Console > ECR

Using Helm Chart hosted on AWS ECR

1. Login to AWS ECR and Authenticate your helm client. (Just like we did in the above step.)

    aws ecr get-login-password \
         --region us-east-1 | helm registry login \
         --username AWS \
         --password-stdin AWS_ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com
   

2. Install the helm chart ( --dry-run to see the output)

    helm install chart-from-ecr oci://AWS_ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/my-chart-for-ecr --version 0.1.0 --dry-run
   

Output:

Pulled: 333333333333.dkr.ecr.us-east-1.amazonaws.com/my-chart-for-ecr:0.1.0
Digest: sha256:5b14b2094ab4581a74ed2bc036993da6cdee10c06cd463027a50f2b227ebf7da
NAME: chart-from-ecr
LAST DEPLOYED: Sat Jun  3 12:04:42 2023
NAMESPACE: crafto
STATUS: pending-install
REVISION: 1
HOOKS:
---
# Source: my-chart-for-ecr/templates/tests/test-connection.yaml
apiVersion: v1
kind: Pod
metadata:
  name: "chart-from-ecr-my-chart-for-ecr-test-connection"
  labels:
    helm.sh/chart: my-chart-for-ecr-0.1.0
    app.kubernetes.io/name: my-chart-for-ecr
    app.kubernetes.io/instance: chart-from-ecr
    app.kubernetes.io/version: "1.16.0"
    app.kubernetes.io/managed-by: Helm
  annotations:
    "helm.sh/hook": test
spec:
  containers:
    - name: wget
      image: busybox
      command: ['wget']
      args: ['chart-from-ecr-my-chart-for-ecr:80']
  restartPolicy: Never
MANIFEST:
---
# Source: my-chart-for-ecr/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: chart-from-ecr-my-chart-for-ecr
  labels:
    helm.sh/chart: my-chart-for-ecr-0.1.0
    app.kubernetes.io/name: my-chart-for-ecr
    app.kubernetes.io/instance: chart-from-ecr
    app.kubernetes.io/version: "1.16.0"
    app.kubernetes.io/managed-by: Helm
---
# Source: my-chart-for-ecr/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: chart-from-ecr-my-chart-for-ecr
  labels:
    helm.sh/chart: my-chart-for-ecr-0.1.0
    app.kubernetes.io/name: my-chart-for-ecr
    app.kubernetes.io/instance: chart-from-ecr
    app.kubernetes.io/version: "1.16.0"
    app.kubernetes.io/managed-by: Helm
spec:
  type: ClusterIP
  ports:
    - port: 80
      targetPort: http
      protocol: TCP
      name: http
  selector:
    app.kubernetes.io/name: my-chart-for-ecr
    app.kubernetes.io/instance: chart-from-ecr
---
# Source: my-chart-for-ecr/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: chart-from-ecr-my-chart-for-ecr
  labels:
    helm.sh/chart: my-chart-for-ecr-0.1.0
    app.kubernetes.io/name: my-chart-for-ecr
    app.kubernetes.io/instance: chart-from-ecr
    app.kubernetes.io/version: "1.16.0"
    app.kubernetes.io/managed-by: Helm
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: my-chart-for-ecr
      app.kubernetes.io/instance: chart-from-ecr
  template:
    metadata:
      labels:
        app.kubernetes.io/name: my-chart-for-ecr
        app.kubernetes.io/instance: chart-from-ecr
    spec:
      serviceAccountName: chart-from-ecr-my-chart-for-ecr
      securityContext:
        {}
      containers:
        - name: my-chart-for-ecr
          securityContext:
            {}
          image: "nginx:1.16.0"
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
          livenessProbe:
            httpGet:
              path: /
              port: http
          readinessProbe:
            httpGet:
              path: /
              port: http
          resources:
            {}

NOTES:
1. Get the application URL by running these commands:
  export POD_NAME=$(kubectl get pods --namespace crafto -l "app.kubernetes.io/name=my-chart-for-ecr,app.kubernetes.io/instance=chart-from-ecr" -o jsonpath="{.items[0].metadata.name}")
  export CONTAINER_PORT=$(kubectl get pod --namespace crafto $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
  echo "Visit http://127.0.0.1:8080 to use your application"
  kubectl --namespace crafto port-forward $POD_NAME 8080:$CONTAINER_PORT

So Yes! 🎉 🎉 🎉 We were able to see all the manifest this helm chart was going to install.

[Bonus] Push your Helm chart to DockerHub

1. Let's continue from the part where we ran the helm package

2. Log in to Docker Hub using your Helm client.

    helm registry login registry-1.docker.io -u DOCKERHUB_USERNAME
   

3. Push your chart to a DockerHub.

    helm push my-chart-for-ecr-0.1.0.tgz oci://registry-1.docker.io/DOCKERHUB_USERNAME
   

4. This will create a docker repository called my-chart-for-ecr in your DockerHub account. (Image would be DOCKERHUB_USERNAME/my-chart-for-ecr)

   

Note - we are able to push/host our helm charts to our same old container registries because Helm Chart is another OCI artifact! (see reference links for more info)

Final words

This was just a way to store your helm charts on ECR / Docker Hub or any other OCI-compliant repositories. You can version control this (either with your services or in a separate repo) and also set up some CI/CD pipeline with some automated testing(maybe using k3s/kind to spin a temp local k8s cluster and to test deploying your helm chart there) to achieve a complete lifecycle of your helm chart, just like a regular software project.

Thank you so much for being here! Have a nice one and see you in the next blog.

References

1. https://docs.docker.com/docker-hub/oci-artifacts/#using-oci-artifacts-with-docker-hub

2. https://helm.sh/docs/topics/registries/#using-an-oci-based-registry

3. https://docs.aws.amazon.com/AmazonECR/latest/userguide/push-oci-artifact.html

4. https://docs.aws.amazon.com/AmazonECR/latest/userguide/ECR_on_EKS.html#using-helm-charts-eks

We all are addicted to our phones, checking notifications, getting distracted, and end up replying to some of those notifications and the cycle repeats!

There is no denying that notifications are an integral part of our life and the apps we all have on our phone constantly bombards us with numerous notifications.

Definitely, they help us by keeping us informed and connected but at times, it can bring anxiety, stress, and feeling of being overwhelmed.

Through this blog, I hope to offer practical tips, tools, and resources to help you reduce the impact of notifications on your life, improve your productivity and focus, and achieve a better balance between your digital and offline worlds.

What is it?

Notification anxiety is a feeling of stress, worry, or pressure that arises from constantly checking and responding to notifications from electronic devices such as smartphones, laptops, or tablets.

The constant barrage of notifications can create a sense of urgency and a fear of missing out (FOMO), leading to a compulsive need to check and respond to notifications even if they are not critical or important.

This can result in a range of negative emotions such as anxiety, distraction, and decreased productivity. It can also lead to a lack of focus, poor sleep, and other negative impacts on mental health and well-being.

Solutions?

1. Differentiate Notifications with sounds

   You can set different notification sounds for different purposes.

   For Example - in WhatsApp, you can set custom notification tones for specific contacts. You can implement this to isolate notification sounds for your favorite people (parents, siblings, partner, etc.) - Sometimes we are excited about some people, and from the alert sound if we are able to figure out about the sender then I think it would keep you calmer.

   Also, For different apps, set different notification tones, so for the frequently used apps, you'll have an instant idea when you receive a notification and hear the sound. That way you can decide quickly if you should check your phone or not.

2. Save notifications for later to reduce noise

   I use an app on my Android called Notisave (Play Store link) - what it does is simple but super efficient for me.

   So what we will be doing is transferring all our notifications from Notification Panel to this app so we can read them later.

   It also saves notifications so we won't be missing anything + it supports the grouping of notifications on the basis of apps so we can make some groups for the office, social media, games, delivery apps, etc.

   To get started, It's quite easy-peasy.

   - First I need to select some apps for which I want to block the notifications.

   

   - From now, every new notification for these blocked apps, won't bother you from the notification bar, instead, it will go to the Notisave for you to read it later.

   

   So these are some of the ways I use to keep my mind calm and relaxed.

Within a short span of time, you will see how many notifications get bombarded to you so frequently but as we don't have any count or estimate of the number we ignore the impact of this on us.

For Example - You can see I got around 9.2k notifications which is the number of times my phone would have vibrated/alerted/distracted me.

In conclusion, while it's natural to feel anxious about notifications, it's important to remember that they don't define us. By setting boundaries and practicing mindfulness, we can learn to manage our notification anxiety and live more peaceful, present lives.

Hope this helps you. See you in the next one!

Thanks for reading 🌻

Today, we are going to learn how we can build docker images for containers that will be compatible with the amd64 and arm64 CPU architecture types.

I chose these two CPU types because amd64 is already widely used with our Linux/Windows/Mac Computers and arm64 is getting popular nowadays, for example - new Mac computers are coming with Apple's in-house M1 processor which is arm64 based and if you talk about Cloud, AWS is offering various VMs which are powered by Graviton processors which are again arm64 based, So if your container images are able to run on these two kinds of CPUs, you will be able to utilize most of it.

Along with that what I want is that we understand each and every word that we will be using.

What is CPU?

CPU is short for Central Processing Unit, which is generally called processor(or microprocessor).

Also called the heart/brain of the computer. It performs all arithmetic and logical operations.

Inside the CPU, there are transistors, which control the flow of electricity. You know, a CPU can perform millions of instructions per second but can only do one instruction at any given point in time.

Then we have multi-core CPUs, each core is a sort of separate CPU, which can help the CPU to do multiple tasks simultaneously.

Read these for more details on:

1. What is CPU - https://www.freecodecamp.org/news/what-is-cpu-meaning-definition-and-what-cpu-stands-for/
2. How does a CPU work - https://www.freecodecamp.org/news/how-does-a-cpu-work/

What are ISA & Micro-architecture?

An instruction set is a group of commands for a central processing unit (CPU) in machine language.

Now, ISA (instruction set architecture) is a part of processor architecture. It works as an interface(communication rules) between hardware and software. It's a group of commands which are implemented in the processor's circuitry also called microarchitecture.

Few examples of what ISA defines:

1. How binary instructions are formatted
2. How RAM/ROM is accessed
3. Maximum bit length for all instructions

It helps us to separate hardware and software development without worrying about the other.

ISA has two major classifications :

1. CISC - Complex instruction set computer

It includes many specialized instructions. It will use fewer instructions but each instruction will take multiple machine cycles.

1. RISC - Reduced instruction set computer

It uses a smaller, optimized set of generalized, simple instructions. It will use more number of instructions but each instruction will take one clock cycle.

What is amd64?

Before that let's see what is 32-bit/64-bit architecture - It simply defines that in a single instruction cycle (fetch-decode-execute) how much data a microprocessor can process.

32-bit means it can process 4 bytes of data and for 64-bit it is 8 bytes.

x86 is a family of instruction set architectures (ISA) for computer processors initially developed by Intel. (86 comes from the microprocessor name which was 8086 )

AMD took this x86 architecture which could only support 32-bit and then added 64-bit computing capabilities, that's why it is known as amd64. Also referred to as, Intel 64, x86-64, x64.

What is arm64, then?

Let's understand what are ARM Processors first.

ARM stands for Advance RISC Machine - It is an ISA developed by a company called ARM Ltd.

So, ARM Licenses the architecture to other companies so they can develop their custom processors. Example - Apple's Silicon processors.

ARM Processors are generally used in smartphones, tablets, IoT devices, etc. due to their low power consumption and high-performance nature.

So ARM64 is the 64-Bit version of ARM processors(aka ARMv8-A).

Back to Docker Image Building

So now as we have some sort of idea about CPUs, we can guess that when we build docker images on a machine with specific architecture(amd64 or arm64), it is compiled in a way that it can run only on that specific type of platforms.

So what we will try to solve here is to build universal docker images, which can run on these platforms, without us worrying about the rebuilding of images.

Hands-On: Running Docker Image cross-platform

I built a docker image on my Macbook with an M1 Pro chip (arm64) and then tried to run it on an EC2 with an amd64 processor, this was the output :

WARNING: The requested image's platform (linux/arm64/v8) does not match the detected host platform (linux/amd64) and no specific platform was requested.

exec /docker-entrypoint.sh: exec format error

Docker Buildx for the rescue!!!

docker buildx is a CLI plugin that enhances the existing docker build capabilities. (which uses another open-source project called buildkit.)

BuildKit is a toolkit for converting source code to build artifacts in an efficient, expressive, and repeatable manner.

If you want to know more about moby/buildkit you can read this Introductory blog post: https://blog.mobyproject.org/introducing-buildkit-17e056cc5317

You can refer to this Installation guide on how to install buildx into your system, It comes pre-installed with Docker Desktop for Windows/macOS.

Let's use buildx to build our docker images

Step-1: List existing builders

docker buildx ls

Step-2: Create a new builder to support multi-architecture

docker buildx create --name mybuilder --driver docker-container --use --bootstrap

We are instructing docker to create a new buildx builder named mybuilder which will use the docker-container driver (which supports multi-architecture builds), also instructing it to use this new builder, also instructing this to boot this builder which means it will pull necessary docker image to run this builder and run the builder container. (as our driver is docker-container)

Step-3: Create a Sample Dockerfile

Consider this simple Dockerfile, which intends to run an NGINX server

Dockerfile

FROM nginx:stable-alpine

LABEL MAINTAINER="KRATIK JAIN"

COPY --chmod=0755 custom-web-page.sh /docker-entrypoint.d/

custom-web-page.sh

#!/bin/sh
uname -a > /usr/share/nginx/html/index.html

what this script does is replaces default HTML with the output of uname -a, which prints out the system info like kernel info, Architecture, etc.

Step-4: Build the container image using the buildx

docker buildx build --platform linux/amd64,linux/arm64,linux/arm/v7 -t k4kratik/multi-platform:v1 --push .

Here you can notice the flag --platform & --push: here you are specifying the platforms for which you want to build the image, and you also wish to push the image once done. That's amazing as build and push commands are in just one command. Output also seems to be more verbose.

Step-5: Run this docker image on various platforms

Step-5.1: Running on EC2 with amd64 arch Ubuntu machine

docker run --rm -itd -p 30000:80 k4kratik/multi-platform:v1

Now hit the server endpoint and check the response

curl localhost:30000

Output

Linux e14cc6b4eb86 5.15.0-1011-aws #14-Ubuntu SMP Wed Jun 1 20:54:22 UTC 2022 x86_64 Linux

Step-5.2: Running on Macbook Pro with arm64 arch M1 Chip

docker run --rm -itd -p 30000:80 k4kratik/multi-platform:v1

Now hit the server endpoint and check the response

curl localhost:30000

Output

Linux 595316994de1 5.10.104-linuxkit #1 SMP PREEMPT Thu Mar 17 17:05:54 UTC 2022 aarch64 Linux

Notice the output from both of the machines and observe the second last string, for EC2, it was: x86_64 and for Macbook it was aarch64 which means it pulled the image according to its CPU architecture.

Bonus: Inspect Images

Using the below command you can inspect the image to know more details about it and to know the platforms available:

docker buildx imagetools inspect k4kratik/multi-platform:v1

Output

Name:      docker.io/k4kratik/multi-platform:v1
MediaType: application/vnd.docker.distribution.manifest.list.v2+json
Digest:    sha256:c534475bb099316622a719407a17e2da21d79d147e9f607b4a1262c2a6f3bb1a

Manifests: 
  Name:      docker.io/k4kratik/multi-platform:v1@sha256:1d321fb1d14afe61e6f5c9ea2fc7e1b20bddd347c98a1b09f9367a6c446898b9
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/amd64

  Name:      docker.io/k4kratik/multi-platform:v1@sha256:47b6ffc133acac83fc70dd176606de8b033ae735d386160e6d68ee7d472119b3
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm64

  Name:      docker.io/k4kratik/multi-platform:v1@sha256:89f7b675b538e2aafe060dda535a48311bd090c63d0e3bb0184a440a32079f7d
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm/v7

Bonus: Pull/Run Image of a specific platform

You can use the following flag to pull or run an image of a specific platform

docker pull --platform linux/arm64 alpine:latest

docker run -itd --platform linux/arm64 nginx-alpine:latest

Well, it was just scratching the surface. I hope it was helpful enough for you to get started on your feet so you can research and dig down further.

See you in the next blog! :)

Update: I can not express my happiness when I got to know that many people are reacting to this blog and this blog got featured on Hashnode

Thank you so much 🫶🏻 everyone! You just made my day 🥂

🛣

From our past blog Best Logging Solution for Docker [Basic Version],

We saw how to make logging a piece of cake using Loki & Grafana.

But I think there is one minor issue If your servers are on the public internet and you have enabled port forwarding for your use case, then anyone using your Loki URL, can add it as a data source in their Grafana Installation and will be able to see all of your sensitive logs, which is the last thing on earth we want.

It's simple right?

When you run Loki, out of the box, it does not come with any kind of additional security layer. So basically anyone who can access that URL, can access and see your logs, simple as that!

Loki Official Documentation says that Grafana Loki does not come with any included authentication layer.

https://grafana.com/docs/loki/latest/operations/authentication/

The Solution(s)

Now as I am thinking about it, feels like we can use some simple tactics to enhance security. Those are :

1. If not required, don't launch your servers on the public subnet. (hence reducing the possibility of someone connecting from outside your network)

2. Do not use port-forwarding, instead use DNS which is supported by the docker network. (for example, instead of http://Server_IP:3100 we can use http://loki:3100)

3. Use some firewall/SecurityGroups to whitelist services/users to allow limited access to required ports.

4. Advance - You can use a reverse proxy(NGINX) with Basic Auth. So even if someone has connectivity to the endpoint, they need to enter basic auth credentials to connect :)

The Implementation of the advanced solution (Just for fun!)

Step-1: Keep your environment ready

I am assuming, you already have a docker environment ready that is already using Loki without any auth method.

You can take a reference from my previous blog as well.

Do you know what I did?

I referred to this official docker-compose file to get started in a few seconds.

version: "3"

networks:
  loki:

services:
  loki:
    image: grafana/loki:2.6.1
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      - loki

  promtail:
    image: grafana/promtail:2.6.1
    volumes:
      - /var/log:/var/log
    command: -config.file=/etc/promtail/config.yml
    networks:
      - loki

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    networks:
      - loki

Step-2: Prepare the NGINX reverse proxy with Basic Auth

Luckily I stumbled upon this cool open-source project which does the job for you :

So now what? How to use this?

You just have to give the required env vars and it will launch NGINX accordingly.

Those are:

1. BASIC_AUTH_USERNAME & BASIC_AUTH_PASSWORD => Basic Auth Creds
2. PROXY_PASS => where you want to forward your request from NGINX reverse proxy
3. PORT => The port, on which you want NGINX to listen.

Note: Here we will add our Loki URL in the PROXY_PASS section so it can forward our request to Loki.

Step-3: Combining all the pieces together

So Now, as we know how can we secure our Loki Installation, let's add this reverse proxy in our docker-compose.yaml file.

version: "3"

networks:
  loki:

services:
  loki:
    image: grafana/loki:2.6.1
    ports:
      - 3100
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      - loki

  promtail:
    image: grafana/promtail:2.6.1
    volumes:
      - /var/log:/var/log
    command: -config.file=/etc/promtail/config.yml
    networks:
      - loki

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    networks:
      - loki

  nginx:
    image: quay.io/dtan4/nginx-basic-auth-proxy
    environment:
      - BASIC_AUTH_USERNAME=loki_user
      - BASIC_AUTH_PASSWORD=loki_super_secret_password
      - PROXY_PASS=http://loki:3100
      - PORT=3200
      - SERVER_NAME=_
    ports:
      - 3200:3200
      - 8090:8090
    networks:
      - loki

Here I also disabled port-forwarding for Loki, because that is not required. Also, notice that in the PROXY_PASS env var, I gave loki:3100 which is the service name in the docker-compose file and is internally resolvable by the containers in the same docker network.

Here, we are running this NGINX server on port 3200 + we did port forwarding on the same port, so we can use localhost:3200 to connect to the NGINX server.

Also, port 8090 is used for NGINX metrics. You can check that on -> :8090/nginx_status

Step-4: See it in action!

Now, up this docker-compose file and see what happens at the Grafana end.

So, let's try to add a data source, just like before.

Trial-1: On old port 3100

As expected, we have disabled the port forwarding, it should fail.

Trial-2: With NGINX Port (w/o creds)

Again, this also fails as above.

Just for curiosity, let's curl and see.

Oh ok, let's try with the creds which we supplied to NGINX in docker-compose.

      - BASIC_AUTH_USERNAME=loki_user
      - BASIC_AUTH_PASSWORD=loki_super_secret_password

Trial-3: NGINX with Basic Auth Creds

Now using the same endpoint, Enable the Basic Auth, and add those basic auth credentials

hmm...

Voilà! - we were able to connect to our Loki Data source with an additional layer of security i.e. Basic Auth.

I know most of you are using Kubernetes out there and only a small set of users can relate to this(and implement it) but it is a really small fun exercise on how to use NGINX reverse proxy, setup Basic Auth, docker-compose(or docker swarm) inter-service communication and securing Loki installation. 😄

and the other reason to be happy is:

That was it! Let me know what you have to say in the comment box.

Thanks for reading :)

Reads:

1. https://grafana.com/docs/loki/latest/operations/authentication/
2. https://docs.nginx.com/nginx/admin-guide/security-controls/configuring-http-basic-authentication/
3. bash script used in the NGINX docker image: https://github.com/dtan4/nginx-basic-auth-proxy/blob/master/files/run.sh
4. Also, the Dockerfile: https://github.com/dtan4/nginx-basic-auth-proxy/blob/master/Dockerfile

When you are running an EKS cluster, you may have encountered some situations where you wanted to authorize your PODs to access AWS services. For Ex - Read/Write a file to/from S3, Trigger lambda functions, etc.

So How do you do it?

There are multiple ways to do it -

1. Kiam

2. kube2iam

3. Associate an IAM role with a Kubernetes service account

In this blog, we will be focusing on point #3

Out of Curiosity, Do you know what access your Pods get by default in an EKS Cluster?

I assume you don't know, so let's check :

First of all, Create a pod that has AWS CLI Installed so you can run the aws sts get-caller-identity command. For that apply below YAML file :

apiVersion: v1
kind: Pod
metadata:
  name: aws-cli
spec:
  containers:
    - image: amazon/aws-cli:latest
      name: aws-cli
      command: ["sleep", "36000"]

kubectl apply -f aws-cli-pod.yml

Now, let's exec into that pod and run the aws cli command to know what IAM identity is currently logged in.

kubectl exec -it aws-cli -- aws sts get-caller-identity

Oh Okay! Do you know what role this is?

Let's observe the UserId

According to official AWS Documentation here :

This means this is a role assigned to an EC2. Digging it further (documentation reference), I got to know that the first part in the UserId (AROASPNKLQGHF35FHTBBB:i-0f066b892b87973af) is a unique identifier for a role. AROA represents that this particular identity is a role.

Ok, So the next question, From where this role is getting assigned and being assumed?

Answer - When we create EKS cluster/Node Groups, either using CloudFormation, Terraform, or AWS Console, we have to assign a role to the EKS Worker Nodes, and as our Pods run on those nodes they inherit(or assume) those permissions of that role, which assigned to EKS Worker nodes.

AWS CloudFormation Reference

AWS Console Reference (check image below)

Terraform Reference

You can check more about the purpose of the EKS Node Role here.

Now as we saw earlier, by default, every pod will assume the IAM role which is assigned to the node, on which the pod is running.

Coming back to the point, I can see a way now using which any Pod can have required IAM access: I just need to add those policies to the EKS Node IAM Role, right?

But wait, it can work for you but NOT A GOOD PRACTICE!!! (All of your pods will be getting that access, even though they don't need that access. [ ⚠️ Violation of the Principle of least privilege])

🤔 Then what should we do now to assign pod a specific role and we won't have to mess with the IAM role of EKS Nodes?

Service Account for the Rescue 😉

1. Create an OIDC provider for your EKS cluster

I am using eksctl(link) utility here for managing and performing operations on my EKS Cluster.

eksctl utils associate-iam-oidc-provider --cluster NAME_OF_YOUR_CLUSTER --approve

Sidenote - also check this AWS article which also includes one more step to check for existing OIDC Issuer for your cluster.

*Open image in New Tab if not clear!

2. Create the role you want to attach to your Pod

2.1 Create the JSON policy document

IAM-Policy-for-listing-buckets.json

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListAllMyBuckets"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}

2.2 Create the IAM policy

Now fire up the AWS CLI Command to create your policy :

aws iam create-policy --policy-name s3-list-buckets-policy --policy-document file://IAM-Policy-for-listing-buckets.json

You will see something like below output :

Copy the ARN of this new policy here.

2.3 Create an IAM role for a service account

let's run eksctl command as below and create IAM role and service account

eksctl create iamserviceaccount \
    --name NAME_OF_YOUR_SERVICE_ACCOUNT \
    --namespace NAME_OF_YOUR_K8S_NAMESPACE \
    --cluster NAME_OF_EKS_CLUSTER \
    --role-name NAME_OF_YOUR_IAM_ROLE \
    --attach-policy-arn ARN_OF_POLICY_YOU_JUST_CREATED \
    --approve \
    --override-existing-serviceaccounts

Output

This will create a CloudFormation Stack (which will create an IAM role) and our service account which we defined in the command args.

I encourage you all to go and check the above CF template, check how the ROLE was created, and most importantly: How the role's AssumeRolePolicyDocument is defined.

To check what happened in our EKS Cluster :

kubectl get serviceaccounts

kubectl get serviceaccounts s3-service-account -o yaml

3. Associate the service account with your Pod so it can have access to your Role

Do you remember, at the start of our blog, we created an aws-cli pod, lets tweak it a bit and see what happens now :

delete the old pod

kubectl delete po aws-cli

Update the file aws-cli-pod.yml as follows :

apiVersion: v1
kind: Pod
metadata:
  name: aws-cli
spec:
  containers:
    - image: amazon/aws-cli:latest
      name: aws-cli
      command: ["sleep", "36000"]
  serviceAccountName: s3-service-account

kubectl apply -f aws-cli-pod.yml

Now check the current logged in IAM entity in our POD :

Wait, this time it is different, Also our new role is there and it is being assumed by our Pod.

Let's verify the access by listing out all S3 buckets :

Woohoo! That works!!!

wait let me add more excitement

What do you think about this whole IAM & Service Account thing? Did you learn something new?

Let me know in the comments. Thanks for reading.

Have you heard about the Principle of least privilege? It dictates - A subject should be given only those privileges needed for it to complete its task. If a subject does not need an access right, the subject should not have that right.

So we should create users as per the requirements and assign them the minimum permissions to function properly, right? but how do we do it?

Just to be clear, Kubernetes does not have the support for users natively. And from the documentation - "Kubernetes does not have objects which represent normal user accounts. Normal users cannot be added to a cluster through an API call."

Ways to set up users in K8s

There are multiple ways to create/manage users(basically the AUTHENTICATION part) in Kubernetes, for example -

1. Bearer Token - Read K8s doc here for more details.
2. Basic Auth (Username/Password) - For this, you need to start kube-apiserver with the --basic-auth-file=name_of_auth_file argument. This file contains list of users in password, user-name, user-id format.
3. Auth using x509 certs - We can leverage Client-Side x509 certs to authenticate ourselves. And using RBAC we can authorize/restrict our users' access. FYI - this is what today's blog is all about.

Note - There are a bunch of other methods also available for the authentication in a k8s cluster. Check this for the same.

Flow

1. Generate a set of x509 certificate/key for a user.
2. Update the current kubeconfig file to add new user details.
3. Create an appropriate role for the user. (Role/ClusterRole)
4. Attach that role to the user. (RoleBinding/ClusterRoleBinding)

Requirements

1. Working K8s Cluster (for sake of our blog post, I used (k3s) to quickly launch a fast & lightweight k8s cluster, must check!)
2. Certificate Authority Credentials (CA Cert and CA Key to approve CSR)
3. Minimal access to apply Role & Role Bindings.

Tutorial - Create a user in K8s

We will stick to the flow which we defined earlier. Suppose we want to create a user for Kratik Jain which is a newly joined Intern in XYZ Technologies. We don't want him to make any changes in our cluster which can mess things up, so we will be giving him read-only access. Let's Start -

Step-1 Generare x509 Certs (or Credentials)

We will be using OpenSSL for the generation and analysis of the certificates.

1.1 Create a private key for our user.

Private keys are meant to be kept private and used in encryption, so only ones who have this private key can decrypt the data. but here, it is used only to identify a user.

openssl genrsa -out kratik.key 2048

1.2 Create a certificate sign request

We will create a CSR for our user with our private key. Basically, it's encoded info requesting a certificate. Later we will use our Certificate Authority's credentials to create a valid certificate. Read more about CSRs here.

A CSR contains various properties but for our use case, one property is most important - Common Name aka CN.

Remember! using this common name, our K8s cluster will identify us. Whatever CN we define here, becomes our username in K8s.

So let's fire the below command to create a CSR -

openssl req -new -key kratik.key -out kratik.csr -subj "/CN=kratik/O=XYZ-Technologies"

1.3 Use CA Credentials and approve the CSR to generate the final Certificate

!Important - Locate your CA Credentials - the CA Cert and CA Key. Depending on your installation procedure, it may be stored at different locations.

For example, If you are using -

1. K3s, you'll find your creds at /var/lib/rancher/k3s/server/tls/client-ca.crt & /var/lib/rancher/k3s/server/tls/client-ca.key

2. Kind - As kind is Kubernetes in docker - you need to find the control-plane Docker container and there in the path /etc/kubernetes/pki/ you will find the ca.crt & ca.key. Tip - Use docker cp command to copy those files to your local machine.

3. Kubeadm - you will find your CA creds in /etc/kubernetes/pki/ directory.

Assuming you have these creds in the current working directory as ca.crt & ca.key, fire the below command to use your CA Creds and approve the CSR, and generate the certificates.

openssl x509 -req -in kratik.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out kratik.crt -days 365

Awesome! your key and cert are ready to be used!

Step-2 Update Kubeconfig to use new user creds

kubectl uses kubeconfig files to organize information about clusters, users, namespaces, and authentication mechanisms.

2.1 Adding user credentials

kubectl config set-credentials kratik --client-certificate=kratik.crt  --client-key=kratik.key

2.2 Setting a new Context

kubectl config set-context kratik-context --cluster=default --user=kratik

Context in Kubeconfig file - From the official docs, "A context element in a kubeconfig file is used to group access parameters under a convenient name. Each context has three parameters: cluster, namespace, and user. By default, the kubectl command-line tool uses parameters from the current context to communicate with the cluster."

2.3 Testing kubectl commands to verify that it detects our user and fails

this will fail because we haven't assigned any role to it yet. (Which is EXPECTED!)

Notice that I am using a newly created context using the --context flag in the kubectl command. (BTW I am using k3s and k3 in my terminal is an alias for that )

Step-3 Create a Role that has fine-tuned permissions outlined

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: role-for-read-only-access
rules:
  - apiGroups: ["extensions", "apps", ""] # "" indicates the core API group
    resources: ["*"]
    verbs: ["get", "watch", "list"]

kubectl apply -f ro-role.yaml

Step-4 Attach this RO role to our user

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: read-only-role-binding
subjects:
  - kind: User
    name: kratik
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: role-for-read-only-access
  apiGroup: rbac.authorization.k8s.io

kubectl apply -f ro-role-binding.yaml

Step-5 Testing the access

Fire the same command which we fired previously to check if the error is gone or not.

kubectl --context=kratik-context get po -A

Also, Let's try deleting the pod, let's see what happens 🤔

So this worked out finally as we wanted!!!

Now you can also tweak and fine-tune your Cluster role to give more granular access. I wanted to create something which is not bounded to some namespace but is usable throughout the cluster, so I chose the ClusterRole.

References

1. Wikipedia - Principle of least privilege
2. Kubernetes - RBAC
3. Kubeconfig file

I am assuming you know about and have some experience in Kubernetes & ConfigMaps.

Anyways I will try to explain concepts as easy as possible 😀

This is how I used to feel when I started :

So Don't worry, Let's start improving ourselves, no matter how much but consistently 👍

When We use ConfigMaps?

When you want to supply environment-specific config values like env vars.

Using ConfigMaps, You can also mount your config as files in a particular directory which can be later read by our application.

So, In Short, using ConfigMaps, you can supply configs by :

1. Setting up env vars
2. Mounting the config values as a file (aka data volume).

Problem Statement

You updated a ConfigMap, Now you want to propagate this updated change to all existing running Pods.

Options

Option #1

You can delete existing pods or restart the deployment rollout.

But You know what this is not feasible in some cases where you don't want any downtime or you simply just don't want to delete any running Pods.

Note - if you are using ConfigMaps to set env vars then this is the only option you have.

Option #2

This option is only applicable if you have mounted your ConfigMap as a file in your Pod. Also if you are using ConfigMap as a subPath volume, this option won't be applicable.

So Let's discuss this option in detail.

The Option #2

This is not something which I have invented, rather I read this on the Official K8s docs page here

Actually, there are two ways you can achieve this,

1. Use Annotation

According to the docs I mentioned above, You can update the pod's annotations which will trigger a refresh and your ConfigMap data volume will be populated with the latest data. Simple! 👀

2. Wait for Kubelet to refresh ConfigMap data

If configured, Kubelet also tries to refresh the ConfigMaps for pods periodically to keep things in sync. This interval is called ConfigMap Cache TTL

Extra - There is one property called ConfigMapAndSecretChangeDetectionStrategy which you can configure when you start Kubelet. Possible values we can pass to this are : Get, Cache & Watch (Default) This property defines how we want Kubelet to detect and propagate ConfigMap and Secret changes.

More info : ConfigMaps | Kubernetes.

I hope this quick & short blog made sense :)

Terminology Used

Kubernetes - Kubernetes is an open-source container orchestration system for automating software deployment, scaling, and management.

Containers - A container is a standard unit of software that packages up code and all its dependencies so the application runs quickly and reliably from one computing environment to another.

Pods - Pods are the smallest, most basic deployable objects in Kubernetes. Pods contain one or more containers.

ConfigMaps - A ConfigMap allows you to decouple environment-specific configuration (mostly environment variables) from your container images so that your applications are easily portable.

References

1. https://en.wikipedia.org/wiki/Kubernetes
2. https://www.docker.com/resources/what-container
3. https://kubernetes.io/docs/concepts/configuration/configmap/
4. https://kind.sigs.k8s.io/docs/user/quick-start/
5. https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#mounted-configmaps-are-updated-automatically
6. https://kubernetes.io/docs/concepts/configuration/configmap/#mounted-configmaps-are-updated-automatically

When I was writing Dockerfiles initially, I found CMD and ENTRYPOINT very confusing to me. I assume there will be folks out there who are still not so sure about these two syntaxes and their use cases.

So let's try to understand the concept in depth for once and all.

The best way to understand a tool or technology is to go to their official documentation and start digging.

In our case, you can follow this link of Docker Official Documentation - https://docs.docker.com/engine/reference/builder/#entrypoint

This is what you will see :

But did you notice the terms - exec form and shell form?

Exec and Shell form

What is exec form?

In this form, you have to pass the executable and all required params as JSON Array.

The exec form makes it possible to avoid shell string munging and to RUN commands using a base image that does not contain the specified shell executable.

Note that as it is a JSON array, make sure you use double quotes, not single quotes.

Example :

ENTRYPOINT ["/usr/sbin/apache2ctl", "-D", "FOREGROUND"]

Here

• We are avoiding any shell parsing and directly using the executable which is apache2ctl
• We are not invoking any shell to run the binary of our program.

This form is recommended over shell form as shell form can cause troubles which make our application inside the container not receive signals properly.

Special Case - In exec form, docker can substitute env variable if set by ENV instruction in Dockerfile.

FROM alpine:latest

ENV VERSION=v1.2.0

RUN ["echo", "$VERSION"]

This will echo v1.2.0 as docker does the substitution for us.

What is shell form?

This is a fairly simple form which we can use, as We don't have to follow any special notation - we just have to write exactly as we run commands on our terminals with shell in our day-to-day life.

Example:

ENTRYPOINT apache2ctl -D FOREGROUND

Here

• Shell Processing and Environment Variable parsing can happen.
• Our binary is not executed directly, first, a shell is invoked with /bin/sh -c and then our executable is started in that shell.
• You can use a \ (backslash) to continue a single RUN instruction onto the next line.
• The SHELL instruction allows the default shell (/bin/sh) used for the shell form of commands to be overridden.

Difference/Comparison

Basic comparison from the documentation : Unlike the shell form, the exec form does not invoke a command shell. This means that normal shell processing does not happen. For example, RUN [ "echo", "$HOME" ] will not do variable substitution on $HOME. If you want shell processing then either use the shell form or execute a shell directly, for example: RUN [ "sh", "-c", "echo $HOME"]

When to Use?

Exec Form

1. When You don't want any shell features like env var substitution, I/O Redirection, piping, chaining multiple commands, etc.
2. When You want to forward process signals to child processes. Ex - Pressing Ctrl + C(SIGINT) to stop the process.
3. Recommended for CMD & ENTRYPOINT in Dockerfile.

Shell Form

1. When you want to use shell features as mentioned above. (Piping, Command chaining using a backslash, I/O Redirection, Shell Variables substitution, etc. )
2. It is extremely useful with RUN instructions in Dockerfile.

As we cleared some basic concepts, Now, Back to the original question: CMD vs ENTRYPOINT?

CMD

• It sets the default parameter for a container.
• It can be overridden easily while running a container with the docker run command.
• If you don't provide any executable in the CMD, it will pass itself as the parameter to the entrypoint.

ENTRYPOINT

• When you want to use your container as executable. Example - See below Dockerfile

FROM alpine

ENTRYPOINT ["echo", "Hello"]

CMD ["World!"]

See in Terminal: (running above Docker Image)

Here you can see that I am passing some parameters and My container is behaving according to that!

• By implementing ENTRYPOINT, we are implicitly specifying that this container is made for some specific use case.

• When we run the container with docker run, all the params are appended to ENTRYPOINT

How to Use CMD/ENTRYPOINT with :

docker run command

CMD

docker run -it <NAME_OF_DOCKER_IMAGE> param1

Here param1 is equivalent to CMD instruction.

ENTRYPOINT

docker run -it --entrypoint /path/of/entrypoint/executable <NAME_OF_DOCKER_IMAGE> param1

Here we are specifying some other entrypoint executable.

docker-compose

version: '3'

services:
  web:
    image : repo/my-web-server:v2
    entrypoint: ["python3", "manage.py"]
    command: ["runserver"]
    ports:
      - "8000:8000"

Kubernetes

apiVersion: v1
kind: Pod
metadata:
  name: command-demo
  labels:
    purpose: demonstrate-command
spec:
  containers:
  - name: command-demo-container
    image: debian
    command: ["printenv"]
    args: ["HOSTNAME", "KUBERNETES_PORT"]
  restartPolicy: OnFailure

Here command corresponds to ENTRYPOINT & args corresponds to CMD in Dockerfile.

This is how I feel when I am comfortable with these small concepts with additional knowledge ;)

Related Articles

1. https://docs.docker.com/engine/reference/builder/#run
2. https://docs.docker.com/engine/reference/builder/#entrypoint
3. https://hynek.me/articles/docker-signals/
4. https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/

Thanks for reading! Hope it adds something to your knowledge base.

Let me know if you have any feedback. 🙏

- Kratik

Problems -

Suppose you have a web server, deployed on AWS EC2. Now to check your app logs you have to SSH to that EC2 and then cat the logs.

Or when you wanted to check some metrics for that EC2, you went to the built-in CloudWatch metrics section on the EC2 dashboard page aka Monitoring. You checked the CPU utilization... that's great! but wait, now you wanted to check current RAM usage or Swap memory usage or Disk Usage.

Wait, what? What did you observe? There are no metrics available for RAM/Swap/Disk usage? Now what?

Make a Wish🌠

I wish I had some simple solution🤞, using which I can check my logs and unlock a few more metrics that are not available to CloudWatch metrics by default, like how much space is left in my EBS volume or how much RAM is being utilized.

The Solution

Don't worry! The unified CloudWatch Agent is here for the rescue!

Features

• Collect internal system-level metrics from Amazon EC2 instances across operating systems.
• Collect system-level metrics from on-premises servers.
• Retrieve custom metrics from your applications or services using the StatsD and collectd protocols.
• Collect logs from Amazon EC2 instances and on-premises servers, running either Linux or Windows Server.

How to set up Cloudwatch Agent for Logs/Metrics Collection?

Assumption

I am assuming that you have a web server that serves a beautiful web app something like below (and it is generating some access and error logs):

Note - If Some Images are not clear due to big dimensions, just open them in a new tab, They will become the full-size image and you will be able to see the content clearly. The Images which I think need to open in a new tab once, I have made them clickable explicitly! So Enjoy :)

Installation of CW Agent

Install the agent using the command line for your OS. FYI, I am using Ubuntu 20.04. Head over to this AWS Documentation and follow the steps.

1. Download the DEB Package

   wget https://s3.amazonaws.com/amazoncloudwatch-agent/debian/amd64/latest/amazon-cloudwatch-agent.deb
   

2. Install the package

   sudo dpkg -i -E ./amazon-cloudwatch-agent.deb
   

Setting up IAM Permissions

No Matter If you using AWS EC2 or an On-Prem Server, you need to provide proper permissions using IAM so the machine can access and post metrics/logs to Cloudwatch!

For now, Let's take an example where we are using EC2, for on-prem servers, you can go through this documentation which will give you an idea around setup.

1. Create an IAM role. Select AWS Service, then EC2 and click on Next

2. In Attach Permissions, select a managed policy named CloudWatchAgentServerPolicy

3. Bonus - Later on, you will be creating a configuration file for CloudWatch, best practice here is that you save this config to AWS Systems Manager in the Parameter Store. You can use it later, also, you can modify it as per your needs. So for that, you need to add permissions to access SSM Param Store. Attach the policy for that as shown below.

4. Click Next, Supply some apt name for it and Finish the creation of Role.

5. Now, Go to EC2 Dashboard, select your EC2, Right-click for options, Select Security, then Modify IAM role and attach this newly created role.

Create CloudWatch agent configuration file

We have to instruct the CloudWatch agent that from where to fetch logs, what metrics to scrap, etc. For that, we need to create/generate the config file. We can do it manually or we can generate the file using a wizard. (Recommended) Follow this Docs to get started.

Before that, let's verify that our role is attached to our EC2!

1. Install AWS CLI

   sudo apt-get update && sudo apt-get -y install awscli
   

2. Check what is the current entity, which is attached to our EC2

   aws sts get-caller-identity
   

3. If You get some output like below, you are good to go, else recheck all the steps!

4. Run the wizard by running the following command:

   sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard
   

   I have used asciinema to record what parameters it asks, you can follow the same to make it work! I have chosen advanced metrics with a high-resolution time frame. + For the log part, I have to give the path of my log file which will be read and the content of it will be sent to CloudWatch. You can check these logs in the CloudWatch Logs later. If you see the recording below you'll see I have put the name of CloudWatch Log Group as access.log

5. That was not it :) Once you have done start the agent using the amazon-cloudwatch-agent-ctl utility :

   sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c ssm:AmazonCloudWatch-linux -s
   

   Output

6. I got some errors when I fired the above command, a file called types.db was missing in the collectd directory! Create both the things :

   sudo mkdir /usr/share/collectd/
   sudo touch /usr/share/collectd/types.db
   

7. Let's Re-run the agent :

   sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c ssm:AmazonCloudWatch-linux -s
   

   From Above Screenshot, I think you can conclude that the run was successful.

8. Let's verify in the CloudWatch, Check logs. In AWS Console, Go To Cloudwatch > log groups You can see that our log group is created and logs are also there :)

9. Let's Verify the Metrics, Go To Cloudwatch > Metrics > All Metrics. You'll see something like this: This means that metrics scraped by CW Agents are available.

Open CWAgent Metrics and select below metrics group :

From Here, select the row which have mem_used_percent

You will be able to see the graph of Memory Usage

Now as you have both, App logs and Advanced Metrics (which are not enabled by default in CloudWatch), Possibilities are infinite. You can create your CloudWatch alarms based on these metrics. You can keep an eye on logs efficiently without logging in to the server.

Extra - Why AWS CloudWatch does not have Memory usage metrics

The default metrics which AWS provides come from the Hypervisor level so we don't have to do anything special to see those. But for some metrics like Memory Usage, Disk Usage you have to get those metrics from the OS level. That's why we had to install an agent on the OS.

References:

1. AWS Official Documentation - Amazon CloudWatch

2. Installing the CloudWatch agent - AWS Docs

Drop Some Emojis if you find this useful. Or leave a comment, That's how I will know what I can improve :)

Thanks for Reading! 🙇‍♂️

Wanna buy me a Coffee ☕ ?

I am assuming that data is important but not that sensitive, so this guide can be applicable for some specific use cases.

For ease of storing and to save some $$$, you chose to host your content on an AWS S3 bucket. That way you will also be utilizing AWS Global Infrastructure to make sure your assets are highly available.

Let's See, How You created Your Bucket -

Enter a suitable name for your bucket :

Select an Encryption Method (To enable Encryption at Rest) :

and leave everything as it is. Now proceed to create your new shiny bucket.

Create a folder for storing your assets, for example - course-videos + maintain the encryption scheme.

Upload the content, for now, I am uploading two videos, see below :

Now, If you try to open your video in the browser using that S3 object URL, you'll see something like this:

and, that is totally fine, by default, your bucket is private and objects can not be accessed directly.

So now what? Ok, let's try to make the objects public and see what happens 🤔

Note - Click the image to enlarge

So If you were unable to understand, let me help you... Did you observe a checklist (aka Block public access (bucket settings)) while creating the bucket?

Which look something like this :

So the thing is this checklist makes sure that all public access to the bucket and its objects is denied!

But for our use case, we need our assets to be public so our users can consume them.

So let's make our objects public now... For that go to your BUCKET > Permissions > Block public access (bucket settings)

Here, uncheck all the settings as I have shown in the image above.

Now, try again to make that folder public for our users.

Woohoo! this time we were able to make our objects public.

I'd recommended you to go through this once to understand better what the Block public access settings offers.

Now let's add these URLs to your platform, I have created a Codepen where I will be referring these videos for demo purposes.

https://codepen.io/k4kratik/pen/NWvxowL?editors=1000

HTML Source : https://www.geeksforgeeks.org/html5-video/

When you add the S3 Object URL to your platform, users are able to see the content, Great!

💡For those who don't know, you can click on any object in S3 and from there you can copy the object URL.

The Problem

Wait what, you thought that was it? 😂 No! Let's see what's the issue here.

Keeping objects as public does not mean anyone should be able to download these objects so easily! We made those objects public so our users on the Internet can access it for viewing, surely not for downloading! 😤

The Workaround

Yup! You read the title right! It is not a solution but a workaround that can be effective for most of the users who can try to download your content! 😜

So even before we start, I want to summarize what we are gonna do - We will make use of Bucket Policies, We will whitelist your platform's domain name so only your platform's webpages can access your assets.

Step-1: Change Block public access (bucket settings)

Again, Go back to Block public access (bucket settings) and keep the checklist as shown below :

In BUCKET > Permissions > Block public access (bucket settings)

You'll see that UI is little changes with these indications:

Don't worry we will fine-tune the access in a bit, Just stick to the guide!

Validation

Now when I check the webpage (In my case, it is Codepen) The video is not accessible anymore! 😥

Step-2: Add the bucket policy to whitelist our domain

ℹ️ As you know from Step-1, our video was unable to play. let's resolve this.

For that go to your BUCKET > Permissions > Bucket policy and paste the below code.

⚠️ Make sure you are replacing kratik-blog-assets with your bucket name.

⚠️ For the value of the Key "aws:Referer", make sure you are adding your domain name(s).

{
    "Version": "2012-10-17",
    "Id": "http referer policy example",
    "Statement": [
        {
            "Sid": "Allow get requests originating from your domains only.",
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "s3:GetObject",
                "s3:GetObjectVersion"
            ],
            "Resource": "arn:aws:s3:::kratik-test/*",
            "Condition": {
                "StringLike": {
                    "aws:Referer": [
                        "https://codepen.io/*",
                        "https://cdpn.io/*"
                    ]
                }
            }
        }
    ]
}

It will look something like this:

Validation

Check the Codepen again and observe if something is different now.

Oh, Wow! Our video is able to play on our platform's Webpage.

ℹ️ As we created the above Bucket policy, it whitelisted our domain and allowed access.

Step-3: Testing the Download

Let's try again downloading our assets.

🎊 🎉 🥳

This is what we wanted right? Users can view your content but won't be able to download it.

isn't it cool?

Disclaimer

As I mentioned this is a workaround but not a 100% foolproof solution, technically advanced users can bypass this policy very easily, So if your data is really very very sensitive, my Suggestion is, Don't do this :)

For more sensitive content there are other methods available that require time and knowledge to set up but yeah, they will def fulfill your use case.

For example :

• S3 signed URLs
• Elastic transcoder

Drop Some Emojis if you find this useful. Or leave a comment.

Thanks for Reading! 🙇‍♂️

Wanna buy me a Coffee ☕ ?

What is Docker 🐬 ?

• An Open-Source tool that can be used to bundle your application with all the required dependencies.
• Similar to VMs but way more lightweight than it.
• Uses Dockerfile
• Written in Golang

What is Docker Compose?

• When you have a requirement of multiple containers and need of managing it nicely, you can achieve that with Docker Compose.
• Here you can define multiple containers, networking and how should they connect to each other, etc.
• Uses docker-compose file
• Written in Python

Ok So this was some basic info. Now, Let me ask you a question - Have you seen something like this before and ever wondered what it does exactly?

docker-compose -f docker-compose.yml -f docker-compose.prod.yml up -d

By observing the few lines, one can deduce that here we are using two compose files and then launching the compose stack in detach mode(-d)

The basic idea behind this is to keep your main compose file generic for all environments (like dev, stage, and prod) as much as possible.

And for all the needs, which are specific to an environment, pass another file with those configs and it will append all the files to generate the desired config.

Use case example - You are using almost the same config on your environments but for dev, the database port is open for devs to testing OR Suppose if you want to run the dev and stage environment in DEBUG mode but not the prod one!

Examples

#1 Usual docker-compose.yml file

web:
  image: example/my_web_app:latest
  build: .
  volumes:
    - ".:/code"
  ports:
    - 8883:80
  depends_on:
    - db
    - cache

db:
  image: postgres:latest
  command: "-d"

cache:
  image: redis:latest
  ports:
    - 6379:6379

Now, to pass some extra configuration updates, use another compose file called ...

#2 docker-compose-dev.yml

web:
  environment:
    DEBUG: 'true'

db:
  environment:
    production: 'true'
  ports:
    - 5432:5432

Notice in the second override file we only wrote the config we wanted at the respective block of each service.

Now when you want to up your stack, you need to fire the below command :

docker-compose -f docker-compose.yml -f docker-compose-dev.yml up -d

Features of using override compose files

1. It does not need to be a valid compose file, just wrote the part you want to update/add.
2. By default, Compose reads two files, a docker-compose.yml and an optional docker-compose.override.yml
3. We need to use -f to specify the file name or path.
4. Helps you to keep your base compose file generic as much as possible, you just have to create separate override files only according to your requirements.

References

1. https://docs.docker.com/get-started/overview/
2. https://docs.docker.com/compose/extends/#multiple-compose-files

Thanks for reading! have a nice day!

Wanna buy me a Coffee ☕?

Attached a screenshot below :

Okay, so this was pretty difficult for me to understand initially😅 and I was kind of misunderstanding this, So let's clear some air and improve our basics.

Let's Start!

Have you ever seen this kind of Security Group Configuration?

Notice the second rule in the SG Inbound rules!

Explaination

It simply means that " allow the traffic coming from all the members attached to that Security Group(which we added as the source in that particular rule)"

• Members means the instances(or other resources which uses a network interface)
• Remember! - Here it points the Private IPs of those Members

Use cases

1. In the case of ELB and EC2s, you can create 2 Security Groups, one for EC2s and one for Load Balancer. Now, In the EC2's Security Group, suppose your app runs at ports 80 and 443, create the inbound rules, and in the source for those rules add the Security Group ID of the Load Balancer's Security Group. That way you are only allowing traffic coming from Load Balancer, All direct traffic will be dropped!

2. Suppose you have a few instances. Now, you want to allow all inter-instance communication b/w them.

One dumb and insecure approach can be :

Here you allowed all Traffic from any source (0.0.0.0/0)

One other and better approach can be to use the concept, what we just learned. Here you can see we have given the ID of the Security Group itself.(Instead of giving 0.0.0.0/0) This means --> All the member(who have this SG attached), will be able to connect to other members on any port.

Note: Instead of all traffic, if you want, you can specify ports as well which are being used.

Private IP Addresses

From the Official Docs, "When you specify a security group as the source for a rule, traffic is allowed from the network interfaces that are associated with the source security group for the specified protocol and port. Incoming traffic is allowed based on the private IP addresses of the network interfaces that are associated with the source security group (and not the public IP or Elastic IP addresses).

• So when we are using this approach, keep in mind that behind the scenes it is whitelisting the private IPs of those Members!, So you won't be able to connect to the specified port if you are using Public or Elastic IP.

Hope you have learned something new from this blog :)

If you have any Queries/Feedback, do let me know 🙏

Thank you so much!

References

1. https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html#SecurityGroupRules

Wanna buy me a Coffee ☕ ?

Image Source: https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Security.html

When You create a new VPC, it comes with a default NACL which allows all ingress/egress(or inbound/outbound) traffic but when you create a custom NACL it denies all the inbound and outbound traffic by default.

If you don't explicitly associate a subnet with a network ACL, the subnet is automatically associated with the default network ACL.

Features of NACLs

1. You can apply NACL at the Subnet level. (A subnet can be associated with only one NACL, you can attach multiple subnets to the same NACL though).
2. You can specify in the rules that if you want to ALLOW or DENY the traffic that matches the particular rule.
3. These are stateless. It means if you allow inbound traffic on port 3000 in NACL, it won't allow the outbound or returning traffic by default (for requests made on port 3000), you have to allow the port(if you know already what port will be used for returning traffic) or port range in the Outbound Rules of NACL (for example, for web HTTP requests, it uses ephemeral ports i.e. 1024–65535).
4. When you are creating rules for an NACL you have to provide that entry a number (max is 32766), the reason for this is, AWS evaluates the rules in order, starting with the lowest numbered rule, to determine whether traffic is allowed in or out.
5. AWS recommends that we should create rules in increments (for example, increments of 10 or 100) so that we can insert new rules later on.
6. NACL rules are applied on the traffic which is inbound or outbound but not for the traffic within resources of the subnet.

How NACLs looks in AWS Console

Difference b/w Security Groups and NACLs

Practical Example

1. Suppose you set up a Web Server on an EC2 that is serving req at port 80.
2. Assume you associated an SG that allows all traffic.
3. Now, you created a custom NACL, as you have just read above that a new NACL denies all traffic. And attached this NACL to the subnet where our EC2 is.
4. Now, You tested the EC2 Public IP with port 80 using curl to check the response but it got timed out. (REASON - NACL is at the subnet level and denying all traffic)
5. Okay, So Now you added a port 80 allow rule in the inbound rules in NACL to allow traffic on port 80.
6. Back to Terminal, Tried curl again but wait what, why it is still not working? REASON/EXPLANATION -> So this is how it behaves, aka stateless behavior. Your web server listens at port 80, that is true but when sending the response it uses some random port (also called short-lived aka ephemeral ports)
7. Now In the NACL Outbound rules, allow traffic for ephemeral ports (1024–65535).
8. Now if you try curl you will be able to see the response in your terminal because now NACL allows the outbound traffic as well. 🎊🎉

Where NACLs are perfect and SGs are not?

There can be a few use cases where your requirement can not be fulfilled by Security Groups and you have to use NACLs. For example - Suppose you want to block traffic from specific IPs, as SG doesn't have the feature of explicitly DENY you have to use NACLs only.

and Remember! NACL is the first line of defense, SGs is the second!

References

1. https://docs.aws.amazon.com/vpc/latest/userguide/vpc-network-acls.html
2. https://digitalcloud.training/certification-training/aws-solutions-architect-associate/networking-and-content-delivery/amazon-vpc/
3. https://stackoverflow.com/a/53625507
4. https://qr.ae/pGS8f1

Thanks for reading! have a nice day!

Wanna buy me a Coffee ☕ ?

Generally, we use commands to check logs of each container, for example,

docker logs -f container_1

and it is fine for a few containers, but suppose, you have a complex architecture, where you are using Docker Swarm or Docker-Compose or just assume you have a number of containers, then what?

Yes! It will be a real pain to maintain those logs, on top of that, other issues like No sharing of logs with the team, No Access-based control, etc. will also be there.

I bet the native Docker plugin kinda feeling similar as below right now :

Let's try a solution for that :)

We will be using Loki and Grafana to achieve our goal. Loki is a set of components that can be composed into a fully-featured logging stack. More info here and here

Before we jump into installation and all other intense stuff, let's see what benefits are we going to get by end of this installation:

1. Restricted User Access
2. Organization Support
3. Log Rotation
4. Centralisation of logs
5. Visualize various logs in a single place using Loki Queries.
6. Live Tailing of logs

Let's start the setup:

1. Configuring the Logging Driver for Docker
2. Setting up the Loki instance with its config file
3. Setting up Grafana Dashboard to visualize the logs.

1. Configuring the Logging Driver for Docker

*you need to repeat these steps for each Docker node if you are using Docker Swarm

- Install the driver

docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions

- Configure the Docker to send logs to Loki Instance

{
    "debug" : false,
    "log-driver": "loki",
    "log-opts": {
        "loki-url": "http://127.0.0.1:3100/loki/api/v1/push"
    }
}

- Restart the Docker Daemon on each node

# Check the number of the nodes in your swarm 
docker node ls
# Drain the node so the running workload can be shifted to other nodes.
docker node update --availability drain node1_name
# Restart the docker daemon on that node
systemctl restart docker
# Make the node active again
docker node update --availability active node1_name

2. Setting up the Loki instance with its config file

To get the codebase, Clone my fork

git clone https://github.com/k4kratik/loki

- Observe the config file loki-local-config.yaml

auth_enabled: false

server:
  http_listen_port: 3100

ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
  chunk_idle_period: 15m

schema_config:
  configs:
  - from: 2019-06-01
    store: boltdb
    object_store: filesystem
    schema: v9
    index:
      prefix: index_
      period: 168h

table_manager:
.
.
.
# Notice this
  retention_deletes_enabled: true
  retention_period: 672h
...

Here we are setting up the auto-deletion of old logs after a certain period, which is 28 days (672 Hours) in our case.

- Now Install the stack

docker stack deploy -c docker-compose.yml loki-stack

- Verify that everything is working fine.

docker service ls

3. Setting up Grafana Dashboard to visualize the logs.

above stack file also contains a grafana instance, which will be available to YOUR_SERVER_IP:3000

Go to Grafana Dashboard, Initial credentials will be admin:admin

Go to Configuration > Data Sources as displayed in the picture.

Now Click on Add Data Source and enter the URL as http://loki:3100

If you have done everything correctly till this point, Clicking Save & Test will emit a success message.

❓ Just one question out of curiosity... We did not enter a IP or Domain Name, then How did it figure out our Loki instance correctly ? 🤔

The answer is : Check the docker-compose file, the loki service is named as loki, and other containers can reach to this container by the service name, which is in our case is loki.

Every container has by default a nameserver entry in /etc/resolve.conf, so it is not required to specify the address of the nameserver from within the container. That is why you can find your service from within the network with service or task_service_n.

- A stackoverflow answer

Using Grafana

Once you have added the Data Source, Go to Explore as instructed below:

Select a container to view its logs

Just like the image below, you can check logs of host, stdout & stderr streams, containers, Docker services, Docker stacks, etc. You can also combine multiple filter to get a consolidated output of your choice.

Security Measures

1. Block the internet access on Loki port 3100 (+ Grafana Port 3000 if you want). Using Firewall is heavily recommended. As the official docs says, Loki does not come with any included authentication layer, So it would be good if you allow only specific IPs.
2. If you want to make it more secure, set up an NGINX reverse proxy with Basic Auth. [For Production Environments]

Making it more Fault-Tolerant and HA

If you want to take this on a whole next level, well, there is good news for you, for the storage part you can also use Cloud DBs like DynamoDB for index and S3 for storing chunks. You can read storage options here.

My excited Indian ass dances like this whenever I complete any setup :) We all love that feeling, don't we?

Anyways, Thanks for reading and bearing all my content and The Office GIFs! Hope you got some knowledge from this blog, I am just writing what I am thinking, still trying to making it as simple and straightforward as possible.

Wanna buy me a Coffee ☕ ?

Hey All!

We all have worked with at least one Version Control System Softwares in our day-to-day tasks and Git is one of those and the most widely used VCS Tool in the world!

What are Hooks in Git?

According to the official site, Hooks are programs that trigger actions at certain points in git’s execution.

How to Add a Hook?

The default directory where Git Hooks resides is .git/hooks. (if you don't know you will find this hidden git folder at the root of your repo)

If you want, you can change this behavior and configure another directory for Hooks. Example -

$ git config core.hooksPath my-custom-hook-directory

So the idea here is that we can configure something in the pre-commit hook and before we commit, it will check If we have modified the restricted file or not and If we have modified the file it will raise an error, and the commit will be exited/rejected.

Git Hook files are nothing but shell scripts, So Before doing any Scripting, let's see what hooks we have by default in our git repo.

So I did ls -lha .git/hooks/ on my repo's root and got the below output:

As you saw in the output of ls command above, by default there are some sample hooks and if you want to use one, simply remove the .sample from the name. And then you can edit the file as you want and the respective hook will be applied automatically. It is clear from the name of the files that when that particular hook will be called. Here we want to use that pre-commit hook.

⚠️ These are Client Side hooks, so if you push your code (For example - to GitHub), the hooks won't get pushed to the remote (or can not be shared amongst other contributors), As the .git directory never gets pushed. You can read some explanations here as well.

That was some basic info about hooks, now Coming to the title of our Blog, which is How to Prevent a file to be overwritten in a Git Repo?

So now if you have guessed already then congrats, but for others, we can configure something in the pre-commit hook which will reject any commit which includes changes of that file.

Add the hook file as I mentioned above (by creating a file called pre-commit in the .git/hooks directory)

So now the question arises How can we check if a file changed or not? We generally run the command git diff to check if a file changed or not. There is an argument available for the above command --exit-code which returns exit code 1 if the file changed, otherwise returns 0 (no error).

A more mature command looks like below:

git diff --exit-code HEAD $FileName >/dev/null 2>&1

which means it is checking the difference between HEAD (current branch or last committed state on current branch) & Current Changes (Index, Staged, or whatever you prefer to call it.)

Now what we can do is, that we can store that exit code into some variable and according to that print a message for the user before rejecting the commit.

For example, In my repo, I have created a file called secure.file and we want to prevent any changes for this file.

So My pre-commit file looks somewhat like below:

FileName="secure.file"

git diff --exit-code HEAD $FileName >/dev/null 2>&1

DIFF_EXIT_CODE=$(echo "$?")

echo [LOG] [DEBUG] Diff Exit code was: $DIFF_EXIT_CODE

if [ $DIFF_EXIT_CODE != 0 ]; then
    RED='\033[0;31m'
    NC='\033[0m' # No Color
    printf "${RED}Error Ocurred! You can not change the${NC} $FileName\n"
    echo "Revert the changes for $FileName file and try again"
    exit 1
fi

It may look a little scary but the above code snippet is too easy to understand and interpret. We Stored the EXIT CODE, Using the If Block and exit command we printed the appropriate message and exited the script. Also, I have used Colourised echo output and a more customized message, just for flex ;)

So now If I try to commit any change to that file, I get the below Error:

So that was a basic example, there is no end to your creativity. Read more about all the hooks here . This way you can implement as many checks and hooks as you want, The exciting part here is that you will be the one who will be tailoring a custom UX for yourself.

As I mentioned that hooks will never get pushed to the central repo, what you can do is store the hooks in a separate directory and Push that to remote. That way, it will be shared amongst other devs and also, it will be version controlled.

Hope this article helped in some way, Let me know if there is any suggestion. Thanks!

Till We meet the next time 👋🏻 :

Wanna buy me a Coffee ☕ ?