# How to use Vault with External Secrets for Kubernetes in Production? Hello, beautiful people on the Internet! 🌻 Today we are going to discuss how can we optimize the flow of storing secrets in Kubernetes and Also, learn to empower our developers to View/Modify secrets deployed in our Kubernetes cluster. We'll also delegate access to our devs based on the GitHub team they belong to. So let me ask you a basic question, how do you store secrets in your K8s cluster? If you answer that **you are using plain base64 encoded manifest files and managing them manually**, then : ![](https://i.giphy.com/media/wHE6Dd6RCVHQfjK5dy/giphy.webp align="center") --- ### Issues in the current approach 1. No easy visibility - No GUI to easily manage secrets 2. No Version Controlling 3. Need to expose K8s creds to devs if you want to share it with them. (You can create and set appropriate RBAC permissions for them though but it can not be used in some use cases) 4. No Support for any 3rd Party Authentication, for example - Using GitHub Team/Org to grant developers a predefined level of access. --- ### Simple Solution - Use Vault with External Secrets Yes, you heard it right, once this is set, delegating access to users would be a breeze :) ![Season 1 Episode 3 GIF by BET Plus](https://media4.giphy.com/media/xjQfDCSRr2jkH3SPab/giphy.gif?cid=ecf05e470rx4qrqb4wgs6x6rvggqr7445wr9rel3iswhemi7&ep=v1_gifs_search&rid=giphy.gif&ct=g align="center") Action Items for us ⚡️ * Install External Secrets Operator * Install Vault * Configure Vault to Allow login using GitHub PAT * Configure Vault auth for External Secrets * Configure External Secrets to create secrets from Vault * Create a sample secret in Vault, and see that being created in K8s --- ## Install External Secrets Operator [https://external-secrets.io/latest/introduction/overview/](https://external-secrets.io/latest/introduction/overview/) It has three components - **External Secrets Controller, Webhook, Cert-Controller** You can install external secrets quickly with Helm! [https://external-secrets.io/latest/introduction/getting-started/](https://external-secrets.io/latest/introduction/getting-started/) `values.yaml` ```yaml #? count of the controller pods for HA replicaCount: 3 #? enable/disable the leader election - at any time, only one active controller is recommended leaderElect: true #? how many secrets to update concurrently concurrent: 3 #? PDB configuration - helpful when doing maintenance tasks podDisruptionBudget: enabled: true minAvailable: 1 #? Recommended - Exposes metrics! serviceMonitor: enabled: true webhook: serviceMonitor: enabled: true certController: serviceMonitor: enabled: true ``` ```bash helm repo add external-secrets https://charts.external-secrets.io helm upgrade --install external-secrets external-secrets/external-secrets -f values.yaml -n external-secrets --create-namespace ``` once deployed, It will look something like below : ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1695553611814/7cf45cac-f55a-4568-9e7f-6c30b0274457.png align="center") **Note** - You can see three deployments of `external-secrets` but as we have enabled `leaderElect` only one of them will be active and hold the lease, other replicas will be just waiting if the primary one goes down and then one of the standby replicas will be the leader. Isn't it cool? ![](https://media3.giphy.com/media/1LFycEz3YJQHhxa9Qv/giphy.gif?cid=ecf05e47bwj9pxf9414upzjbexojv9h829fv7hh9gbhvsl7r&ep=v1_gifs_search&rid=giphy.gif&ct=g align="center") **Bonus!** - More on K8s leader election : [https://carlosbecker.com/posts/k8s-leader-election/](https://carlosbecker.com/posts/k8s-leader-election/) --- ## Install Vault #### Working *The internal working of Vault is quite interesting and a little complex to sum up in few words, we will need another blog to explain it in more detail. If you are completely clueless, then you can read more about this here:* 1. [https://developer.hashicorp.com/vault/docs/what-is-vault#what-is-vault-1](https://developer.hashicorp.com/vault/docs/what-is-vault#what-is-vault-1) 2. [https://developer.hashicorp.com/vault/docs/internals/architecture](https://developer.hashicorp.com/vault/docs/internals/architecture) To install Vault, We will be using [Banzaicloud](https://github.com/bank-vaults/bank-vaults)'s Vault Operator aka [Bank Vaults](https://github.com/banzaicloud/bank-vaults)! *To deploy anything in the Kubernetes Environment and to ensure its reliability and HA,* ***Operators*** *are becoming de-facto standards.* 💡 **Do you know :** *A Kubernetes operator is an application-specific controller that extends the functionality of the Kubernetes API to create, configure, and manage instances of complex applications on behalf of a Kubernetes user.* Check more here : [https://operatorframework.io/what/](https://operatorframework.io/what/) ### Install Vault-Operator Using Helm ```bash helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com helm repo update helm upgrade --install vault-operator banzaicloud-stable/vault-operator -n vault-operator ``` ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1697653951779/984560b7-055b-4586-bad6-46a5069c75d6.png align="center") ### Setup RBAC `rbac.yaml` - need to apply this to give needed RBAC access to our setup. %[https://gist.github.com/k4kratik/5864d652a384601cb4a2718a902ba20b] ```bash kubectl apply -f vault/rbac.yml -n vault-operator ``` **💡 Explanation -** Here, We are creating a Secret **vault-sa-token-manual** which will hold Kubernetes access token for ServiceAccount named **vault *(Before Kubernetes v1.24 we didn't have to create the secret manually, see - https://stackoverflow.com/a/72258300)*.** This ServiceAccount has access to read/update pods and all access to secrets. **Note** - You can also fine-tune the access by adding only required verbs/resource names/namespaces. Also, did you notice we are binding a default `ClusterRole` named `system:auth-delegator` to `Vault` ServiceAccount? [From the official docs](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#other-component-roles) - *Allows delegated authentication and authorization checks.* The `system:auth-delegator` ClusterRole has the permissions to call the **Token Review** API. ![Christian Bale GIF by PeacockTV](https://media4.giphy.com/media/v1.Y2lkPTc5MGI3NjExYTJ6MDhnY2F5cWpseGJ6eTB6MXdsemg3eHdiNnRyenZrMXFzdXlmYSZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw/eKNrUbDJuFuaQ1A37p/giphy.gif align="center") ### Deploying Vault with CRD Now, as we have powered our cluster with Vault Operator, we can deploy Vault with a simple YAML file. `vault-deploy.yaml` %[https://gist.github.com/k4kratik/8a18b1d0a4b40debe06fc0614efa3508] ## Authentication & Authorization Here we are authenticating our users using [GitHub auth backend](https://developer.hashicorp.com/vault/docs/auth/github). And their access level is defined by the GitHub teams they belong to. For Example, my account ([k4kratik](https://github.com/k4kratik)) is assigned the admin policy `vault_admin`. Similarly, I have created a GitHub Team `DEVS_RW` in my organization which will grant its members access defined in the policy `rw_access_policy` and Also created a team `DEVS_RO` for devs for which I only want to configure read-only access, this team will be assigned with the `ro_access_policy`. The above is visualized below : ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1697713087784/5ea7d446-fa3e-4ba7-a9a4-68d69d4a91b3.png align="center") ## Config file Explanation : From [Line #1 to #102](https://gist.github.com/k4kratik/8a18b1d0a4b40debe06fc0614efa3508#file-vault-deploy-yml-L1-L102), I think it's pretty much self-explanatory. Let's Start with Line#106. (where ***externalConfig*** starts) #### externalConfig.policies Just like any other setup, we are defining IAM policies here, these will be assigned to users/teams. * `ro_access_policy` : only `read` permissions. * `rw_access_policy` : RW permissions. \["read", "list", "update"\] * `vault_admin` : Just defining `*` with verbs does not work to grant all permissions. So, we have defined all the admin-level access explicitly. **Why ?** : more info : [https://discuss.hashicorp.com/t/vault-admin-policy/39803/2](https://discuss.hashicorp.com/t/vault-admin-policy/39803/2) admin required access: [https://developer.hashicorp.com/vault/tutorials/policies/policies#write-a-policy](https://developer.hashicorp.com/vault/tutorials/policies/policies#write-a-policy) #### externalConfig.auth:kubernetes * `type` : As we are setting this up for `external-secrets` running on Kubernetes, its value should be `kubernetes` * `path` : the identifier for this auth entry in the vault. * `config` * `token_reviewer_jwt` : The JWT Token for `vault` ServiceAccount because it has the `system:auth-delegator` role, which can help us to use K8s' `TokenReview` API to validate other JWTs. To fetch its value, run the following command as in the earlier section of the blog, we have already set up the required RBAC. **Command** (copy this and replace with `TOKEN_REVIEWER_JWT_TOKEN_HERE` ) ```bash kubectl get secret vault-sa-token-manual -n vault-operator -o go-template='{{.data.token}}' | base64 --decode ``` * `kubernetes_ca_cert` : PEM encoded **CA cert** for use by the TLS client used to talk with the **Kubernetes API**. **Command** ```bash kubectl get secret vault-sa-token-manual -n vault-operator -o go-template='{{index .data "ca.crt"}}' | base64 --decode ``` * `kubernetes_host` : Endpoint for `Kubernetes API server`. Make sure it's reachable from vault pods. ```bash kubectl config view --minify --output jsonpath="{.clusters[*].cluster.server}" ``` * `disable_issuer_verification` : `true` - To disable the issuer verification. * `roles` - *define roles attached to this auth mechanism* * `name`: name of the role * `bound_service_account_names` : `["external-secrets"]` - We just want to allow external secrets SA, so adding that only. * `bound_service_account_namespaces` `["external-secrets"]` - We just want to allow external secrets namespace, so adding that only. * `token_policies` : `ro_access_policy` - Important! - the name of the policy to attach, Our **external-secrets** just need read access to read the secrets. * `token_max_ttl` : token expiry time. #### externalConfig.auth:github * `type` : As we are setting this up for auth via GitHub, its value should be `github` * `path` : the identifier for this auth entry in the vault. * `config` * `organization`: name of your GitHub organization. (*Replace GITHUB\_ORGANIZATION\_NAME\_HERE with this)* **Note!** - ***You must be part of this organization***, or else you will not be able to log in, and you'll get the error : `Configuration is not set!` * `token_ttl`: lifetime for generated tokens. `map` (mapping for GitHub) * `teams` (mapping for GitHub Teams with Vault policies) * **format**: * TEAM\_NAME : POLICY\_NAME * **e.g.** * `DEV_RW`: `rw_access_policy` * `DEV_RO`: `ro_access_policy` * `users` (mapping for GitHub Users with Vault policies) * e.g. `k4kratik`: `vault_admin` #### externalConfig.secrets * `path` : `secret` - path where the Vault will mount this secret engine. * `type` : `kv` - Secret Engine type : Key/Value * `options` : * `version` : 2 (recommended) #### startupSecrets * A test secret, which will be created at the startup. #### audit * Audit devices are the components in Vault that collectively keep a detailed log of all requests to Vault. * The file audit device writes audit logs to a file. [https://developer.hashicorp.com/vault/docs/audit/file#configuration](https://developer.hashicorp.com/vault/docs/audit/file#configuration) --- ## Deploy Vault ```bash kubectl apply -f vault/vault-deploy.yml ``` it will deploy pods as shown below: ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1697654333940/10e46cc8-6d29-47e3-9d02-a1b8d23b2f33.png align="center") ## Login To Vault UI * Let's expose the Vault UI for login ```bash kubectl port-forward svc/vault -n vault-operator 8200:8200 ``` ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1697654921736/55e15d2e-4018-4cf4-983d-616f4aa5188d.png align="center") * You will be welcome with the below screen : ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1697654480467/f8fa9853-50c3-4cf9-9d43-770b41cac92b.png align="center") * **Login with GitHub PAT (Personal Access Token)** * Generate a GitHub PAT with at least `read:org` permission ([guide here](https://docs.github.com/en/enterprise-server@3.6/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-personal-access-token)) * We will use that PAT to login into Vault ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1697651896103/4d03a720-5f3b-4fb8-9cc9-1b8ce3625bfb.png align="center") **Note** - Remember the auth policy we created, according to our GitHub username, org or Team, We will be getting relevant access. ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1697652699101/979a2bcc-9cdc-4750-a456-5788e4407cad.png align="center") * Check the `secret` engine, you will find our test secret : `TEST_PROJECT_ONE/test` ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1697652842671/2975c798-7a0b-4e25-8f49-57daa34fd646.png align="center") * **Login with Root Token \[Not Recommended\]** * Get the root token ```bash kubectl get secrets vault-unseal-keys -o jsonpath={.data.vault-root} | base64 --decode ``` * Login with this token as shown below ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1697655513417/76942d2a-db36-40b5-9e6b-4a28cef5811f.png align="center") * **Check What we configured :** * Check the auth we defined - \`github\` and \`k8s-one\` ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1697652052543/e9aa79e3-3124-4c3a-b07a-4542057d318c.png align="center") * Let's check in `k8s-one` auth method, if our role `k8s-one-external-secrets-role` exists or not ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1697653247859/260e40d6-0595-408f-a90f-f3f4ff1766f6.png align="center") --- This sums up the basic vault authentication and authorization setup + external secrets. ![Jimmy Fallon GIF by The Tonight Show Starring Jimmy Fallon](https://media3.giphy.com/media/eTxBDDpyjgF4wUVDmM/giphy.gif?cid=ecf05e47licwcjwb48e61n0x24usdbth9j7xkmps3ttukclo&ep=v1_gifs_search&rid=giphy.gif&ct=g align="center") but hold your horses! we still need to create secrets from the Vault using External Secrets. Apologies if the blog seems too long, but trust me, it will be all worth it in the end. ![Season 1 GIF by Showtime](https://media2.giphy.com/media/v1.Y2lkPTc5MGI3NjExY2w4YTV2YTd0YnZoODAyN3JxZWllaGNxYTRwdWMxdHI3bndyMmw3NCZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw/eMsBUFTW96Y5kzcwMy/giphy.gif align="center") --- ## Setup External Secrets External Secrets have two important types of objects 1. `SecretStore`(or `ClusterSecretStore`) - It defines how to access the external API secret provider. (e.g. Vault, AWS Secrets Manager) 2. `ExternalSecret` (or `ClusterExternalSecret`) - It describes what data should be fetched, how the data should be transformed and saved as a K8s `Secret`. * Let's **create a Cluster-scoped Secret Store** to connect it with the Vault. * %[https://gist.github.com/k4kratik/9b8583d155423490cee0cd823d017da2] ```bash kubectl apply -f external-secrets/secret-store.yml ``` * It will look something like below ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1697705007783/4f045495-3d9d-455e-9fba-557da8b24877.png align="center") * Before creating the `ExternalSecret`, let's create some test secrets in Vault ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1697710464666/55f06d6d-f8a5-4afa-b064-30e267483a47.png align="center") ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1697710598124/d59acec7-d619-426c-b734-50849b384721.png align="center") * Now, Let's create an `ExternalSecret` to use our `ClusterSecretStore` and create a k8s secret using it. `sample-external-secret.yaml` ([GitHub](https://github.com/k4kratik/vault-with-external-secrets-k8s/blob/main/external-secrets/sample-external-secret.yaml)) * %[https://gist.github.com/k4kratik/1df82010dd5f7263608b03adaecd5069] ```bash k apply -f external-secrets/sample-external-secret.yaml ``` ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1697705904475/ec846bf3-422d-49c8-bdc4-fb6f5dca8155.png align="center") * Let's Verify it by checking the secret ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1697706041068/cbf68135-c517-48f6-8ea9-e27eafa8b1dc.png align="center") ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1697706301266/90b6edf1-583f-4aae-987a-41162904b9c3.png align="center") Voila! we can see our secret is created and we can also see our values from Vault went through 🥳 🎉 ![That Really Did The Trick Kyle Broflovski Sticker - That Really Did The Trick Kyle Broflovski South Park Stickers](https://media.tenor.com/JVie7tHp0NAAAAAi/that-really-did-the-trick-kyle-broflovski.gif align="center") *I had to decide between these two GIFs to place here, couldn't resist myself from using both* 😛 😅 ![The Office Office GIF - The Office Office David GIFs](https://media.tenor.com/Z_z9W1qlVAQAAAAC/the-office-office.gif align="center") **Setup Alternatives (or Future Improvements?)** 1. Instead of Vault, you can also use AWS Secrets-manager. ([Official Provider](https://external-secrets.io/main/provider/aws-secrets-manager/)) 2. Instead of a self-signed cert, you can also use certs created by some reputed CA. 3. You can also use your domain and point the vault there with SSL. (ex. - AWS ALB Ingress w/ ACM SSL Cert) 4. You can create more fine-grained policies as per your use case. --- ## References * [https://developer.hashicorp.com/vault/docs/auth/kubernetes](https://developer.hashicorp.com/vault/docs/auth/kubernetes) * [https://developer.hashicorp.com/vault/docs/configuration](https://developer.hashicorp.com/vault/docs/configuration) * [https://kubernetes.io/docs/reference/access-authn-authz/authentication/](https://kubernetes.io/docs/reference/access-authn-authz/authentication/) * [https://kubernetes.io/docs/reference/kubernetes-api/authentication-resources/token-review-v1/](https://kubernetes.io/docs/reference/kubernetes-api/authentication-resources/token-review-v1/) * [https://learnk8s.io/microservices-authentication-kubernetes](https://learnk8s.io/microservices-authentication-kubernetes) * [https://developer.hashicorp.com/vault/tutorials/policies/policies#write-a-policy](https://developer.hashicorp.com/vault/tutorials/policies/policies#write-a-policy) * [https://developer.hashicorp.com/vault/docs/auth/github](https://developer.hashicorp.com/vault/docs/auth/github) * [https://external-secrets.io/latest/api/externalsecret/](https://external-secrets.io/latest/api/externalsecret/) * [https://external-secrets.io/latest/api/secretstore/](https://external-secrets.io/latest/api/secretstore/)