Search for a command to run...
![Securing your Logging Solution for Docker [with NGINX Reverse Proxy]](/_next/image?url=https%3A%2F%2Fcdn.hashnode.com%2Fres%2Fhashnode%2Fimage%2Fupload%2Fv1660320741107%2FRRonXGBVU.webp&w=3840&q=75)
Hey, Y'all 👋 Kratik Here :)
Hailing from The City of Lakes, India.
I love to learn and share stuff about DevOps, SRE, Security, Cost Optimization, etc.
Join me on this journey where we will learn together with the help bunch of interesting blogs.
No comments yet. Be the first to comment.
DNS Search Domains, ndots, and Kubernetes Resolver Behavior
Understand how IAM Access Entries can replace aws-auth for managing EKS IAM authentication.
Effective Techniques for Providing IAM Access to EKS Pods
Ever wondered how can you create your own custom Github Action and publish it to the GitHub Marketplace? Let's learn it step by step in this blog.
Let's setup Vault with External-secrets to manage secrets in Kubernetes like never before!
From our past blog Best Logging Solution for Docker [Basic Version],
We saw how to make logging a piece of cake using Loki & Grafana.
But I think there is one minor issue If your servers are on the public internet and you have enabled port forwarding for your use case, then anyone using your Loki URL, can add it as a data source in their Grafana Installation and will be able to see all of your sensitive logs, which is the last thing on earth we want.
It's simple right?
When you run Loki, out of the box, it does not come with any kind of additional security layer. So basically anyone who can access that URL, can access and see your logs, simple as that!
Loki Official Documentation says that Grafana Loki does not come with any included authentication layer.
https://grafana.com/docs/loki/latest/operations/authentication/
Now as I am thinking about it, feels like we can use some simple tactics to enhance security. Those are :
If not required, don't launch your servers on the public subnet. (hence reducing the possibility of someone connecting from outside your network)
Do not use port-forwarding, instead use DNS which is supported by the docker network. (for example, instead of http://Server_IP:3100 we can use http://loki:3100)
Use some firewall/SecurityGroups to whitelist services/users to allow limited access to required ports.
Advance - You can use a reverse proxy(NGINX) with Basic Auth. So even if someone has connectivity to the endpoint, they need to enter basic auth credentials to connect :)
I am assuming, you already have a docker environment ready that is already using Loki without any auth method.
You can take a reference from my previous blog as well.
Do you know what I did?
I referred to this official docker-compose file to get started in a few seconds.
version: "3"
networks:
loki:
services:
loki:
image: grafana/loki:2.6.1
ports:
- "3100:3100"
command: -config.file=/etc/loki/local-config.yaml
networks:
- loki
promtail:
image: grafana/promtail:2.6.1
volumes:
- /var/log:/var/log
command: -config.file=/etc/promtail/config.yml
networks:
- loki
grafana:
image: grafana/grafana:latest
ports:
- "3000:3000"
networks:
- loki
Luckily I stumbled upon this cool open-source project which does the job for you :
So now what? How to use this?
You just have to give the required env vars and it will launch NGINX accordingly.
Those are:
Note: Here we will add our Loki URL in the PROXY_PASS section so it can forward our request to Loki.
So Now, as we know how can we secure our Loki Installation, let's add this reverse proxy in our docker-compose.yaml file.
version: "3"
networks:
loki:
services:
loki:
image: grafana/loki:2.6.1
ports:
- 3100
command: -config.file=/etc/loki/local-config.yaml
networks:
- loki
promtail:
image: grafana/promtail:2.6.1
volumes:
- /var/log:/var/log
command: -config.file=/etc/promtail/config.yml
networks:
- loki
grafana:
image: grafana/grafana:latest
ports:
- "3000:3000"
networks:
- loki
nginx:
image: quay.io/dtan4/nginx-basic-auth-proxy
environment:
- BASIC_AUTH_USERNAME=loki_user
- BASIC_AUTH_PASSWORD=loki_super_secret_password
- PROXY_PASS=http://loki:3100
- PORT=3200
- SERVER_NAME=_
ports:
- 3200:3200
- 8090:8090
networks:
- loki
Here I also disabled port-forwarding for Loki, because that is not required. Also, notice that in the PROXY_PASS env var, I gave loki:3100 which is the service name in the docker-compose file and is internally resolvable by the containers in the same docker network.
Here, we are running this NGINX server on port 3200 + we did port forwarding on the same port, so we can use localhost:3200 to connect to the NGINX server.
Also, port 8090 is used for NGINX metrics. You can check that on -> :8090/nginx_status
Now, up this docker-compose file and see what happens at the Grafana end.
So, let's try to add a data source, just like before.
As expected, we have disabled the port forwarding, it should fail.
Again, this also fails as above.
Just for curiosity, let's curl and see.
Oh ok, let's try with the creds which we supplied to NGINX in docker-compose.
- BASIC_AUTH_USERNAME=loki_user
- BASIC_AUTH_PASSWORD=loki_super_secret_password
Now using the same endpoint, Enable the Basic Auth, and add those basic auth credentials
hmm...
Voilà! - we were able to connect to our Loki Data source with an additional layer of security i.e. Basic Auth.
I know most of you are using Kubernetes out there and only a small set of users can relate to this(and implement it) but it is a really small fun exercise on how to use NGINX reverse proxy, setup Basic Auth, docker-compose(or docker swarm) inter-service communication and securing Loki installation. 😄
and the other reason to be happy is:
That was it! Let me know what you have to say in the comment box.
Thanks for reading :)