The Issue

From our past blog Best Logging Solution for Docker [Basic Version],

We saw how to make logging a piece of cake using Loki & Grafana.

But I think there is one minor issue If your servers are on the public internet and you have enabled port forwarding for your use case, then anyone using your Loki URL, can add it as a data source in their Grafana Installation and will be able to see all of your sensitive logs, which is the last thing on earth we want.

It's simple right?

When you run Loki, out of the box, it does not come with any kind of additional security layer. So basically anyone who can access that URL, can access and see your logs, simple as that!

---

Loki Official Documentation says that Grafana Loki does not come with any included authentication layer.

grafana.com/docs/loki/latest/operations/aut..

---

The Solution(s)

Now as I am thinking about it, feels like we can use some simple tactics to enhance security. Those are :

1. If not required, don't launch your servers on the public subnet. (hence reducing the possibility of someone connecting from outside your network)

2. Do not use port-forwarding, instead use DNS which is supported by the docker network. (for example, instead of http://Server_IP:3100 we can use http://loki:3100)

3. Use some firewall/SecurityGroups to whitelist services/users to allow limited access to required ports.

4. Advance - You can use a reverse proxy(NGINX) with Basic Auth. So even if someone has connectivity to the endpoint, they need to enter basic auth credentials to connect :)

---

The Implementation of the advanced solution (Just for fun!)

Step-1: Keep your environment ready

I am assuming, you already have a docker environment ready that is already using Loki without any auth method.

You can take a reference from my previous blog as well.

Do you know what I did?

I referred to this official docker-compose file to get started in a few seconds.

version: "3"

networks:
  loki:

services:
  loki:
    image: grafana/loki:2.6.1
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      - loki

  promtail:
    image: grafana/promtail:2.6.1
    volumes:
      - /var/log:/var/log
    command: -config.file=/etc/promtail/config.yml
    networks:
      - loki

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    networks:
      - loki

Step-2: Prepare the NGINX reverse proxy with Basic Auth

Luckily I stumbled upon this cool open-source project which does the job for you :

So now what? How to use this?

You just have to give the required env vars and it will launch NGINX accordingly.

Those are:

1. BASIC_AUTH_USERNAME & BASIC_AUTH_PASSWORD => Basic Auth Creds
2. PROXY_PASS => where you want to forward your request from NGINX reverse proxy
3. PORT => The port, on which you want NGINX to listen.

Note: Here we will add our Loki URL in the PROXY_PASS section so it can forward our request to Loki.

Step-3: Combining all the pieces together

So Now, as we know how can we secure our Loki Installation, let's add this reverse proxy in our docker-compose.yaml file.

version: "3"

networks:
  loki:

services:
  loki:
    image: grafana/loki:2.6.1
    ports:
      - 3100
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      - loki

  promtail:
    image: grafana/promtail:2.6.1
    volumes:
      - /var/log:/var/log
    command: -config.file=/etc/promtail/config.yml
    networks:
      - loki

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    networks:
      - loki

  nginx:
    image: quay.io/dtan4/nginx-basic-auth-proxy
    environment:
      - BASIC_AUTH_USERNAME=loki_user
      - BASIC_AUTH_PASSWORD=loki_super_secret_password
      - PROXY_PASS=http://loki:3100
      - PORT=3200
      - SERVER_NAME=_
    ports:
      - 3200:3200
      - 8090:8090
    networks:
      - loki

Here I also disabled port-forwarding for Loki, because that is not required. Also, notice that in the PROXY_PASS env var, I gave loki:3100 which is the service name in the docker-compose file and is internally resolvable by the containers in the same docker network.

Here, we are running this NGINX server on port 3200 + we did port forwarding on the same port, so we can use localhost:3200 to connect to the NGINX server.

Also, port 8090 is used for NGINX metrics. You can check that on -> :8090/nginx_status

Step-4: See it in action!

Now, up this docker-compose file and see what happens at the Grafana end.

So, let's try to add a data source, just like before.

Trial-1: On old port 3100

As expected, we have disabled the port forwarding, it should fail.

Trial-2: With NGINX Port (w/o creds)

Again, this also fails as above.

Just for curiosity, let's curl and see.

Oh ok, let's try with the creds which we supplied to NGINX in docker-compose.

      - BASIC_AUTH_USERNAME=loki_user
      - BASIC_AUTH_PASSWORD=loki_super_secret_password

Trial-3: NGINX with Basic Auth Creds

Now using the same endpoint, Enable the Basic Auth, and add those basic auth credentials

hmm...

Voilà! - we were able to connect to our Loki Data source with an additional layer of security i.e. Basic Auth.

I know most of you are using Kubernetes out there and only a small set of users can relate to this(and implement it) but it is a really small fun exercise on how to use NGINX reverse proxy, setup Basic Auth, docker-compose(or docker swarm) inter-service communication and securing Loki installation. 😄

and the other reason to be happy is:

---

That was it! Let me know what you have to say in the comment box.

Thanks for reading :)

---

Reads:

1. grafana.com/docs/loki/latest/operations/aut..
2. docs.nginx.com/nginx/admin-guide/security-c..
3. bash script used in the NGINX docker image: github.com/dtan4/nginx-basic-auth-proxy/blo..
4. Also, the Dockerfile: github.com/dtan4/nginx-basic-auth-proxy/blo..