Search for a command to run...
Let's see how can you ace the logging setup for your containerized applications running with Docker!
![Best Logging Solution for Docker [Basic Version]](/_next/image?url=https%3A%2F%2Fcdn.hashnode.com%2Fres%2Fhashnode%2Fimage%2Fupload%2Fv1625404988207%2FhpCJimcOG.jpeg&w=3840&q=75)
Hey, Y'all 👋 Kratik Here :)
Hailing from The City of Lakes, India.
I love to learn and share stuff about DevOps, SRE, Security, Cost Optimization, etc.
Join me on this journey where we will learn together with the help bunch of interesting blogs.
No comments yet. Be the first to comment.
I have assumed that you have some basic knowledge of Docker & Docker Compose! What is Docker 🐬 ? An Open-Source tool that can be used to bundle your application with all the required dependencies. Similar to VMs but way more lightweight than it. Us...
DNS Search Domains, ndots, and Kubernetes Resolver Behavior
Understand how IAM Access Entries can replace aws-auth for managing EKS IAM authentication.
Effective Techniques for Providing IAM Access to EKS Pods
Ever wondered how can you create your own custom Github Action and publish it to the GitHub Marketplace? Let's learn it step by step in this blog.
Let's setup Vault with External-secrets to manage secrets in Kubernetes like never before!
Logs are important, right? While debugging something, It helps us more than anything.
Generally, we use commands to check logs of each container, for example,
docker logs -f container_1
and it is fine for a few containers, but suppose, you have a complex architecture, where you are using Docker Swarm or Docker-Compose or just assume you have a number of containers, then what?
Yes! It will be a real pain to maintain those logs, on top of that, other issues like No sharing of logs with the team, No Access-based control, etc. will also be there.
I bet the native Docker plugin kinda feeling similar as below right now :
Let's try a solution for that :)
We will be using Loki and Grafana to achieve our goal. Loki is a set of components that can be composed into a fully-featured logging stack. More info here and here
Before we jump into installation and all other intense stuff, let's see what benefits are we going to get by end of this installation:
Let's start the setup:
*you need to repeat these steps for each Docker node if you are using Docker Swarm
docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions
{
"debug" : false,
"log-driver": "loki",
"log-opts": {
"loki-url": "http://127.0.0.1:3100/loki/api/v1/push"
}
}
# Check the number of the nodes in your swarm
docker node ls
# Drain the node so the running workload can be shifted to other nodes.
docker node update --availability drain node1_name
# Restart the docker daemon on that node
systemctl restart docker
# Make the node active again
docker node update --availability active node1_name
To get the codebase, Clone my fork
git clone https://github.com/k4kratik/loki
loki-local-config.yamlauth_enabled: false
server:
http_listen_port: 3100
ingester:
lifecycler:
address: 127.0.0.1
ring:
kvstore:
store: inmemory
replication_factor: 1
chunk_idle_period: 15m
schema_config:
configs:
- from: 2019-06-01
store: boltdb
object_store: filesystem
schema: v9
index:
prefix: index_
period: 168h
table_manager:
.
.
.
# Notice this
retention_deletes_enabled: true
retention_period: 672h
...
Here we are setting up the auto-deletion of old logs after a certain period, which is 28 days (672 Hours) in our case.
docker stack deploy -c docker-compose.yml loki-stack
docker service ls
above stack file also contains a grafana instance, which will be available to YOUR_SERVER_IP:3000
Go to Grafana Dashboard, Initial credentials will be admin:admin
Go to Configuration > Data Sources as displayed in the picture.
Now Click on Add Data Source and enter the URL as http://loki:3100
If you have done everything correctly till this point, Clicking Save & Test will emit a success message.
❓ Just one question out of curiosity... We did not enter a IP or Domain Name, then How did it figure out our Loki instance correctly ? 🤔
The answer is : Check the docker-compose file, the loki service is named as loki, and other containers can reach to this container by the service name, which is in our case is loki.
Every container has by default a nameserver entry in /etc/resolve.conf, so it is not required to specify the address of the nameserver from within the container. That is why you can find your service from within the network with service or task_service_n.
Once you have added the Data Source, Go to Explore as instructed below:
Select a container to view its logs
Just like the image below, you can check logs of host, stdout & stderr streams, containers, Docker services, Docker stacks, etc. You can also combine multiple filter to get a consolidated output of your choice.
If you want to take this on a whole next level, well, there is good news for you, for the storage part you can also use Cloud DBs like DynamoDB for index and S3 for storing chunks. You can read storage options here.
My excited Indian ass dances like this whenever I complete any setup :) We all love that feeling, don't we?
Anyways, Thanks for reading and bearing all my content and The Office GIFs! Hope you got some knowledge from this blog, I am just writing what I am thinking, still trying to making it as simple and straightforward as possible.
