t0t0 by Toon Lamberigts and Tomas Vercautter

During week 7 & 8 at Small Town Heroes, we researched and deployed a centralized logging system for our Docker environment. We use Fluentd to gather all logs from the other running containers, forward them to a container running ElasticSearch and display them by using Kibana. The result is similar to the ELK (Elasticsearch, Logstash, Kibana) stack, only we use Fluentd instead of Logstash.

One container to log them all

To manage logging Docker has implemented a system that makes use of Logging Drivers. By specifying the --log-driver in your docker run command your container will be configured to use this driver.

Docker provides these drivers for logging:

Basically if you would like to see the logs of a specific container, you’d have to use the docker logs command. However this command is only compatible with the json-file and journald logging drivers. Naturally this means that for displaying logs via another logging driver, you’d have to have other means of displaying these logs.
In any case it’s clear that there are a lot of options to choose from where most depends on personal choice.

The EFK stack

For our solution we went with the default JSON logging driver. This gives us the flexibility to still be able to use the docker logs command when we are logged in to the server as well as forwarding our logs to our log storage container. For collecting, storing and displaying the logs we ultimately went with the elasticsearch-fluentd-kibana or EFK stack.

One container to collect them all

So for collecting all the containers’ logs we chose to use Fluentd. At first we experimented with using the fluentd logging driver but then a few problems arised. First of all we can not start any containers when the fluentd container is not started due to the required --log-driver=fluentd. When executing the docker run command, this parameter will tell the container to use the Fluentd logging driver. As a result, the container will fail to start when there is no Fluentd container running.
Secondly, when the Fluentd container crashes/stops for some reason, all the logs for the period that it wasn’t running are lost. At this point the running containers are still sendig their logs to localhost:24224 (the default Fluentd port) yet there isn’t anything listening on that port, obviously.
Lastly, the other great disadvantage that the logging driver system brings, is the fact that all containers’ logs are being forwarded as “stdout” to a published port on the fluentd container. We want to be able to filter and format our containers’ logs to our own preferences, yet doing this with “stdout” output isn’t easy to say the least. The Fluentd logging driver is able to send the following metadata in the log message: container_id, container_name and source. In our case, this simply isn’t good enough. Plus, in the case of for example Nginx’s logs there are 2 types of log messages: access logs and error logs. Both types use a different logging format. Fluentd log driver will put the whole container’s log message in the field log: .... This log message needs still to be parsed to improve readability. Because access and error logs use a different format, you’ll have to use your regex magic skills to parse these messages appropriately.

To break this down, our reasons for not using the logging driver system:

• Using the logging driver system by Docker creates an extra dependency, something we particularly want to avoid.
• In case of failure, there is no option to recover logs.
• Filtering and formatting different types of logs is a serious pain to do when all logs are forwarded as stdout to the same port.

Alternatively, we have our Fluentd container tail the logs from the default JSON-file logs created by Docker. The obvious benefit of this is the fact that all log messages are neatly structured in a JSON format. Also, Fluentd using a “tail” source type is able to store its current position in a position file to keep track of which log messages he has already tailed and forwarded. When the Fluentd container would stop running, it can just read the position file and continue after it has been restarted. Even while the Fluentd container isn’t running, containers’ logs are still being saved on the Docker machine. In case of failure, this proves to be an easy back-up solution.

This was the config file we started with.

<source>
  type tail
  read_from_head true
  pos_file fluentd-docker.pos
  path /var/lib/docker/containers/*/*-json.log
  time_format %Y-%m-%dT%H:%M:%S
  tag docker.*
  format json
</source>

<match docker.**>
  type elasticsearch
  log_level info
  host elastic
  port 9200
  include_tag_key true 
  logstash_format true
  flush_interval 5s
</match>

With this config file, all that Fluentd does is tail the JSON log files from the Docker directory, store its current position in the file fluentd-docker.pos and finally tag them with the prefix “docker”. Lastly all the logs with this prefix (so all of them) are sent to the Elasticsearch container. To make the log messages collected by Fluentd compatible with Elasticsearch, we use fluent-plugin-elasticsearch.
This configuration had almost everything we need except we had no idea what exact container the logs were coming from, the only thing added was the long container id string (which is coming from the filepath of the specific log).
Specifically, we wanted to have the hostname we originally set when starting the containers. We started using the module fluent-plugin-docker_metadata_filter to provide us with the additional Docker metadata. We ran into a problem with the version on gem repository not being the latest version (and containing a bug) so we had to install it this way.

git clone https://github.com/fabric8io/fluent-plugin-docker_metadata_filter && \
      cd fluent-plugin-docker_metadata_filter && \
      gem build fluent-plugin-docker_metadata_filter.gemspec && \
      gem install --no-ri --no-rdoc fluent-plugin-docker_metadata_filter

After installing the ruby gem all we had to do was add this snippet to our fluentd config file:

<filter docker.var.lib.docker.containers.*.*.log>
  type docker_metadata
</filter>

Now we have logs with the added docker metadata for that container. The added Docker metadata is this:

• Container ID
• Container name
• Container hostname
• Image name
• Image ID
• Labels

One container to store them all

As mentioned before, to store all the container’s logs we went with an Elasticsearch container. Elasticsearch isn’t really a regular database but actually a search server based on Apache Lucene. Basically Elasticsearch is a “cluster of nodes”. A cluster being a collection of one or more servers (a.k.a. nodes) that contains the entire data. The strong point of Elasticsearch which makes it so popular is the fact that it provides a scalable, nearreal-time search. Next to that, it’s also massively distributed. This means that data can be divided into shards (or pieces of data) and each shard can have multiple replicas while each node contains one or more shards. Thus Elasticsearch combines the commonly known practices of “sharding” and “replication”. If you’re not familiar with either of those, this is why they are important:

• Sharding
• horizontally split/scale your content volume
• distribute and parallelize operations across shards (potentially on multiple nodes) thus increasing performance/throughput
• Replication
• high availability in case a shard/node fails
• scale out your search volume/throughput since searches can be executed on all replicas in parallel

One container to show them all

Logically, for displaying our containers’ logs we chose Kibana. Kibana is also a product of Elastic just like Elasticsearch which means the two work together seamlessly. There wasn’t really much configuration needed here. Basically we just specified a --link with our Elastic container in our docker run command for the Kibana container.
The only thing we added here was means of authentication. There are a lot of complicated ways to do this but we chose to simply add authentication through our nginx. But because we are using docker-gen for this as explained in our previous blog post, we had to add authentication to our template. This turned out to be rather simple just by adding the next few lines of code inside the ‘location /’ block

		{{ if hasPrefix "logging-stage" $host }}
		auth_basic	"Restricted {{ $host }}";
		auth_basic_user_file	{{ (printf "/etc/nginx/htpasswd/%s" $host) }};
		{{ end }}

What this does is checking if the ‘VIRTUAL_HOST’ variable has a prefix logging-stage and if so, add basic authentication to that location block. The password file is premade and can be copied when building the docker image or just add a volume mounting the file on the desired location.