Sending Docker Container Logs to Azure Application Insights Using GELF Driver

Docker has a default logging driver, which each container uses. However, you can configure the Docker daemon, or for that matter a Docker container to use a different logging driver. One such driver is Graylog Extended Format logging driver. The Graylog Extended Log Format (GELF) is understood by a number of tools such as Graylog, Logstash, Fluentd, etc. But you can write your own GELF server, which is basically an UDP server. Once you get the GELF message in the UDP server, you can send the message to where you want and for this post, I use Azure Application Insights. Of course, you can create your own Docker logging driver too but I use the GELF driver because it is less work to do! I do not know what the legal/license implications are, in terms of using the GELF driver With Azure Application Insights but the point of this post is to illustrate what is possible.

First, you need an UDP server. I’ll use a simple .NET Core Console App for this. I have added the following NuGet packages to the project: Newtonsoft.Json and Microsoft.ApplicationInsights.

using Microsoft.ApplicationInsights;
using Microsoft.ApplicationInsights.Extensibility;
using Newtonsoft.Json;
using System;
using System.Collections.Generic;
using System.Net;
using System.Net.Sockets;
using System.Text;

namespace UdpServer
{
    class Program
    {
        private const int listenPort = 11000;

        static void Main(string[] args)
        {
            bool done = false;

            TelemetryConfiguration.Active.InstrumentationKey = "<Copy-paste the key here>";
            TelemetryClient client = new TelemetryClient();

            UdpClient listener = new UdpClient(listenPort);
            IPEndPoint ep = new IPEndPoint(IPAddress.Any, listenPort);

            try
            {
                Console.WriteLine("Starting UDP Listener...");

                while (!done)
                {
                    byte[] bytes = listener.Receive(ref ep);

                    string message = Encoding.UTF8.GetString(bytes, 0, bytes.Length);

                    Console.WriteLine(message);

                    var props = JsonConvert
                                  .DeserializeObject<Dictionary<string, string>>(message);
                    client.TrackEvent("docker", props);
                    client.Flush();
                }
            }
            catch (Exception e)
            {
                client.TrackException(e);
            }
            finally
            {
                listener.Close();
            }
        }
    }
}

You can build and run this .NET Console App in the Linux instance. Do install .NET Core, if you haven’t done so already. The preceding code receives the GELF message, which is a JSON (see below) and converts that into a dictionary and sends that to Azure Application Insight as an event.

{
	"version": "1.1",
	"host": "ip-10-0-0-220",
	"short_message": "hello world",
	"timestamp": 1520656966.257,
	"level": 6,
	"_command": "echo hello world",
	"_container_id": "a717f570d503566d2c6e60800d6ce676f81a68b1bc241bb3e8be25f005d7c6a5",
	"_container_name": "keen_nightingale",
	"_image_name": "alpine",
	"_tag": "a717f570d503",
        ...
}

The final part is to run a Docker container. I use alpine to just do a hello world. As I run the container, I specify the GELF driver and the options. Here, my UDP listener runs on the IP 10.0.0.220 and the port 11000. Since the UDP listener is a trivial one, I don’t use any compression. By default, GELF uses compression and to make it not use compression, I use gelf-compression-type=none.

sudo docker run \
      --log-driver gelf
      --log-opt gelf-address=udp://10.0.0.220:11000 \
      --log-opt gelf-compression-type=none \
      alpine echo hello world

With that, you should now see the event appearing in Azure Application Insights (after a time lag) with the event name of “docker” and a bunch of custom data that we sent in the dictionary.

_command echo hello world
_container_id a717f570d503566d2c6e60800d6ce676f81a68b1bc241bb3e8be25f005d7c6a5
_container_name keen_nightingale
_image_id sha256:3fd9065eaf02feaf94d68376da52541925650b81698c53c6824d92ff63f98353
_image_name alpine
short_message hello world

Now, I run an NGINX container.

sudo docker pull nginx
sudo docker run -p 80:80 \
           --log-driver gelf \
           --log-opt gelf-address=udp://10.0.0.220:11000 \
           --log-opt gelf-compression-type=none \
           nginx

If you now make an HTTP GET to NGINX home page, you can see NGINX logs appearing in Azure Application Insights as well.

_command nginx -g daemon off;
_container_id 71eac7120881722ab8695e7f6147985a9a6c7c65f5516fcf8a19596ea36ad699
_container_name jovial_shockley
_image_id sha256:e548f1a579cf529320a3c1d8789399e5fea0bfaa04cbb70d03890afafb748a2f
_image_name nginx
short_message 11.111.11.1 – – [10/Mar/2018:04:47:16 +0000] “GET / HTTP/1.1” 200 612 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36” “-“
Advertisements

One thought on “Sending Docker Container Logs to Azure Application Insights Using GELF Driver

  1. Excellent Badri. I have done a similar thing with Kubernetes also where I was pumping logs from POD. Point to note that this will work with Docker ( and docker compose and may be docker awarm) . But with Kubernetes they have a different set of approach.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.