elk-docker-ec2-discovery

Sebastien Pujadas' ELK docker image with the discovery plugin installed

View the Project on GitHub

Elasticsearch, Logstash, Kibana (ELK) Docker image documentation

This web page documents how to use the sebp/elk Docker image, which provides a convenient centralised log server and log management web interface, by packaging Elasticsearch, Logstash, and Kibana, collectively known as ELK.

Contents

Prerequisites

You need:

  1. An amazon AWS account with web-console and/or API keys
  2. IAM user with permissions to EC2, ECS.

Port configuration - Security Groups, nACL, etc.

Installation

To pull this image from the Docker registry, open a shell prompt and enter:

$ sudo docker pull sebp/elk

Note – This image has been built automatically from the source files in the source Git repository on GitHub. If you want to build the image yourself, see the Building the image section.

This image has only been tested on ELK version 5.4.0.

Usage

Note – The whole ELK stack will be started. See the Starting services selectively section to selectively start part of the stack.

This command publishes the following ports, which are needed for proper operation of the ELK stack:

Note – The image also exposes Elasticsearch’s transport interface on port 9300. Use the -p 9300:9300 option with the docker command above to publish it. This transport interface is notably used by Elasticsearch’s Java client API, and to run Elasticsearch in a cluster.

Starting services selectively

By default, when starting a container, all three of the ELK services (Elasticsearch, Logstash, Kibana) are started.

The following environment variables may be used to selectively start a subset of the services:

For example, the following command starts Elasticsearch only:

$ sudo docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -it \
	-e LOGSTASH_START=0 -e KIBANA_START=0 --name elk sebp/elk

Note that if the container is to be started with Elasticsearch disabled, then:

Overriding start-up variables

The following environment variables can be used to override the defaults used to start up the services:

As an illustration, the following command starts the stack, running Elasticsarch with a 2GB heap size, Logstash with a 1GB heap size and Logstash’s configuration auto-reload disabled:

$ sudo docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -it \
	-e ES_HEAP_SIZE="2g" -e LS_HEAP_SIZE="1g" -e LS_OPTS="--no-auto-reload" \
	--name elk sebp/elk

Forwarding logs

Forwarding logs from a host relies on a forwarding agent that collects logs (e.g. from log files, from the syslog daemon) and sends them to our instance of Logstash.

Forwarding logs with Filebeat

Install Filebeat on the host you want to collect and forward logs from (see the References section for links to detailed instructions).

Note – Make sure that the version of Filebeat is the same as the version of the ELK image.

Example Filebeat set-up and configuration

Note – The nginx-filebeat subdirectory of the source Git repository on GitHub contains a sample Dockerfile which enables you to create a Docker image that implements the steps below.

Here is a sample /etc/filebeat/filebeat.yml configuration file for Filebeat, that forwards syslog and authentication logs, as well as nginx logs.

output:
  logstash:
    enabled: true
    hosts:
      - elk:5044
    ssl:
	  certificate_authorities:
  	    - /etc/pki/tls/certs/logstash-beats.crt
    timeout: 15

filebeat:
  prospectors:
    -
      paths:
        - /var/log/syslog
        - /var/log/auth.log
      document_type: syslog
    -
      paths:
        - "/var/log/nginx/*.log"
      document_type: nginx-access

In the sample configuration file, make sure that you replace elk in elk:5044 with the hostname or IP address of the ELK-serving host.

You’ll also need to copy the logstash-beats.crt file (which contains the certificate authority’s certificate – or server certificate as the certificate is self-signed – for Logstash’s Beats input plugin; see Security considerations for more information on certificates) from the source repository of the ELK image to /etc/pki/tls/certs/logstash-beats.crt.

Note – Alternatively, when using Filebeat on a Windows machine, instead of using the certificate_authorities configuration option, the certificate from logstash-beats.crt can be installed in Windows’ Trusted Root Certificate Authorities store.

Note – The ELK image includes configuration items (/etc/logstash/conf.d/11-nginx.conf and /opt/logstash/patterns/nginx) to parse nginx access logs, as forwarded by the Filebeat instance above.

Before starting Filebeat for the first time, run this command (replace elk with the appropriate hostname) to load the default index template in Elasticsearch:

	curl -XPUT 'http://elk:9200/_template/filebeat?pretty' -d@/etc/filebeat/filebeat.template.json

Start Filebeat:

	sudo /etc/init.d/filebeat start

Building the image

To build the Docker image from the source files, first clone the Git repository, go to the root of the cloned directory (i.e. the directory that contains Dockerfile), and:

Tweaking the image

There are several approaches to tweaking the image:

The next few subsections present some typical use cases.

Updating Logstash’s configuration

The image contains several configuration files for Logstash (e.g. 01-lumberjack-input.conf, 02-beats-input.conf), all located in /etc/logstash/conf.d.

To modify an existing configuration file, you can bind-mount a local configuration file to a configuration file within the container at runtime. For instance, if you want to replace the image’s 30-output.conf Logstash configuration file with your local file /path/to/your-30-output.conf, then you would add the following -v option to your docker command line:

$ sudo docker run ... \
	-v /path/to/your-30-output.conf:/etc/logstash/conf.d/30-output.conf \
	...

To create your own image with updated or additional configuration files, you can create a Dockerfile that extends the original image, with contents such as the following:

FROM sebp/elk

# overwrite existing file
ADD /path/to/your-30-output.conf /etc/logstash/conf.d/30-output.conf

# add new file
ADD /path/to/new-12-some-filter.conf /etc/logstash/conf.d/12-some-filter.conf

Then build the extended image using the docker build syntax.

Installing Elasticsearch plugins

Elasticsearch’s home directory in the image is /opt/elasticsearch, its plugin management script (elasticsearch-plugin) resides in the bin subdirectory, and plugins are installed in plugins.

Elasticsearch runs as the user elasticsearch. To avoid issues with permissions, it is therefore recommended to install Elasticsearch plugins as elasticsearch, using the gosu command (see below for an example, and references for further details).

A Dockerfile like the following will extend the base image and install the GeoIP processor plugin (which adds information about the geographical location of IP addresses):

FROM sebp/elk

ENV ES_HOME /opt/elasticsearch
WORKDIR ${ES_HOME}

RUN gosu elasticsearch bin/elasticsearch-plugin install \
    -Edefault.path.conf=/etc/elasticsearch ingest-geoip

You can now build the new image (see the Building the image section above) and run the container in the same way as you did with the base image.

Installing Logstash plugins

The name of Logstash’s home directory in the image is stored in the LOGSTASH_HOME environment variable (which is set to /opt/logstash in the base image). Logstash’s plugin management script (logstash-plugin) is located in the bin subdirectory.

Logstash runs as the user logstash. To avoid issues with permissions, it is therefore recommended to install Logstash plugins as logstash, using the gosu command (see below for an example, and references for further details).

The following Dockerfile can be used to extend the base image and install the RSS input plugin:

FROM sebp/elk

WORKDIR ${LOGSTASH_HOME}
RUN gosu logstash bin/logstash-plugin install logstash-input-rss

See the Building the image section above for instructions on building the new image. You can then run a container based on this image using the same command line as the one in the Usage section.

Installing Kibana plugins

The name of Kibana’s home directory in the image is stored in the KIBANA_HOME environment variable (which is set to /opt/kibana in the base image). Kibana’s plugin management script (kibana-plugin) is located in the bin subdirectory, and plugins are installed in installedPlugins.

Kibana runs as the user kibana. To avoid issues with permissions, it is therefore recommended to install Kibana plugins as kibana, using the gosu command (see below for an example, and references for further details).

The following Dockerfile can be used to extend the base image and install the latest version of the Sense plugin, a handy console for interacting with the REST API of Elasticsearch:

FROM sebp/elk

WORKDIR ${KIBANA_HOME}
RUN gosu kibana bin/kibana-plugin install elastic/sense

See the Building the image section above for instructions on building the new image. You can then run a container based on this image using the same command line as the one in the Usage section. The Sense interface will be accessible at http://<your-host>:5601/apss/sense (e.g. http://localhost:5601/app/sense for a local native instance of Docker).

Persisting log data

In order to keep log data across container restarts, this image mounts /var/lib/elasticsearch — which is the directory that Elasticsearch stores its data in — as a volume.

You may however want to use a dedicated data volume to persist this log data, for instance to facilitate back-up and restore operations.

One way to do this is to mount a Docker named volume using docker’s -v option, as in:

$ sudo docker run -p 5601:5601 -p 9200:9200  -p 5044:5044 \
	-v elk-data:/var/lib/elasticsearch --name elk sebp/elk

This command mounts the named volume elk-data to /var/lib/elasticsearch (and automatically creates the volume if it doesn’t exist; you could also pre-create it manually using docker volume create elk-data).

Note – By design, Docker never deletes a volume automatically (e.g. when no longer used by any container). Whilst this avoids accidental data loss, it also means that things can become messy if you’re not managing your volumes properly (e.g. using the -v option when removing containers with docker rm to also delete the volumes… bearing in mind that the actual volume won’t be deleted as long as at least one container is still referencing it, even if it’s not running). You can keep track of existing volumes using docker volume ls.

See Docker’s page on Managing Data in Containers and Container42’s Docker In-depth: Volumes page for more information on managing data volumes.

In terms of permissions, Elasticsearch data is created by the image’s elasticsearch user, with UID 991 and GID 991.

There is a known situation where SELinux denies access to the mounted volume when running in enforcing mode. The workaround is to use the setenforce 0 command to run SELinux in permissive mode.

Setting up an Elasticsearch cluster

The ELK image can be used to run an Elasticsearch cluster, either on separate hosts or (mainly for test purposes) on a single host, as described below.

For more (non-Docker-specific) information on setting up an Elasticsearch cluster, see the Life Inside a Cluster section section of the Elasticsearch definitive guide.

Running Elasticsearch nodes on different hosts

To run cluster nodes on different hosts, you’ll need to update Elasticsearch’s /etc/elasticsearch/elasticsearch.yml file in the Docker image so that the nodes can find each other:

As an example, start an ELK container as usual on one host, which will act as the first master. Let’s assume that the host is called elk-master.example.com.

Have a look at the cluster’s health:

$ curl http://elk-master.example.com:9200/_cluster/health?pretty
{
  "cluster_name" : "elasticsearch",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 6,
  "active_shards" : 6,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 6,
  "delayed_unassigned_shards" : 6,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 50.0
}

This shows that only one node is up at the moment, and the yellow status indicates that all primary shards are active, but not all replica shards are active.

Then, on another host, create a file named elasticsearch-slave.yml (let’s say it’s in /home/elk), with the following contents:

network.host: 0.0.0.0
network.publish_host: <reachable IP address or FQDN>
discovery.zen.ping.unicast.hosts: ["elk-master.example.com"]

You can now start an ELK container that uses this configuration file, using the following command (which mounts the configuration files on the host into the container):

$ sudo docker run -it --rm=true -p 9200:9200 -p 9300:9300 \
  -v /home/elk/elasticsearch-slave.yml:/etc/elasticsearch/elasticsearch.yml \
  sebp/elk

Once Elasticsearch is up, displaying the cluster’s health on the original host now shows:

$ curl http://elk-master.example.com:9200/_cluster/health?pretty
{
  "cluster_name" : "elasticsearch",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 6,
  "active_shards" : 12,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Running Elasticsearch nodes on a single host

Setting up Elasticsearch nodes to run on a single host is similar to running the nodes on different hosts, but the containers need to be linked in order for the nodes to discover each other.

Start the first node using the usual docker command on the host:

$ sudo docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -it --name elk sebp/elk

Now, create a basic elasticsearch-slave.yml file containing the following lines:

network.host: 0.0.0.0
discovery.zen.ping.unicast.hosts: ["elk"]

Start a node using the following command:

$ sudo docker run -it --rm=true \
  -v /var/sandbox/elk-docker/elasticsearch-slave.yml:/etc/elasticsearch/elasticsearch.yml \
  --link elk:elk --name elk-slave sebp/elk

Note that Elasticsearch’s port is not published to the host’s port 9200, as it was already published by the initial ELK container.

Optimising your Elasticsearch cluster

You can use the ELK image as is to run an Elasticsearch cluster, especially if you’re just testing, but to optimise your set-up, you may want to have:

An even more optimal way to distribute Elasticsearch, Logstash and Kibana across several nodes or hosts would be to run only the required services on the appropriate nodes or hosts (e.g. Elasticsearch on several hosts, Logstash on a dedicated host, and Kibana on another dedicated host).

Security considerations

As it stands this image is meant for local test use, and as such hasn’t been secured: access to the ELK services is unrestricted, and default authentication server certificates and private keys for the Logstash input plugins are bundled with the image.

To harden this image, at the very least you would want to:

The sebp/elkx image, which extends the ELK image with X-Pack, may be a useful starting point to improve the security of the ELK services.

If on the other hand you want to disable certificate-based server authentication (e.g. in a demo environment), see Disabling SSL/TLS.

Notes on certificates

Dummy server authentication certificates (/etc/pki/tls/certs/logstash-*.crt) and private keys (/etc/pki/tls/private/logstash-*.key) are included in the image.

Note – For Logstash 2.4.0 a PKCS#8-formatted private key must be used (see Breaking changes for guidance).

The certificates are assigned to hostname *, which means that they will work if you are using a single-part (i.e. no dots) domain name to reference the server from your client.

Example – In your client (e.g. Filebeat), sending logs to hostname elk will work, elk.mydomain.com will not (will produce an error along the lines of x509: certificate is valid for *, not elk.mydomain.com), neither will an IP address such as 192.168.0.1 (expect x509: cannot validate certificate for 192.168.0.1 because it doesn't contain any IP SANs).

If you cannot use a single-part domain name, then you could consider:

The following commands will generate a private key and a 10-year self-signed certificate issued to a server with hostname elk for the Beats input plugin:

$ cd /etc/pki/tls
$ sudo openssl req -x509 -batch -nodes -subj "/CN=elk/" \
	-days 3650 -newkey rsa:2048 \
	-keyout private/logstash-beats.key -out certs/logstash-beats.crt

As another example, when running a non-predefined number of containers concurrently in a cluster with hostnames directly under the .mydomain.com domain (e.g. elk1.mydomain.com, elk2.mydomain.com, etc.; not elk1.subdomain.mydomain.com, elk2.othersubdomain.mydomain.com etc.), you could create a certificate assigned to the wildcard hostname *.example.com by using the following command (all other parameters are identical to the ones in the previous example).

$ cd /etc/pki/tls
$ sudo openssl req -x509 -batch -nodes -subj "/CN=*.example.com/" \
	-days 3650 -newkey rsa:2048 \
	-keyout private/logstash-beats.key -out certs/logstash-beats.crt

To make Logstash use the generated certificate to authenticate to a Beats client, extend the ELK image to overwrite (e.g. using the Dockerfile directive ADD):

Additionally, remember to configure your Beats client to trust the newly created certificate using the certificate_authorities directive, as presented in Forwarding logs with Filebeat.

Disabling SSL/TLS

Certificate-based server authentication requires log-producing clients to trust the server’s root certificate authority’s certificate, which can be an unnecessary hassle in zero-criticality environments (e.g. demo environments, sandboxes).

To disable certificate-based server authentication, remove all ssl and ssl-prefixed directives (e.g. ssl_certificate, ssl_key) in Logstash’s input plugin configuration files.

For instance, with the default configuration files in the image, replace the contents of 02-beats-input.conf (for Beats emitters) with:

input {
  beats {
    port => 5044
  }
}

Frequently encountered issues

Elasticsearch is not starting (1): max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]

If the container stops and its logs include the message max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144], then the limits on mmap counts are too low, see Prerequisites.

Elasticsearch is not starting (2): cat: /var/log/elasticsearch/elasticsearch.log: No such file or directory

If Elasticsearch’s logs are not dumped (i.e. you get the following message: cat: /var/log/elasticsearch/elasticsearch.log: No such file or directory), then Elasticsearch did not have enough memory to start, see Prerequisites.

Elasticsearch is not starting (3): bootstrap tests

As from version 5, if Elasticsearch is no longer starting, i.e. the waiting for Elasticsearch to be up (xx/30) counter goes up to 30 and the container exits with Couln't start Elasticsearch. Exiting. and Elasticsearch’s logs are dumped, then read the recommendations in the logs and consider that they must be applied.

In particular, in case (1) above, the message max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144] means that the host’s limits on mmap counts must be set to at least 262144.

Elasticsearch is suddenly stopping after having started properly

With the default image, this is usually due to Elasticsearch running out of memory after the other services are started, and the corresponding process being (silently) killed.

As a reminder (see Prerequisites), you should use no less than 3GB of memory to run the container… and possibly much more.

Troubleshooting

Important – If you need help to troubleshoot the configuration of Elasticsearch, Logstash, or Kibana, regardless of where the services are running (in a Docker container or not), please head over to the Elastic forums. The troubleshooting guidelines below only apply to running a container using the ELK Docker image.

Here are a few pointers to help you troubleshoot your containerised ELK.

AWS ECS Logging (drivers)

Reporting issues

Important – For non-Docker-related issues with Elasticsearch, Kibana, and Elasticsearch, report the issues on the appropriate Elasticsearch, Logstash, or Kibana GitHub repository.

You can report issues with this image using GitHub’s issue tracker.

Bearing in mind that the first thing I’ll need to do is reproduce your issue, please provide as much relevant information (e.g. logs, configuration files, what you were expecting and what you got instead, any troubleshooting steps that you took, what is working) as possible for me to do that.

Pull requests are also welcome if you have found an issue and can solve it.

References

About

Written by Sébastien Pujadas, released under the Apache 2 license.

Modified by eternaltyro.