Running Splunk In Docker is about as easy as it should be

You have fully embraced the world of Docker hosted containers, and no longer have the patience for local installations of “Enterprise” software that requires a developer account to download. Fortunately, there’s a Splunk Enterprise Docker Image just for you. It even comes preconfigured with a 500M per day developer license. No account required!

Starting Splunk in Docker

The Splunk Enterprise Docker Image makes installation as simple as this:

$ docker run -p "8000:8000" -p "8088:8088" -p "8089:8089" --env SPLUNK_USER="root" --env SPLUNK_START_ARGS="--accept-license --seed-passwd passw0rd" splunk/splunk

This tells Docker to start up an instance of splunk/splunk:latest using its default ports 8000 for the web interface, 8088 for the HTTP Event Collector, and 8089 for the Splunk API services.

You will see some output to the console as Splunk is starting up…

...
	
Generating a 2048 bit RSA private key
..............+++
................................................+++
writing new private key to 'privKeySecure.pem'
-----
Signature ok
subject=/CN=a6dbb1181186/O=SplunkUser
Getting CA Private Key
writing RSA key
	All installed files intact.
	Done
All preliminary checks passed.
	
Starting splunk server daemon (splunkd)...
Done
	
	
Waiting for web server at http://127.0.0.1:8000 to be available.... Done
	
	
If you get stuck, we're here to help.
Look for answers here: http://docs.splunk.com
	
The Splunk web interface is at http://a6dbb1181186:8000

You can login at http://localhost:8000 with username admin and password passw0rd.

Create a Splunk HTTP Event Collector for Docker Logging

Before another Docker container can log directly to Splunk, you need to create a new HTTP Event Collector that listens for logging events on port 8088:

  1. Login to the Splunk web interface at: http://localhost:8000 using the values for admin and passw0rd for the username and password.
  2. Navigate to the Settings → Data Inputs → HTTP Event Collector page and click the New Token button.
  3. Enter the Name docker and click the Next button.
  4. Under the Index section, click the Create a new index link.
  5. Enter the Index Name docker and click the Save button.
  6. The Default Index will now read docker. Click the Review button.
  7. Review these settings and click the Submit button.
  8. You will then read Token has been created successfully and can copy the hexadecimal Token Value to your clipboard for use in subsequent docker run commands…
  9. Finally, navigate to Settings → Data Inputs → HTTP Event Collector and click the Global Settings button.
  10. Select All Tokens Enabled and click Save. This enables the HTTP Event Collector on port 8088.

That sounds like a lot of work, but it’s almost entirely default values and you will only need to do this once after creating a new Splunk container.

Log Another Docker Container Directly To Splunk

Once the Splunk HTTP Event Collector is enabled and you have a valid Token Value for logging, it’s comparatively easy for Docker to log the rest of your containers directly to it.

Runing a command similar to the following tells Docker to log the output from your container directly to Splunk:

$ docker run --log-driver=splunk --log-opt splunk-token=2320e24a-06e1-4993-ad0b-56d49e6541f4 --log-opt splunk-url=https://11.22.33.44:8088 --log-opt splunk-format=raw --log-opt tag="" --log-opt splunk-insecureskipverify=true alpine echo "{\"msg\":\"hello from alpine\"}"

The docker run command supports the --log-driver and --log-opt switches which allow us to tell Docker how to handle logging:

  • --log-driver=splunk tells Docker to send logs to Splunk’s HTTP Event Collector.
  • --log-opt splunk-token is where we put the Token Value we created for logging.
  • --log-opt splunk-url is the HTTPS link to the Splunk HTTP Collector on port 8088.
  • --log-opt splunk-format=raw tells Docker to send the raw text from our logs.
  • --log-opt splunk-insecureskipverify=true tells Docker to skip cert verification.

That’s a lot of options, but it’s everything you’ll need to get those logs sent directly to Splunk for further analysis. Use the host IP address of the docker server, or an appropriate DNS entry (don’t use localhost). Docker will exit immediately if this connection fails, providing you with a hint that you may not have completed step 10 above.

A logging connection failure looks like this upon executing the docker command:

docker: Error response from daemon: failed to initialize logging driver: Options https://11.22.33.44:8088/services/collector/event/1.0: EOF.
ERRO[0000] error waiting for container: context canceled

Start Searching in Splunk

If you were able to start up a Docker container using all of the Splunk logging options in the previous section, the logging output from that container is now searchable in Splunk.

Try this simple search to see what events have been received:

index="docker"

If you ran the alpine example above, you will see the JSON-based output from it in the results:

{ [-]
  "msg":"hello from alpine"
}

Although Splunk handles JSON output exceptionally well, any text output from a logged container will be entered as a searchable event.

References