Here are a few things that we considered doing with Splunk:
Highlight BUILD SUCCESS and BUILD FAILURE outcomes.
Watch builds in realtime using complex event filters.
Benchmark parts of the build process.
Create Splunk Alerts that respond to specific error messages.
Search specifically for warnings that are often ignored.
Note: You don’t specifically need JSON to do these things with Splunk, but I’ve found that it makes constructing Splunk queries more rewarding.
Get a Maven Project
Although the Maven project you choose to build is not important here, Maven can quickly make one for you:
You will find that Maven has put a pom.xml, App.java, and AppTest.java in the appropriate places for a simple Java JAR project.
Make Some Noise If You’re With Me
The first thing we need are Maven logs.
Routing Maven output to a file in your home directory keeps things simple:
This command should complete with no output after a few seconds.
The JSON (and non-JSON) output generated by Maven are simply appended to ~/mvn.out. You can check for yourself:
Use jq to filter this into something more readable:
Getting Maven Build Output Into Splunk
Using Splunk to monitor your local build assumes that you are also using a local Splunk instance, and this makes the configuration of data inputs relatively simple using the Splunk web interface:
Login to the local Splunk web application
Navigate to Settings -> Data Inputs
Under Files & Directories click on +Add New
Enter the full path to your mvn.out file (ie: /Users/pcunningham/mvn.out) and click Next.
You will be able to preview the data here. Click Next.
Enter maven as the Name when it asks to Save Source Type.
Click Create New Index and use Index Namemaven.
Click Review and Submit and you are ready to start searching:
Use Splunk’s New Search to run the following query to see the results from your previous maven build:
Time Traveling With Splunk
Setting the Splunk time-range to Last 15 Minutes is probably sufficient to view your most recent results, but now that it’s in Splunk you can always go back in time to view them.
If you need to investigate that strange build warning you were having last Friday, Splunk will find it and everything else that happened in that build.
Use This Command To Build
Now anytime you are building a Maven project, use a command similar to this:
This uses the -B option with Maven to prevent the ANSI color encoded output in the msg field.
It uses the tee command which allows us to continue viewing the output on the console, while simultaneously writing it to ~/mvn.out.
Finally it uses the jq command to filter and pretty-print the JSON in a more readable format.
Watching Events in Real-Time in Splunk
If you change your Time Range to use one of the Real-Time settings, you can watch the build events appear as they are generated. Give Splunk a few seconds to catch up, but it works pretty well when everything is local.
Just do a Real-Time search using 5 minute window to get the latest output:
Building on Success
You can easily search in Splunk to see if it was successful:
If you change your time range to use Real-Time (5 minute window works well), your BUILD SUCCESS event will appear as soon as the build is finished. No need to watch everything going by.
Because your historical Maven output is now available, you can easily find out if your builds are getting longer:
From the events output by this search, you can find out the number of seconds each build took.
Looking for Errors and Warnings
You might be watching a particularly long build for errors, or may need to find errors in the build you completed over a week ago:
This will give you a quick breakdown of all ERRORs that were logged.
A search for WARNs might reveal the following example:
Very often these errors and warnings go by so fast that they avoid recognition and reconsiliation. Using Splunk to explicitly search for them provides the information in a more useful context and allows you to view them as a group and weigh the overall consequences of leaving them unresolved. Stats and charts are useful for illustrating their effects across multiple builds.
One More Thing
If you change your ~/.m2/logback.xml to this, your JSON will include two additional fieldsargs and project which can discern between different projects and their corresponding maven goals:
This logback configuration change uses the pattern provider to feed the environment variables MAVEN_CMD_LINE_ARGS and MAVEN_PROJECTBASEDIR via new fields named args and project.
With this extra bit of information, you can easily search for specific builds:
This would yield only events from the test application that were built with the clean goal.