Introduction

As described in my last post Sending Gelf messages to Graylog2 via PowerShell we were testing Graylog2 for log and meter collection. It soon became clear that the current selection of adapters to get data out of Graylog2 was far from perfect. We needed a specific output action to send data to an incident management system that could only be automated by a command line tool. So I decided to create the SCRIPT OUTPUT Plugin for Graylog2 (which you can download from our GitHub account) and which I will quickly present in this post.

As it is written on Writing plugins, Graylog2 offers an extensible plugin model which you can use to create additional output adapters (and other types) tailored to your needs. Documentation turned out to be a little bit sparse, but luckily there is the Pagerduty AlarmCallback Plugin for reference on their GitHub account. With Maven and a Graylog2 Plugin archetype to the rescue it seemed fairly easy to get on the road to my first Graylog2 output plugin …

Here is a brief overview of what is covered:

• The Tools

• Development

• Deployment

• Testing

• API View

• Download and Notes

The Tools

Here is a quick summary of what I used as the development environment (feel free to skip if you are not interested):

• A CentOS 7.0 virtual machine with fairly little resources (2 CPU, 4 GB RAM)

• IntelliJ IDEA for my development IDE as recommended by Lennart Koopmann

• The Graylog2 Plugin Archetype

• A Graylog2 instance based on Docker from the official Graylog2 Docker image definition (and a quick docker pull graylog2/allinone as described in my previous post)

• A PowerShell load script to insert messages into Graylog2

• drone.io for an automatic build of the plugin
  Note: I am currently testing drone.io as a CI environment. To be on the safe side, build the source on your own trusted platform.

Development

• Creating the project
  I created the skeleton interactively via mvn archetype:generate (and used a not so clever naming for my plugin):

  mvn archetype:generate -DarchetypeGroupId=org.graylog2 -DarchetypeArtifactId=graylog2-plugin-archetype
  
  groupId: biz.dfch.j.graylog2.plugin.output
  artifactId: execscript
  version: 1.0.0-SNAPSHOT
  package: biz.dfch.j.graylog2.plugin.output
  pluginClassName: dfchBizExecScript
  

UPDATE 2015-01-22 Beware that you use the correct version of the Archetype (depending on the Graylog2 you are using as there is currently some ‘motion’ (see also the discussion at Crash on launch input in 0.91.3 (and higher)). For the output plugin this is not really an issue, but if something goes wrong or the plugin does not load, make sure you check this as well.

After that I opened the project from IntelliJ IDEA and set a JDK path (using OpenJDK 1.7)

• Implementing the interface
  As the PagerDuty plugin was not an Output plugin I had to look around to see which messages had to be overriden to make it work. Essentially I had to override these methods from MessageOutput:

  public interface MessageOutput {
      void initialize(Configuration var1) throws MessageOutputConfigurationException;
  
      void stop();
  
      boolean isRunning();
  
      void write(Message var1) throws Exception;
  
      void write(List<Message> var1) throws Exception;
  
      ConfigurationRequest getRequestedConfiguration();
  
      String getName();
  
      String getHumanName();
  
      String getLinkToDocs();
  }
  

The most important methods is write where you actually place your functionality what your plugin should do.

It only took me three attempts to get the methods right and upon second try -to my surprise- it already showed up as an output module in Graylog2.

• Filling in the blanks
  For ease of implementation I used the ScriptEngine class to execute my javascript files (that in turn would trigger the aforementioned command line tool). I also inserted the processed message into the session context, so the script could happily grab the pieces it actually needed for forwarding:

  root@50167a4a0370:/opt/graylog2# cat /opt/graylog2/plugin/helloworld.js
  // message has custom fields exposed as properties
  // standard GELF fields are properties directly on the message object
  print(message + "rn");
  print(message.message + "rn");
  print(message.source + "rn");
  print(message.getField("guid") + "rn");
  

UPDATE 2015-03-31: I now bundled the Jython artifact into the plugin, so you can also call and execute Python scripts.

Deployment

As it is rather difficult to copy data from a host into a running Docker container, I used NetCat to pipe the resulting JAR into the container (followed by a restart of the Graylog2 service):

edgar@CONTAINER:
nc -l 10101 > /opt/graylog2/plugin/DfchBizTestOutput-1.0.0-SNAPSHOT.jar

edgar@HOST:
cat target/DfchBizTestOutput-1.0.0-SNAPSHOT.jar | nc 172.17.0.7 10101

Note: see the Notes at the end of the post on how to automatically integrate the plugin into Graylog2.

Testing

• Message Input
  I created a GELF TCP input and inserted data via Insert-GelfTcp.ps1

• Stream Definition
  I defined a stream that matched on every message with a guid field

• Message Output
  I instantiated my shiny new Output plugin, associated it with the stream and had it execute a javascript file in the container with its output being mirrored on the console

• So for every message sent into the system the plugin would execute the javascript file – a guarantee for a (s)low performing system …

API View of Configuration

Everything I configured via the web interface could also be checked and retrieved via the REST API:

• Output Definition
  Here you can see if your plugin is visible and available for use:
  

• Stream Definition
  The defined stream shows the associated output:
  

Download

• You can download the SCRIPT OUTPUT Plugin source at our GitHub account

• You can also download the binary at our drone.io account. This gets built every time we commit something to the master branch of the repository.

• The plugin is LICENSED under Apache 2.0 so you can freely use and modify it

Notes

• Real documentation except for the single page about plugins is really hard to find (as the plugin concept is rather new) – you will probably have to wade through some examples at their Github repo

• Deployment into the Docker container is not straightforward (but you do not have to use Docker for that)

• Reconfiguration of an output plugin is not possible (so you have to delete and recreate it your adapter all the time)

• Dynamic reloading is not supported, so you have to restart the Graylog2 node every time you redeploy

• I actually thought it to be harder to develop a plugin, but to my surprise it was relatively easy

• For ease of testing or deployment you can also use the automatic build at drone.io to deploy the plugin directly in your docker file (see this post for further information).

• The Plugin Architecture is currently changing (even introducing breaking changes) so it is partially a little bit frustrating and involves additional troubleshooting. But as of v1.0.0 this seems to have a little bit stabilised.

• There is also an AlarmCallback plugin that you can use to execute scripts. See biz.dfch.j.graylog2.plugin.alarm.execscript.