[HOWTO] Creating a Graylog2 Output Plugin

Introduction

As described in my last post Sending Gelf messages to Graylog2 via PowerShell we were testing Graylog2 for log and meter collection. It soon became clear that the current selection of adapters to get data out of Graylog2 was far from perfect. We needed a specific output action to send data to an incident management system that could only be automated by a command line tool. So I decided to create the SCRIPT OUTPUT Plugin for Graylog2 (which you can download from our GitHub account) and which I will quickly present in this post.

As it is written on Writing plugins, Graylog2 offers an extensible plugin model which you can use to create additional output adapters (and other types) tailored to your needs. Documentation turned out to be a little bit sparse, but luckily there is the Pagerduty AlarmCallback Plugin for reference on their GitHub account. With Maven and a Graylog2 Plugin archetype to the rescue it seemed fairly easy to get on the road to my first Graylog2 output plugin …

Here is a brief overview of what is covered:

The Tools

Here is a quick summary of what I used as the development environment (feel free to skip if you are not interested):

Development

  • Creating the project
    I created the skeleton interactively via mvn archetype:generate (and used a not so clever naming for my plugin):

    mvn archetype:generate -DarchetypeGroupId=org.graylog2 -DarchetypeArtifactId=graylog2-plugin-archetype
    
    groupId: biz.dfch.j.graylog2.plugin.output
    artifactId: execscript
    version: 1.0.0-SNAPSHOT
    package: biz.dfch.j.graylog2.plugin.output
    pluginClassName: dfchBizExecScript
    

UPDATE 2015-01-22 Beware that you use the correct version of the Archetype (depending on the Graylog2 you are using as there is currently some ‘motion’ (see also the discussion at Crash on launch input in 0.91.3 (and higher)). For the output plugin this is not really an issue, but if something goes wrong or the plugin does not load, make sure you check this as well.

After that I opened the project from IntelliJ IDEA and set a JDK path (using OpenJDK 1.7)

  • Implementing the interface
    As the PagerDuty plugin was not an Output plugin I had to look around to see which messages had to be overriden to make it work. Essentially I had to override these methods from MessageOutput:

    public interface MessageOutput {
        void initialize(Configuration var1) throws MessageOutputConfigurationException;
    
        void stop();
    
        boolean isRunning();
    
        void write(Message var1) throws Exception;
    
        void write(List<Message> var1) throws Exception;
    
        ConfigurationRequest getRequestedConfiguration();
    
        String getName();
    
        String getHumanName();
    
        String getLinkToDocs();
    }
    

The most important methods is write where you actually place your functionality what your plugin should do.

It only took me three attempts to get the methods right and upon second try -to my surprise- it already showed up as an output module in Graylog2.

  • Filling in the blanks
    For ease of implementation I used the ScriptEngine class to execute my javascript files (that in turn would trigger the aforementioned command line tool). I also inserted the processed message into the session context, so the script could happily grab the pieces it actually needed for forwarding:

    root@50167a4a0370:/opt/graylog2# cat /opt/graylog2/plugin/helloworld.js
    // message has custom fields exposed as properties
    // standard GELF fields are properties directly on the message object
    print(message + "rn");
    print(message.message + "rn");
    print(message.source + "rn");
    print(message.getField("guid") + "rn");
    

UPDATE 2015-03-31: I now bundled the Jython artifact into the plugin, so you can also call and execute Python scripts.

Deployment

As it is rather difficult to copy data from a host into a running Docker container, I used NetCat to pipe the resulting JAR into the container (followed by a restart of the Graylog2 service):

edgar@CONTAINER:
nc -l 10101 > /opt/graylog2/plugin/DfchBizTestOutput-1.0.0-SNAPSHOT.jar

edgar@HOST:
cat target/DfchBizTestOutput-1.0.0-SNAPSHOT.jar | nc 172.17.0.7 10101

Note: see the Notes at the end of the post on how to automatically integrate the plugin into Graylog2.

Testing

  • Message Input
    I created a GELF TCP input and inserted data via Insert-GelfTcp.ps1

  • Stream Definition
    I defined a stream that matched on every message with a guid field

Manage Outputs

  • Message Output
    I instantiated my shiny new Output plugin, associated it with the stream and had it execute a javascript file in the container with its output being mirrored on the console

Defining new Output

Defining new Output

Outputs per Stream [UI]

  • So for every message sent into the system the plugin would execute the javascript file – a guarantee for a (s)low performing system …

Script Output

API View of Configuration

Everything I configured via the web interface could also be checked and retrieved via the REST API:

  • Output Definition
    Here you can see if your plugin is visible and available for use:
    Output Definition

  • Stream Definition
    The defined stream shows the associated output:
    Stream Definition

Download

Notes

  • Real documentation except for the single page about plugins is really hard to find (as the plugin concept is rather new) – you will probably have to wade through some examples at their Github repo

  • Deployment into the Docker container is not straightforward (but you do not have to use Docker for that)

  • Reconfiguration of an output plugin is not possible (so you have to delete and recreate it your adapter all the time)

  • Dynamic reloading is not supported, so you have to restart the Graylog2 node every time you redeploy

  • I actually thought it to be harder to develop a plugin, but to my surprise it was relatively easy

  • For ease of testing or deployment you can also use the automatic build at drone.io to deploy the plugin directly in your docker file (see this post for further information).

  • The Plugin Architecture is currently changing (even introducing breaking changes) so it is partially a little bit frustrating and involves additional troubleshooting. But as of v1.0.0 this seems to have a little bit stabilised.

  • There is also an AlarmCallback plugin that you can use to execute scripts. See biz.dfch.j.graylog2.plugin.alarm.execscript.

Comments

  1. Ronald Rink says:

    Based on a discussion at [Crash on launch input in 0.91.3 (and higher)](https://github.com/Graylog2/graylog2-input-mqtt/issues/1) I updated the post as the plugin architecture is currently somewhat changing (introducing breaking changes).

Trackbacks

  1. […] As we are currently evaluating Graylog2 as a central log collector and processor we soon realised that we would need quite some additional processing options (beyond what is built-in the product) that we were luckily able to add via Graylog2’s built-in plugin mechanism. As we are using Docker images for our tests it soon turned out to be quite cumbersome to integrate the plugins into the standard image (in the long run cat 'plugin.jar' | nc 'graylog2-server' appeared not to be the greatest deployment option, as described in Creating a Graylog2 Output Plugin). […]

  2. […] the article about Creating a Graylog2 Output Plugin this post will cover the steps needed to create a filter plugin for Graylog2. While the SCRIPT […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: