Introduction

UPDATE 2015-02-26 The plugin now works with the Graylog v1 plugin API

Following the article about Creating a Graylog2 Output Plugin this post will cover the steps needed to create a filter plugin for Graylog2. While the SCRIPT OUTPUT Plugin enabled a potential user to run scripts for post-processing of messages, the SCRIPT FILTER Plugin will be very creative and offer the same functionality: executing scripts based on incoming messages. This will certainly not make that much sense in high volume / load environments, but is sufficient for illustrative purposes.

Here is a brief overview of what is covered:

• Filter Processing

• The Tools

• Development

• Deployment

• Testing

• Download and Notes

Filter Processing

A quick recap: Writing plugins describes the different plugin types that are available in Graylog2, and the diagram about Stream Processing (in Streams) shows the built-in filter chain, that consists of the ExtractorFilter, StaticFieldFilter and the StreamMatcherFilter. And this is where our filter plugin will integrate.

When taking a closer look at the Graylog2 source (in lieu of thorough documentation), each filter has a priority, which consists of an integer with lower priorities being executed first. So depending on the priority the the message to be processed will either be not yet assigned to a stream or already be assigned to zero, one or more streams. Keeping this in mind, it is possible to create filters that will only execute for specific streams – and chaining them).

A drawback with the current implementation of the filter architecture is that filters are loaded during ProcessBuffer initialisation, meaning no user interface is available for configuring them. They cannot be paused, stopped or changed either.

A last thing, within a filter plugin the complete Message is available, so it is possible to read and alter existing fields as well as adding new fields. This processing of messages is done via the filter method, and beside that method there is not much else about filters.

The Tools

Here is a quick summary of what I used as the development environment (feel free to skip if you are not interested):

• A CentOS 7.0 virtual machine with fairly little resources (2 CPU, 4 GB RAM)

• Oracle JDK 7 (not tested with JDK 8 or OpenJDK)

• IntelliJ IDEA for my development IDE as recommended by Lennart Koopmann

• The Graylog2 Plugin Archetype
  Make sure you select the right version for your environment (also see Notes at the end for further information on this)!

• A Graylog2 instance based on Docker from the official Graylog2 Docker image definition (and a quick docker pull graylog2/allinone as described in my previous post)

• A PowerShell load script to insert messages into Graylog2

• drone.io for an automatic build of the plugin
  Note: I am currently testing drone.io as a CI environment. To be on the safe side, build the source on your own trusted platform.

Development

Creating the project

Create the skeleton interactively via mvn archetype:generate:

mvn archetype:generate -DarchetypeGroupId=org.graylog2 -DarchetypeArtifactId=graylog2-plugin-archetype

groupId: biz.dfch.j.graylog2.plugin.filter
artifactId: execscript
version: 1.0.0-SNAPSHOT
package: biz.dfch.j.graylog2.plugin.filter
pluginClassName: dfchBizExecScript

This will create you all the source file you need, the necessary script and configuration files will be created later.

Implementing the interface

• dfchBizExecScriptModule.java
  The module class will specify what kind of plugin we implement.

• dfchBizExecScriptPlugin.java
  In this file we will only specify the class name that will implement the actual plugin.

• dfchBizExecScriptMetadata.java
  Here we define several parameters like plugin name, description and required minimum version. This is the information shown when looking the at the ‘Installed plugins’ in the ‘Details’ view in the System/Node menu.
  One thing to point out is the ‘uuid’ that every plugin has to specify. But it is actually not clear if this id has to be unique acrose session lifetime of the plugin or if this is used somewhere else as well.

• dfchBizExecScript.java
  This class contains the actual implementation. While most of the methods to overload are just for setup and initialisation, we will implement a little bit more than required to support dynamic configuration parameters.

import org.graylog2.plugin.Message;
/**
*
* @author Lennart Koopmann <lennart@socketfeed.com>
*/
public interface MessageFilter 
{
  /**
  * Process a Message
  *
  * @return true if this message should not further be handled (for example for blacklisting purposes)
  */
  public boolean filter(Message msg);
  /**
  * @return The name of this filter. Should not include whitespaces or special characters.
  */
  public String getName();
  /**
  * For determining the runtime order of the filter, specify a priority.
  * Lower priorty values are run earlier, if two filters have the same priority, their name will be compared to
  * guarantee a repeatable order.
  *
  * @return the priority
  */
  int getPriority();
}

See Message.java for complete source.

• Constructor dfchBizExecScript::dfchBizExecScript()
  As written before a filter does not have any user interface or configuration means, therefore we will implement this via a json-encoded config file (for a plugin execscript.jar the configuration will be loaded from execscript.conf in the same directory).

{
DF_SCRIPT_ENGINE" : "javascript" // specifies the script engine to use
,
//"DF_SCRIPT_PATH_AND_NAME" : "/opt/dfchBizMessageFilterExecScript.js" // either supply a full path ...
//,
"DF_SCRIPT_NAME" : "dfchBizMessageFilterExecScript.js" // ... or just the script name
,
"DF_SCRIPT_CACHE_CONTENTS" : false // set to true to reload the script on every use
,
"DF_SCRIPT_DISPLAY_OUTPUT" : true // set to true to display the script output to STDOUT
,
"DF_PLUGIN_PRIORITY" : "42" // the priority of the filter
,
"DF_PLUGIN_DROP_MESSAGE" : false // set to true to drop every message
,
"DF_PLUGIN_DISABLED" : false // set to true to disable the plugin
}

• Initialiser dfchBizExecScript::initialise
  Most of the other plugin types have some kind of configuration interface and an initialise method that will (not surprisingly) initialise the plugin. In this method will except a Configuration object. We will mimick this behaviour so that we can reuse some code and logic when writing other plugins.
  Our example plugin will execute a script so we will set up the ScriptEngine and other necessary objects here.

• Filter dfchBizExecScript::filter
  As discussed earlier we implement the acutal logic inside this method. Returning true means to filter out or drop the message entirely, returning false means the message will be further processed.

• Priority dfchBizExecScript::getPriority
  The priority is taken from the configuration file.

• Multithreading
  Depending on the number of ProcessBuffers (see graylog2.conf) the plugin will be initialised several times. For each initialisation the config parameters will be logged to STDOUT (so seeing the initialisation message more than once is not an error).

Deployment

As it is rather difficult to copy data from a host into a running Docker container, I used NetCat to pipe the resulting JAR into the container (followed by a restart of the Graylog2 service):

edgar@CONTAINER:
nc -l 10101 > /opt/graylog2/plugin/execScript-1.0.0-SNAPSHOT.jar

edgar@HOST:
cat target/execScript-1.0.0-SNAPSHOT.jar | nc 172.17.0.7 10101

Note: make sure you also deploy the configuration and script files to the target node.

Another option for deployment is to integrate your plugin in a custom Dockerfile based on the official graylog2/allinone Dockerfile that is also used for the Docker Hub Registry. In this Dockerfile you would integrate your plugin binary via an automated build create a Docker image from that on every commit (as described in Using drone.io to automatically integrate your Graylog2 Plugins in Docker).

Testing

You can send messages as described in SCRIPT OUTPUT Plugin Testing. Here is an example script that reads and adds message parameters:

print("Now this is a message: " + message + "\r\n");
//print(message.id + "\r\n");
//print(message.source + "\r\n");
//print(message.message + "\r\n");
// get specific additional field in message
//print(message.getField("guid") + "\r\n");
// add field
//message.addField("FullName", "Edgar Schnittenfittich);

Download

• You can download the SCRIPT FILTER Plugin source at our GitHub account

• You can also download the binary at our drone.io account. This gets built every time we commit something to the master branch of the repository.

• The plugin is LICENSED under Apache 2.0 so you can freely use and modify it.

Notes

• Real documentation except for the single page about plugins is really hard to find (as the plugin concept is rather new) – you will probably have to wade through some examples at their Github repo

• You could implement a FileSystemWatcher to dynamically reload the configuration when the config file changes. (In addition you could implement a REST resource to change the configuration.)

• The Plugin Architecture is currently changing (even introducing breaking changes) so check if something has changed.

• When deploying the name of the jar is not relevant, what matters is the reference in the properties file inside the car. This means you can deploy the same plugin several times (only makes sense for filter plugins) and define different priorities and configuration values for it