Recently I was testing Graylog2 as a store for metering data and had to bulk-load a larger amount of data into its database. My goal was to create synthetic and randomised metric information for a couple of virtual servers (5’000) over a period of one year at a sampling rate of 5s. This would make a total number of 31'536'000'000 (= 365 * 24 * 60 * (60/5) * 5000) (or 31 billion) entries. So manual insertion of these records was top priority to avoid. Instead of using curl as described on the Gelf Format Specification I thought I would give PowerShell a try…

After setting up the respective Gelf HTTP Input and starting to insert messages via Invoke-RestMethod (see Insert-GelfHttp.ps1) I noticed that this was way too slow. There had to be something faster than performing an HTTP POST for every metric. So I switched to Gelf via UDP (using System.Net.Sockets.Socket, see Insert-GelfUdp.ps1) – however it soon became clear that running a graylog2 docker instance within a virtualised CentOS 7 was overloading the VMware NAT stack on my Windows notebook (losing loads of packets). So I finally settled with the Gelf TCP Input (see Insert-GelfTcp.ps1) and was eventually able to insert my test data (though it still took quite a while, i.e. days) …

Bulk loading data into Graylog2

Displaying metering data in Graylog2

You can find the complete source at our Gist or at the end of this Post.

Note

  • While inserting data for past dates I first fooled myself as I was not able to see anything in the Graylog2 search dialogue (because it only show the last couple of minutes by default). I had to switch to “show all messages” to actually see all my inserted data

  • When searching for messages before or after a specific timestamp you have to supply the timestamp in Epoch milliseconds. This is counter-intuitive as the timestamp in the Gelf messasges is in Epoch seconds. See the discussion at GELF spec 1.2.

  • When using the current graylog2 docker instance (id 387b6e81d77d) the container will not be able to start after you once stopped it. See Docker container cannot restart services after being stopped for a possible workaround.

  • When sending messages via Gelf TCP or UDP you might have to adjust/increase the receive buffer size or insert client side throttling or you might lose some of them.

  • Keep in mind you have to append a leading CRLF or 00 byte when inserting messages via TCP or UDP (depending on your input configuration.

  • When opening and closing sockets to the server your client may run quickly out of ephemeral (client) ports unless you specify to re-use sockets.

  • Not closing connections to the server may exhaust the server as well.

  • I had to increase ProcessBuffers and heap size (of master cache) significantly as my loader script was much faster than graylog2/elasticsearch (with having millions of messages in the backlog).

Scripts

If you cannot see the script code below you can view it directly at our Gist.

view raw

### Sending Gelf messages to Graylog2 via PowerShell.md

hosted with ❤ by GitHub


[CmdletBinding(
SupportsShouldProcess = $true
,
ConfirmImpact = 'High'
,
HelpURI = 'https://d-fens.ch/2014/12/31/nobrainer-sending-gelf-messages-to-graylog2-via-powershell'
,
DefaultParameterSetName = 'EndDate'
)]
PARAM
(
[Uri] $Uri = 'http://192.168.174.161:12202/gelf'
,
[DateTime] $StartDate = [DateTime]::Now.AddDays(-7).ToString('yyyy-MM-dd')
,
[Parameter(Mandatory = $false, ParameterSetName = 'EndDate')]
[DateTime] $EndDate = [DateTime]::Now.ToString('yyyy-MM-dd')
,
[Parameter(Mandatory = $false, ParameterSetName = 'Days')]
[int] $Days = 7
,
[Parameter(Mandatory = $false)]
[Object] $Object
,
[int] $IncrementInSeconds = 5
,
[int] $TotalEntities = 100
,
[string] $DnsSuffix = 'dfch.biz'
)
try
{
if('Days' -eq $PSCmdlet.ParameterSetName)
{
$EndDate = $StartDate.AddDays($Days);
}
$ShouldProcessMessage = "{0} entities: '{1}' – '{2}' @{3}s [{4} total messages]" -f $TotalEntities, $StartDate.ToString('u'), $EndDate.ToString('u'), $IncrementInSeconds, (($EndDate – $StartDate).TotalSeconds / $IncrementInSeconds * $TotalEntities);
if(!$PSCmdlet.ShouldProcess($ShouldProcessMessage))
{
Exit;
}
# default parameters to save typing and space
$PSDefaultParameterValues.'Invoke-RestMethod:Uri' = $Uri.AbsoluteUri;
$PSDefaultParameterValues.'Invoke-RestMethod:Method' = 'POST';
$PSDefaultParameterValues.'Invoke-RestMethod:ContentType' = 'application/json';
$PSDefaultParameterValues.'Invoke-RestMethod:Verbose' = $false;
$PSDefaultParameterValues.'ConvertTo-Json:Compress' = $true;
# generate some server ids
$al = New-Object System.Collections.ArrayList($TotalEntities);
$null = $al.Add('deaddead-dead-dead-dead-deaddeaddead');
for($c = 1; $c -lt $TotalEntities; $c++)
{
$null = $al.Add([Guid]::NewGuid().Guid);
}
$CurrentDate = $StartDate;
while($EndDate -gt $CurrentDate)
{
$ProgressPreference = 'Continue';
Write-Progress –id 1 –Activity ("CurrentDate '{0}' / EndDate '{1}'" -f $CurrentDate, $EndDate) –CurrentOperation $Uri.AbsoluteUri;
$ProgressPreference = 'silentlyContinue';
$Epoch = [Math]::Floor( [decimal] (Get-Date (([dateTime] $CurrentDate).ToUniversalTime()) –UFormat '%s'));
for($c = 0; $c -lt $TotalEntities; $c++)
{
$Guid = $al[$c];
if($Object)
{
$JsonBody = '{0}{1}' -f ($Object | ConvertTo-Json), "`0";
}
else
{
$JsonBody = @{ host = ('s{0:D7}.{1}' -f $c, $DnsSuffix); timestamp = $Epoch; short_message = 'metrics'; _cpu = (Get-Random –Minimum 0 –Maximum 100); _mem = ((Get-Random –Minimum 1 –Maximum 16) * 1024); _disk = ((Get-Random –Minimum 20 –Maximum 512) * 1024); _guid = $Guid; } | ConvertTo-Json;
}
$null = Invoke-RestMethod –Body $JsonBody;
}
$CurrentDate = $CurrentDate.AddSeconds($IncrementInSeconds);
}
}
catch
{
throw $_;
}
<#
Copyright 2014-2015 d-fens GmbH
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
#>

view raw

Insert-GelfHttp.ps1

hosted with ❤ by GitHub


[CmdletBinding(
SupportsShouldProcess = $true
,
ConfirmImpact = 'High'
,
HelpURI = 'https://d-fens.ch/2014/12/31/nobrainer-sending-gelf-messages-to-graylog2-via-powershell'
,
DefaultParameterSetName = 'EndDate'
)]
PARAM
(
[Uri] $Uri = 'tcp://192.168.174.161:12201'
,
[DateTime] $StartDate = [DateTime]::Now.AddDays(-7).ToString('yyyy-MM-dd')
,
[Parameter(Mandatory = $false, ParameterSetName = 'EndDate')]
[DateTime] $EndDate = [DateTime]::Now.ToString('yyyy-MM-dd')
,
[Parameter(Mandatory = $false, ParameterSetName = 'Days')]
[int] $Days = 7
,
[Parameter(Mandatory = $false)]
[Object] $Object
,
[int] $IncrementInSeconds = 5
,
[int] $TotalEntities = 5000
,
[string] $DnsSuffix = 'dfch.biz'
)
try
{
if('Days' -eq $PSCmdlet.ParameterSetName)
{
$EndDate = $StartDate.AddDays($Days);
}
$ShouldProcessMessage = "{0} entities: '{1}' – '{2}' @{3}s [{4} total messages]" -f $TotalEntities, $StartDate.ToString('u'), $EndDate.ToString('u'), $IncrementInSeconds, (($EndDate – $StartDate).TotalSeconds / $IncrementInSeconds * $TotalEntities);
if(!$PSCmdlet.ShouldProcess($ShouldProcessMessage))
{
Exit;
}
$PSDefaultParameterValues.'ConvertTo-Json:Compress' = $true;
$Address = [System.Net.IPAddress]::Parse($Uri.Host);
# Create Endpoint and Socket
$EndPoint = New-Object System.Net.IPEndPoint($Address, $Uri.Port);
# Connect to Socket
$Socket = New-Object System.Net.Sockets.Socket($Address.AddressFamily, [System.Net.Sockets.SocketType]::Stream, $Uri.Scheme);
$Socket.Connect($EndPoint);
# generate some server ids
$al = New-Object System.Collections.ArrayList($TotalEntities);
$null = $al.Add('deaddead-dead-dead-dead-deaddeaddead');
for($c = 1; $c -lt $TotalEntities; $c++)
{
$null = $al.Add([Guid]::NewGuid().Guid);
}
$CurrentDate = $StartDate;
while($EndDate -gt $CurrentDate)
{
$ProgressPreference = 'Continue';
Write-Progress –id 1 –Activity ("CurrentDate '{0}' / EndDate '{1}'" -f $CurrentDate, $EndDate) –CurrentOperation $Uri.AbsoluteUri;
$ProgressPreference = 'silentlyContinue';
$Epoch = [Math]::Floor( [decimal] (Get-Date (([dateTime] $CurrentDate).ToUniversalTime()) –UFormat '%s'));
for($c = 0; $c -lt $TotalEntities; $c++)
{
$Guid = $al[$c];
if($Object)
{
$JsonBody = '{0}{1}' -f ($Object | ConvertTo-Json), "`0";
}
else
{
$JsonBody = '{0}{1}' -f (@{ version = '1.2'; host = ('s{0:D7}.{1}' -f $c, $DnsSuffix); timestamp = ('{0}.000' -f $Epoch); short_message = 'metrics'; _cpu = (Get-Random –Minimum 0 –Maximum 100); _mem = ((Get-Random –Minimum 1 –Maximum 16) * 1024); _disk = ((Get-Random –Minimum 20 –Maximum 512) * 1024); _guid = $Guid; } | ConvertTo-Json), "`0";
}
$Sent = $Socket.Send([System.Text.Encoding]::UTF8.GetBytes($JsonBody));
}
$CurrentDate = $CurrentDate.AddSeconds($IncrementInSeconds);
}
}
catch
{
throw $_;
}
finally
{
if($Socket)
{
$Socket.Disconnect($true);
$Socket.Close();
$Socket.Dispose();
}
}
<#
Copyright 2014-2015 d-fens GmbH
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
#>

view raw

Insert-GelfTcp.ps1

hosted with ❤ by GitHub


[CmdletBinding(
SupportsShouldProcess = $true
,
ConfirmImpact = 'High'
,
HelpURI = 'https://d-fens.ch/2014/12/31/nobrainer-sending-gelf-messages-to-graylog2-via-powershell'
,
DefaultParameterSetName = 'EndDate'
)]
PARAM
(
[Uri] $Uri = 'udp://192.168.174.161:49165'
,
[DateTime] $StartDate = [DateTime]::Now.AddDays(-7).ToString('yyyy-MM-dd')
,
[Parameter(Mandatory = $false, ParameterSetName = 'EndDate')]
[DateTime] $EndDate = [DateTime]::Now.ToString('yyyy-MM-dd')
,
[Parameter(Mandatory = $false, ParameterSetName = 'Days')]
[int] $Days = 7
,
[Parameter(Mandatory = $false)]
[Object] $Object
,
[int] $IncrementInSeconds = 5
,
[int] $TotalEntities = 100
,
[string] $DnsSuffix = 'dfch.biz'
)
try
{
if('Days' -eq $PSCmdlet.ParameterSetName)
{
$EndDate = $StartDate.AddDays($Days);
}
$ShouldProcessMessage = "{0} entities: '{1}' – '{2}' @{3}s [{4} total messages]" -f $TotalEntities, $StartDate.ToString('u'), $EndDate.ToString('u'), $IncrementInSeconds, (($EndDate – $StartDate).TotalSeconds / $IncrementInSeconds * $TotalEntities);
if(!$PSCmdlet.ShouldProcess($ShouldProcessMessage))
{
Exit;
}
$PSDefaultParameterValues.'ConvertTo-Json:Compress' = $true;
$Address = [System.Net.IPAddress]::Parse($Uri.Host);
# Create Endpoint and Socket
$EndPoint = New-Object System.Net.IPEndPoint($Address, $Uri.Port);
$Socket = New-Object System.Net.Sockets.Socket($Address.AddressFamily, [System.Net.Sockets.SocketType]::Dgram, $Uri.Scheme);
# Connect to Socket
$Socket.Connect($EndPoint);
# generate some server ids
$al = New-Object System.Collections.ArrayList($TotalEntities);
$null = $al.Add('deaddead-dead-dead-dead-deaddeaddead');
for($c = 1; $c -lt $TotalEntities; $c++)
{
$null = $al.Add([Guid]::NewGuid().Guid);
}
$CurrentDate = $StartDate;
while($EndDate -gt $CurrentDate)
{
$ProgressPreference = 'Continue';
Write-Progress –id 1 –Activity ("CurrentDate '{0}' / EndDate '{1}'" -f $CurrentDate, $EndDate) –CurrentOperation $Uri.AbsoluteUri;
$ProgressPreference = 'silentlyContinue';
$Epoch = [Math]::Floor( [decimal] (Get-Date (([dateTime] $CurrentDate).ToUniversalTime()) –UFormat '%s'));
for($c = 0; $c -lt $TotalEntities; $c++)
{
$Guid = $al[$c];
if($Object)
{
$JsonBody = '{0}{1}' -f ($Object | ConvertTo-Json), "`0";
}
else
{
$JsonBody = '{0}{1}' -f (@{ version = '1.2'; host = ('s{0:D7}.{1}' -f $c, $DnsSuffix); timestamp = ('{0}.000' -f $Epoch); short_message = 'metrics'; _cpu = (Get-Random –Minimum 0 –Maximum 100); _mem = ((Get-Random –Minimum 1 –Maximum 16) * 1024); _disk = ((Get-Random –Minimum 20 –Maximum 512) * 1024); _guid = $Guid; } | ConvertTo-Json), "`0";
}
$Sent = $Socket.Send([System.Text.Encoding]::UTF8.GetBytes($JsonBody));
}
$CurrentDate = $CurrentDate.AddSeconds($IncrementInSeconds);
}
}
catch
{
throw $_;
}
finally
{
if($Socket)
{
$Socket.Close();
}
}
<#
Copyright 2014-2015 d-fens GmbH
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
#>

view raw

Insert-GelfUdp.ps1

hosted with ❤ by GitHub


d-fens Sending Gelf messages to Graylog2 via PowerShell
Copyright 2014 d-fens GmbH
This product includes software developed at
d-fens GmbH (https://d-fens.ch/)
d-fens GmbH
General-Guisan-Strasse 6
CH-6300 Zug
Switzerland

view raw

NOTICE

hosted with ❤ by GitHub

Posted on 3 By Ronald Rink PowerShell Posted in PowerShell, Various Products Tagged #, , ,

Ronald Rink

I am a senior auditor and consultant at d-fens for business processes and information systems.

3 Comments »

  1. Pingback: [HOWTO] Creating a Graylog2 Output Plugin – d-fens GmbH
  2. Pingback: [NoBrainer] Using drone.io to automatically integrate your Graylog2 Plugins in Docker – d-fens GmbH
  3. Pingback: [HOWTO] Creating a Graylog2 Filter Plugin – d-fens GmbH

Leave a Reply

Fill in your details below or click an icon to log in:

Gravatar
WordPress.com Logo

You are commenting using your WordPress.com account. ( Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. ( Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. ( Log Out /  Change )

Cancel

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.