We are using PowerShell to automate various backend systems quite heavily. Sometimes these interfaces (which may be REST APIs, message queues or something completely different) are shortly not available because of intermittent failures or simply because of network issues. In any case, most of the time a script will crash, though quite a bunch of work in this script might already have been successfully executed.

Therefore it is favourable to re-try these failed operations without having to abandon (and potentionally roll-back) the whole script. In this blog post I present you a quick and easy way to enable your scripts to support such a retry mechanism. The Cmdlet presented here is based on an idea of the Microsoft Enterprise Library, specifically the Transient Fault Handling Application Block.

The concept is easy: our Cmdlet Invoke-WithRetry will take a [ScriptBlock] and execute it with Invoke-Command. If the scriptblock throws an exception, it will be executed again – up to the number defined in MaxAttempts (default is 5). Between each retry operation the Cmdlet waits a number of milliseconds based on the RetryStrategy.

There are three RetryStrategy types defined:

• Fixed – the wait time stays always the same

200, 200, 200, 200, 200

• Incremental – the wait time incrementally and linearly adds with each iteration

200, 400, 600, 800, 1000

• Exponential – the wait time doubles with each iteration (default strategy)

200, 400, 800, 1600, 3200

The base wait interval can also be specified via InitialWaitTimeInMilliseconds while having a default value of 200ms.

Here are some examples how to use the Cmdlet:

• In this example we define a scriptblock that returns the current time to the caller.
• This scriptblock is then executed via the ‘Fixed’ RetryStrategy. The returned
  value is then saved to the ‘$result’ variable and written to the console.

PS > [scriptblock] $scriptblock = {
return [System.DateTimeOffset]::Now.ToString('yyyy-MM-dd HH:mm:ss.fff');
}

PS > $result = $scriptblock | Invoke-WithRetry -RetryStrategy Fixed;
PS > $result
2016-01-01 22:48:48.849

• In this example we define a scriptblock that outputs a string and the current time. To simulate an error the scriptblock also throws an exception.
• This scriptblock is then executed via the ‘Fixed’ RetryStrategy as you can see from the time stamps.

PS > [scriptblock] $scriptblock = {
Write-Host ("arbitrary string {0}" -f [System.DateTimeOffset]::Now.ToString('yyyy-MM-dd HH:mm:ss.fff'));
throw
}

PS > $scriptblock | Invoke-WithRetry -RetryStrategy Fixed;
arbitrary string 2016-01-01 22:43:04.147
arbitrary string 2016-01-01 22:43:04.375
arbitrary string 2016-01-01 22:43:04.596
arbitrary string 2016-01-01 22:43:04.804
arbitrary string 2016-01-01 22:43:05.017

• In this example we define a scriptblock that outputs a string and the current time. To simulate an error the scriptblock also throws an exception.
• This scriptblock is then executed via the ‘Incremental’ RetryStrategy as you can see from the time stamps.

PS > [scriptblock] $scriptblock = {
Write-Host ("arbitrary string {0}" -f [System.DateTimeOffset]::Now.ToString('yyyy-MM-dd HH:mm:ss.fff'));
throw
}

PS > $scriptblock | Invoke-WithRetry -RetryStrategy Incremental -Step 2000 -MaxAttempts 3;
arbitrary string 2016-01-01 22:46:16.979
arbitrary string 2016-01-01 22:46:18.995
arbitrary string 2016-01-01 22:46:23.003

• In this example we define a scriptblock that accepts an input parameter and throws an assertion if the input is 1. The scriptblock itself will output an arbitrary string and the specified input.
• In the example you see how the wait time between each interval doubles.

PS > [scriptblock] $scriptblock = {
PARAM
(
$count
)
Write-Host ("arbitrary string {0} {1}" -f $count, [System.DateTimeOffset]::Now.ToString('yyyy-MM-dd HH:mm:ss.fff'));
Contract-Assert ($count -ne 1)
}

PS > $scriptblock | Invoke-WithRetry -RetryStrategy Exponential -ArgumentList 1 -MaxAttempts 6;
arbitrary string 1 2016-01-01 22:42:39.394
WARNING: : Assertion failed: ($count -ne 1)
arbitrary string 1 2016-01-01 22:42:39.615
WARNING: : Assertion failed: ($count -ne 1)
arbitrary string 1 2016-01-01 22:42:40.032
WARNING: : Assertion failed: ($count -ne 1)
arbitrary string 1 2016-01-01 22:42:40.860
WARNING: : Assertion failed: ($count -ne 1)
arbitrary string 1 2016-01-01 22:42:42.482
WARNING: : Assertion failed: ($count -ne 1)
arbitrary string 1 2016-01-01 22:42:45.709
WARNING: : Assertion failed: ($count -ne 1)

Below you find the source code in an abbreviated form. For the full version you should check Invoke-WithRetry on GitHub or download the package from NuGet.

Please note that this Cmdlet uses Cmdlets for logging and error handling that are part of biz.dfch.PS.System.Logging and biz.dfch.PS.System.Utilities.

Function Invoke-WithRetry {
PARAM
(
[ValidateNotNull()]
[Parameter(Mandatory = $true, ValueFromPipeline = $true, Position = 0)]
[Alias('InputObject')]
[scriptblock] $ScriptBlock
,
[Parameter(Mandatory = $false, Position = 1)]
[Object[]] $ArgumentList
,
[ValidateRange(1,[int]::MaxValue)]
[Parameter(Mandatory = $false)]
[int] $MaxAttempts = 5
,
[Parameter(Mandatory = $false)]
[Alias('Step')]
[int] $InitialWaitTimeInMilliseconds = 200
,
[ValidateSet('Exponential', 'Fixed', 'Incremental')]
[Parameter(Mandatory = $false)]
[string] $RetryStrategy = 'Exponential'
)

Begin
{
$currentAttempt = 0;
$currentWaitTimeMs = $InitialWaitTimeInMilliseconds;
$fCompleted = $false;
}

Process
{
do
{
$currentAttempt++;

if($PSCmdlet.ShouldProcess(('{0}: {1}/{2} @{3}ms' -f $RetryStrategy, $currentAttempt, $MaxAttempts, $currentWaitTimeMs)))
{
try
{
if($ArgumentList)
{
$result = Invoke-Command -ScriptBlock $ScriptBlock -ArgumentList $ArgumentList;
}
else
{
$result = Invoke-Command -ScriptBlock $ScriptBlock;
}
$fReturn = $true;
break;
}
catch
{
if($currentAttempt -lt $MaxAttempts)
{
Start-Sleep -Milliseconds $currentWaitTimeMs;
}
}
}

switch($RetryStrategy)
{
"Exponential"
{
$currentWaitTimeMs *= 2;
}
"Incremental"
{
$currentWaitTimeMs += $InitialWaitTimeInMilliseconds;
}
}
}
while($currentAttempt -lt $MaxAttempts);

return $result;
}

}