Transient Error Handling with automatic Retries in C#

Inspired by Transient Fault Handling we created a quick helper class to faciliate the automatic retry of various operation that may fail in today’s interconnected environments.

Large parts of our Appclusive framework deal with fetching from and pushing information to various backend systems. As they all can (and eventuall will) fail we needed a flexible approach on how to retry these failed operations. When we first addresses this issue we used the The Transient Fault Handling Application Block from Microsoft patterns & practices. It however turned out for us that this was either too much overhead or not flexible enough to deal with various errors and exceptions we experienced in our environments. There we decided not to “reinvent the wheel” but add some functionality to it and decouple if from the application block (as it did not seem to get updated in quite some time).

What we needed

  1. We needed an easy method for executing arbitrary piece of code with a given retry strategy. The default strategy would use some kind of exponential back-off timer but with an upper limit. In addition we would favour a maximum retry time over a certain number of retry attempts.

  2. We wanted to support different retry strategies and be open for custom implementations. The basic retry strategies sould be:
    a. Exponential
    b. Fixed
    c. Incremental
    d. Custom

  3. Default return values should be optional and be used in cases where none of the retries would have been successful.

  4. Exception filters should allow for custom exception handling so it can be decided depending on the exception if retries should be performed or not.

What we got

We defined a base class for all our retry strategies which looks like this (you can find the complete code here or at the end of this blog post):

public abstract class RetryStrategyBase
{
  public int MaxAttempts = int.MaxValue;

  public int CurrentAttempt = 1;

  public int MaxWaitTimeMilliseconds = 300 * 1000;

  public int MinWaitTimeIntervalMilliseconds = 200;

  public int MaxWaitTimeIntervalMilliseconds = 20 * 1000;

  public object StateInfo;

  public int NextAttempt()
  {
    return ++CurrentAttempt;
  }

  public abstract int GetNextWaitTime(int currentWaitTime);
}

Our Retry would take whatever strategy we specified (or take the Exponential defaulT) and could be invoke as easy as this:

var result = Retry.Invoke
(
  func: () => 1 / 0, 
  exceptionFilter: null, 
  defaultReturnValueFactory: () => 42
);
Assert.AreEqual(42, result);

Here we would try to divide by zero, and return a default value of 42 in case we would not succeed. The chosen retry strategy defaults to Exponential so we would try until the upper limit of 300 seconds was reached by starting with a base wait time of 200 ms which would get multiplied by a factor of ~1.41 (sqrt(2)) on every attempts.

Getting some data from a REST API or arbitrary web server could then be written as easy as this:

var result = Retry.Invoke(() => {
  using (var client = new HttpClient())
  using (var response = await client.GetAsync("http://www.example.com"))
  using (var content = response.Content)
  {
    return await content.ReadAsStringAsync();
  }
});

As long as the server returns some data (and no exception is generated) the data will be returned. Upon receiving an exception (e.g. HTTP 500) the attempt would be retried.

Retrying operations based on the exception can then be controlled as seen in the following example. Only if an HttpRequestException with an inner WebException.StatusCode == NameResolutionFailure is thrown the attempt will be retried:

var result = Retry.Invoke(() => {
  using (var client = new HttpClient())
  using (var response = await client.GetAsync("http://www.example.com"))
  using (var content = response.Content)
  {
    return await content.ReadAsStringAsync();
  }
},
exceptionFilter: ex => {
var httpRequestException as HttpRequestException;
if(null == httpRequestException)
{
  throw;
}

if
(
  ex.InnerException is WebException && 
  WebExceptionStatus.NameResolutionFailure == ((WebException) ex.InnerException).Status
)
{
  // try again
  return;
}

// abort retry
throw;

},
defaultReturnValueFactory: null);

For custom retry operations, Retry.Invoke can also pass in the RetryStrategy. This might be handy, if we want a custom retry based on a number of occurrences of a specific exception. For this the strategy provides an StateInfo property that an exception filter can use to persist information between the various retries:

var result = Retry.Invoke(strategy => {
  var stateInfo = null == strategy.StateInfo
    ? new StateInfo()
    : (StateInfo) strategy.StateInfo;

  Logger.GetOrDefault().TraceInformation("{0}: {1}", strategy.CurrentAttempt, stateInfo.CorrelationId);

  using (var client = new HttpClient())
  using (var response = await client.GetAsync("http://www.example.com"))
  using (var content = response.Content)
  {
    return await content.ReadAsStringAsync();
  }
},
exceptionFilter: (strategy, ex) => {
  var stateInfo = (StateInfo) strategy.StateInfo;
  stateInfo.CorrelationId = new SomeCorrelationManager(ex).CorrelationId;
},
defaultReturnValueFactory: null);

This is it for today’s post. Attached you find the complete source:

/**
* Copyright 2017 d-fens GmbH
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
using System;
using System.Diagnostics;
using System.Diagnostics.Contracts;
namespace Net.Appclusive.Core.Domain.General
{
public class Retry
{
public static RetryStrategyExponential ExponentialStrategy = new RetryStrategyExponential();
public static RetryStrategyIncremental IncrementalStrategy = new RetryStrategyIncremental();
public static RetryStrategyFixed FixedStrategy = new RetryStrategyFixed();
public static TResult Invoke<TResult>
(
Func<RetryStrategyBase, TResult> func,
Action<RetryStrategyBase, Exception> exceptionFilter = null,
Func<RetryStrategyBase, TResult> defaultReturnValueFactory = null,
RetryStrategyExponential strategy = null
)
{
Contract.Requires(null != func);
if (null == strategy)
{
strategy = ExponentialStrategy;
}
var waitTime = strategy.MinWaitTimeIntervalMilliseconds;
var stopwatch = Stopwatch.StartNew();
for (;;)
{
try
{
return func.Invoke(strategy);
}
catch (Exception ex)
{
exceptionFilter?.Invoke(strategy, ex);
}
if (strategy.CurrentAttempt >= strategy.MaxAttempts)
{
break;
}
// check if maximum time has already elapsed
if (stopwatch.ElapsedMilliseconds >= strategy.MaxWaitTimeMilliseconds)
{
break;
}
// wait for next retry
System.Threading.Thread.Sleep(waitTime);
// get next wait time interval
waitTime = strategy.GetNextWaitTime(waitTime);
strategy.NextAttempt();
}
return null != defaultReturnValueFactory
? defaultReturnValueFactory.Invoke(strategy)
: default(TResult);
}
public static TResult Invoke<TResult>
(
Func<TResult> func,
Action<Exception> exceptionFilter = null,
Func<TResult> defaultReturnValueFactory = null,
RetryStrategyExponential strategy = null
)
{
Contract.Requires(null != func);
if (null == strategy)
{
strategy = ExponentialStrategy;
}
var waitTime = strategy.MinWaitTimeIntervalMilliseconds;
var stopwatch = Stopwatch.StartNew();
for (;;)
{
try
{
return func.Invoke();
}
catch (Exception ex)
{
exceptionFilter?.Invoke(ex);
}
if (strategy.CurrentAttempt >= strategy.MaxAttempts)
{
break;
}
// check if maximum time has already elapsed
if (stopwatch.ElapsedMilliseconds >= strategy.MaxWaitTimeMilliseconds)
{
break;
}
// wait for next retry
System.Threading.Thread.Sleep(waitTime);
// get next wait time interval
waitTime = strategy.GetNextWaitTime(waitTime);
strategy.NextAttempt();
}
return null != defaultReturnValueFactory
? defaultReturnValueFactory.Invoke()
: default(TResult);
}
}
}

view raw
Retry.cs
hosted with ❤ by GitHub

/**
* Copyright 2017 d-fens GmbH
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
namespace Net.Appclusive.Core.Domain.General
{
public abstract class RetryStrategyBase
{
public int MaxAttempts = int.MaxValue;
public int CurrentAttempt = 1;
public int MaxWaitTimeMilliseconds = 300 * 1000;
public int MinWaitTimeIntervalMilliseconds = 200;
public int MaxWaitTimeIntervalMilliseconds = 20 * 1000;
public object StateInfo;
public int NextAttempt()
{
return ++CurrentAttempt;
}
public abstract int GetNextWaitTime(int currentWaitTime);
}
}

view raw
RetryStrategyBase.cs
hosted with ❤ by GitHub

/**
* Copyright 2017 d-fens GmbH
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
using System;
using System.Diagnostics.Contracts;
namespace Net.Appclusive.Core.Domain.General
{
public class RetryStrategyCustom : RetryStrategyBase
{
private readonly Func<RetryStrategyCustom, int> waitTimeFactory;
protected RetryStrategyCustom(Func<RetryStrategyCustom, int> waitTimeFactory)
{
Contract.Requires(null != waitTimeFactory);
this.waitTimeFactory = waitTimeFactory;
}
public override int GetNextWaitTime(int currentWaitTime)
{
return waitTimeFactory(this);
}
}
}

/**
* Copyright 2017 d-fens GmbH
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
using System;
namespace Net.Appclusive.Core.Domain.General
{
public class RetryStrategyExponential : RetryStrategyBase
{
public double Factor = Math.Sqrt(2);
public override int GetNextWaitTime(int currentWaitTime)
{
return (int) Math.Min(MaxWaitTimeIntervalMilliseconds, Math.Max(MinWaitTimeIntervalMilliseconds, currentWaitTime * Factor));
}
}
}

/**
* Copyright 2017 d-fens GmbH
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
namespace Net.Appclusive.Core.Domain.General
{
public class RetryStrategyFixed : RetryStrategyBase
{
public override int GetNextWaitTime(int currentWaitTime)
{
return currentWaitTime;
}
}
}

/**
* Copyright 2017 d-fens GmbH
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
using System;
namespace Net.Appclusive.Core.Domain.General
{
public class RetryStrategyIncremental : RetryStrategyBase
{
public override int GetNextWaitTime(int currentWaitTime)
{
return Math.Min(MaxWaitTimeIntervalMilliseconds, Math.Max(MinWaitTimeIntervalMilliseconds, currentWaitTime));
}
}
}

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: