Inspired by Transient Fault Handling we created a quick helper class to faciliate the automatic retry of various operation that may fail in today’s interconnected environments.

Large parts of our Appclusive framework deal with fetching from and pushing information to various backend systems. As they all can (and eventuall will) fail we needed a flexible approach on how to retry these failed operations. When we first addresses this issue we used the The Transient Fault Handling Application Block from Microsoft patterns & practices. It however turned out for us that this was either too much overhead or not flexible enough to deal with various errors and exceptions we experienced in our environments. There we decided not to “reinvent the wheel” but add some functionality to it and decouple if from the application block (as it did not seem to get updated in quite some time).

What we needed

1. We needed an easy method for executing arbitrary piece of code with a given retry strategy. The default strategy would use some kind of exponential back-off timer but with an upper limit. In addition we would favour a maximum retry time over a certain number of retry attempts.

2. We wanted to support different retry strategies and be open for custom implementations. The basic retry strategies sould be:
   a. Exponential
   b. Fixed
   c. Incremental
   d. Custom

3. Default return values should be optional and be used in cases where none of the retries would have been successful.

4. Exception filters should allow for custom exception handling so it can be decided depending on the exception if retries should be performed or not.

What we got

We defined a base class for all our retry strategies which looks like this (you can find the complete code here or at the end of this blog post):

public abstract class RetryStrategyBase
{
  public int MaxAttempts = int.MaxValue;

  public int CurrentAttempt = 1;

  public int MaxWaitTimeMilliseconds = 300 * 1000;

  public int MinWaitTimeIntervalMilliseconds = 200;

  public int MaxWaitTimeIntervalMilliseconds = 20 * 1000;

  public object StateInfo;

  public int NextAttempt()
  {
    return ++CurrentAttempt;
  }

  public abstract int GetNextWaitTime(int currentWaitTime);
}

Our Retry would take whatever strategy we specified (or take the Exponential defaulT) and could be invoke as easy as this:

var result = Retry.Invoke
(
  func: () => 1 / 0, 
  exceptionFilter: null, 
  defaultReturnValueFactory: () => 42
);
Assert.AreEqual(42, result);

Here we would try to divide by zero, and return a default value of 42 in case we would not succeed. The chosen retry strategy defaults to Exponential so we would try until the upper limit of 300 seconds was reached by starting with a base wait time of 200 ms which would get multiplied by a factor of ~1.41 (sqrt(2)) on every attempts.

Getting some data from a REST API or arbitrary web server could then be written as easy as this:

var result = Retry.Invoke(() => {
  using (var client = new HttpClient())
  using (var response = await client.GetAsync("http://www.example.com"))
  using (var content = response.Content)
  {
    return await content.ReadAsStringAsync();
  }
});

As long as the server returns some data (and no exception is generated) the data will be returned. Upon receiving an exception (e.g. HTTP 500) the attempt would be retried.

Retrying operations based on the exception can then be controlled as seen in the following example. Only if an HttpRequestException with an inner WebException.StatusCode == NameResolutionFailure is thrown the attempt will be retried:

var result = Retry.Invoke(() => {
  using (var client = new HttpClient())
  using (var response = await client.GetAsync("http://www.example.com"))
  using (var content = response.Content)
  {
    return await content.ReadAsStringAsync();
  }
},
exceptionFilter: ex => {
var httpRequestException as HttpRequestException;
if(null == httpRequestException)
{
  throw;
}

if
(
  ex.InnerException is WebException && 
  WebExceptionStatus.NameResolutionFailure == ((WebException) ex.InnerException).Status
)
{
  // try again
  return;
}

// abort retry
throw;

},
defaultReturnValueFactory: null);

For custom retry operations, Retry.Invoke can also pass in the RetryStrategy. This might be handy, if we want a custom retry based on a number of occurrences of a specific exception. For this the strategy provides an StateInfo property that an exception filter can use to persist information between the various retries:

var result = Retry.Invoke(strategy => {
  var stateInfo = null == strategy.StateInfo
    ? new StateInfo()
    : (StateInfo) strategy.StateInfo;

  Logger.GetOrDefault().TraceInformation("{0}: {1}", strategy.CurrentAttempt, stateInfo.CorrelationId);

  using (var client = new HttpClient())
  using (var response = await client.GetAsync("http://www.example.com"))
  using (var content = response.Content)
  {
    return await content.ReadAsStringAsync();
  }
},
exceptionFilter: (strategy, ex) => {
  var stateInfo = (StateInfo) strategy.StateInfo;
  stateInfo.CorrelationId = new SomeCorrelationManager(ex).CorrelationId;
},
defaultReturnValueFactory: null);

This is it for today’s post. Attached you find the complete source: