Creating benchmarks with BenchmarkDotNet is very easy and running them is even easier with the Visual C# Interactive Compiler (CSI for short). If you need more information about CSI, read Essential .NET – C# Scripting to get started).

And here is how it works:

1. Select the Developer Command Prompt for VS2015 from the Start Menu
2. Run csi.exe
3. Change to the bin\Release (or bin\Debug) directory of your project
4. Import the BenchmarkDotNet assembly
5. Import your assembly with defined benchmarks
6. Add your usings for BenchmarkDotNet and your benchmark classes
7. Run your benchmarks and wait to see its outcome

#r "biz.dfch.CS.Commons.Benchmarks.dll"
#r "BenchmarkDotNet.dll"

using BenchmarkDotNet.Running;
using biz.dfch.CS.Commons.Benchmarks;

var summary = BenchmarkDotNet.Running.BenchmarkRunner.Run<MyBenchmarks>();

An (abbreviated) example output might look like this:

MyBenchmarks.MyBenchmark: DefaultJob
Runtime = Clr 4.0.30319.42000, 64bit RyuJIT-v4.6.1586.0; GC = Concurrent Workstation
Mean = 17.3845 ns, StdErr = 0.1444 ns (0.83%); N = 12, StdDev = 0.5003 ns
Min = 16.7288 ns, Q1 = 17.0285 ns, Median = 17.2448 ns, Q3 = 17.7290 ns, Max = 18.4030 ns
IQR = 0.7005 ns, LowerFence = 15.9776 ns, UpperFence = 18.7798 ns
ConfidenceInterval = [17.1015 ns; 17.6676 ns] (CI 95%)
Skewness = 0.37, Kurtosis = 2.08


Total time: 00:00:46 (46.54 sec)

// * Summary *

BenchmarkDotNet=v0.10.1, OS=Microsoft Windows NT 6.2.9200.0
Processor=Intel(R) Core(TM) i7-6700HQ CPU 2.60GHz, ProcessorCount=8
Frequency=2531249 Hz, Resolution=395.0619 ns, Timer=TSC
  [Host]     : Clr 4.0.30319.42000, 64bit RyuJIT-v4.6.1586.0
  DefaultJob : Clr 4.0.30319.42000, 64bit RyuJIT-v4.6.1586.0

Gen 0=0.0074  Allocated=24 B

      Method |       Mean |    StdDev |
------------ |----------- |---------- |
 MyBenchmark | 17.3845 ns | 0.5003 ns |

*** Warnings ***
Environment
  MyBenchmarks.MyBenchmark: Default -> Benchmark was built in DEBUG configuration. Please, build it in RELEASE.

*** Hints ***
Outliers
  MyBenchmarks.MyBenchmark: Default -> 3 outliers were removed

// * Diagnostic Output - MemoryDiagnoser *
Note: the Gen 0/1/2 Measurements are per 1k Operations

// ***** BenchmarkRunner: End *****

You can further check the summary.ResultsDirectoryPath property to inspect the contents of the various report files (eg. in markdown or HTML).

If you prefer to roll the whole thing on your own without a precompiled assembly with your benchmarks, Roslyn is there to help you. To make this work we have to Install-Package BenchmarkDotNet and copy all the downloaded assemblies to a single location. From there we dynamically define our benchmarks, compile them on the fly and run them afterwards:

#r "BenchmarkDotNet.dll"
#r "Microsoft.CodeAnalysis.dll"

using System.Reflection;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;
using BenchmarkDotNet.Toolchains;
using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.CSharp;

var source = @"
using BenchmarkDotNet.Attributes;

namespace biz.dfch.CS.Commons.Benchmarks
{
  public class MyBenchmarks
  {
    [Benchmark]
    public static void MyBenchmark()
    {
      var sut = new object();
    }
  }
}
";

var assemblyName = Path.GetRandomFileName();
var assemblyLocation = Path.Combine(Directory.GetCurrentDirectory(), string.Concat(assemblyName, ".dll"));

var syntaxTree = CSharpSyntaxTree.ParseText(source);
var references = new[]
{
  MetadataReference.CreateFromFile(typeof(object).Assembly.Location),
  MetadataReference.CreateFromFile(typeof(BenchmarkRunner).Assembly.Location),
  MetadataReference.CreateFromFile(typeof(BenchmarkAttribute).Assembly.Location),
};
var compilation = CSharpCompilation.Create
(
  assemblyName,
  new[] { syntaxTree },
  references,
  new CSharpCompilationOptions(OutputKind.DynamicallyLinkedLibrary)
);

var emitResult = compilation.Emit(assemblyLocation);

var assembly = Assembly.Load(AssemblyName.GetAssemblyName(assemblyLocation));
var type = assembly.DefinedTypes.First();

var summary = BenchmarkRunner.Run(type);

Note: when compiling on the fly, we have to add references manually via MetadataReference. In case we missed something, the assembly will not compile, which we can check via emitResult.Success and emitResult.Diagnostics. To actually invoke the benchmark runner we have to retrieve the type from the dynamically loaded assembly and its DefinedTypes collection. In the example I just ran against the First() type – adjust accordingly. To make things a little bit easier we could also load the benchmarks from CSharp files via File.ReadAllText().

As it is quite difficult to spot missing references and coding errors I would recommend to leave the benchmarks in a separate pre-compiled assembly and just run the tests ad-hoc via CSI (as shown in the first part of the post).

Note: when compiling your assemblies yourself, make sure they have a .DLL extension, otherwise the BenchmarkRunner will fail with a NullReferenceException as described in BenchmarkDotNet/issues/354.