Ad Hoc Performance Benchmarking with @EgorBot

When you need to validate the performance impact of a code change, follow this process to write a BenchmarkDotNet benchmark and trigger @EgorBot to run it. The bot will notify you when results are ready, so don't wait for them.

Step 1: Write the Benchmark

Create a BenchmarkDotNet benchmark that tests the specific operation being changed. Follow these guidelines:

Benchmark Structure

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

BenchmarkSwitcher.FromAssembly(typeof(Bench).Assembly).Run(args);

public class Bench
{
    // Add setup/cleanup if needed
    [GlobalSetup]
    public void Setup()
    {
        // Initialize test data
    }

    [Benchmark]
    public void MyOperation()
    {
        // Test the operation
    }
}

Best Practices

For comprehensive guidance, see the Microbenchmark Design Guidelines.

Key principles:

• Move initialization to [GlobalSetup]: Separate setup logic from the measured code to avoid measuring allocation/initialization overhead
• Return values from benchmark methods to prevent dead code elimination
• Avoid loops: BenchmarkDotNet invokes the benchmark many times automatically; adding manual loops distorts measurements
• No side effects: Benchmarks should be pure and produce consistent results
• Focus on common cases: Benchmark hot paths and typical usage, not edge cases or error paths
• Use consistent input data: Always use the same test data for reproducible comparisons
• Avoid [DisassemblyDiagnoser]: It causes crashes on Linux. Use --envvars DOTNET_JitDisasm:MethodName instead
• Benchmark class requirements: Must be public, not sealed, not static, and must be a class (not struct)

Example: String Operation Benchmark

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

BenchmarkSwitcher.FromAssembly(typeof(Bench).Assembly).Run(args);

[MemoryDiagnoser]
public class Bench
{
    private string _testString = default!;

    [Params(10, 100, 1000)]
    public int Length { get; set; }

    [GlobalSetup]
    public void Setup()
    {
        _testString = new string('a', Length);
    }

    [Benchmark]
    public int StringOperation()
    {
        return _testString.IndexOf('z');
    }
}

Example: Collection Operation Benchmark

using System.Linq;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

BenchmarkSwitcher.FromAssembly(typeof(Bench).Assembly).Run(args);

[MemoryDiagnoser]
public class Bench
{
    private int[] _array = default!;
    private List<int> _list = default!;

    [Params(100, 1000, 10000)]
    public int Count { get; set; }

    [GlobalSetup]
    public void Setup()
    {
        _array = Enumerable.Range(0, Count).ToArray();
        _list = _array.ToList();
    }

    [Benchmark]
    public bool AnyArray() => _array.Any();

    [Benchmark]
    public bool AnyList() => _list.Any();

    [Benchmark]
    public int SumArray() => _array.Sum();

    [Benchmark]
    public int SumList() => _list.Sum();
}

Step 2: Mention @EgorBot in a comment/PR description

Post a comment on the PR to trigger EgorBot with your benchmark. The general format is:

@EgorBot [targets] [options] [BenchmarkDotNet args]

// Your benchmark code here

Note: The @EgorBot command must not be inside the code block. Only the benchmark code should be inside the code block.

Target Flags

• -linux_amd
• -linux_intel
• -windows_amd
• -windows_intel
• -linux_arm64
• -osx_arm64 (baremetal, feel free to always include it)

The most common combination is -linux_amd -osx_arm64. Do not include more than 4 targets.

Common Options

Use -profiler when absolutely necessary along with -linux_arm64 and/or -linux_amd to include perf profiling and disassembly in the results.

Example: Basic PR Benchmark

To benchmark the current PR changes against the base branch:

@EgorBot -linux_amd -osx_arm64

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

BenchmarkSwitcher.FromAssembly(typeof(Bench).Assembly).Run(args);

[MemoryDiagnoser]
public class Bench
{
    [Benchmark]
    public int MyOperation()
    {
        // Your benchmark code
        return 42;
    }
}

Important Notes

• Bot response time: EgorBot uses polling and may take up to 30 seconds to respond
• Supported repositories: EgorBot monitors dotnet/runtime and EgorBot/runtime-utils
• PR mode (default): When posting in a PR, EgorBot automatically compares the PR changes against the base branch
• Results variability: Results may vary between runs due to VM differences. Do not compare results across different architectures or cloud providers
• Check the manual: EgorBot replies include a link to the manual for advanced options

Additional Resources

• Microbenchmark Design Guidelines - Essential reading for writing effective benchmarks
• BenchmarkDotNet CLI Arguments
• EgorBot Manual