BenchmarkMockNet: Benchmarking .NET Mocking Libraries

Analysis on the latest results from my tongue-twister benchmarking library

About a year and a half ago, I was trying to figure out what mocking library to use for testing an app I was writing. I had been burned in the past by an enterprise system with thousands of unit tests, where the mocking library used was the bottleneck in the test runs, causing them to take several minutes to run.

Setting out to determine the fastest library to avoid this problem, I created the mouthful BenchmarkMockNet to benchmark those which worked with .NET Core, leveraging BenchmarkDotNet under the hood (hence the name).

Fast-forward to the present, BenchmarkDotNet has since become part of the .NET Foundation, and my BenchmarkMockNet results have been updated with the latest versions of everything!

I won't go into the details of the results here, you can check those out in the repo, but here's one table (creating the mock and having one of its methods return a value) to show the overall theme:

Method Mean Error StdDev Scaled ScaledSD
Stub 2.111 ns 0.5955 ns 0.1547 ns 1.00 0.00
FakeItEasy 45,354.683 ns 2,775.1171 ns 720.8263 ns 21,574.78 1,449.12
Moq 11,737.904 ns 261.7736 ns 67.9947 ns 5,583.61 367.65
NSubstitute 19,128.425 ns 544.4572 ns 141.4207 ns 9,099.20 600.30
Rocks 8,047.362 ns 4,548.2656 ns 1,181.3951 ns 3,828.05 562.92

The first thing that pops out is that, compared to the simple Stub class as a baseline, you can really see the cost of these libraries compared to the more static DIY approach. Of course, the flip-side is less code to write yourself, so you'll need to determine your own tipping point there.

Bottom line: if you can afford to maintain your own stubs, do it.

The other thing that pops out is that the fastest mocking library is relative-newcomer Rocks, which uses Roslyn's Compiler API instead of hooking directly into IL (hence its name, R[oslyn M]ocks). This becomes much more apparent when simulating tests with fixture- or assembly-level setup methods, or those with many parts to mock (which, for the record, I would argue is an anti-pattern):

Method Mean Error StdDev Scaled ScaledSD
Stub 3.322 ns 0.6730 ns 0.1748 ns 1.00 0.00
FakeItEasy 16,297.547 ns 30,322.5410 ns 7,876.1672 ns 4,916.27 2,139.20
Moq 870.410 ns 137.8692 ns 35.8110 ns 262.57 15.44
NSubstitute 4,408.761 ns 264.2877 ns 68.6478 ns 1,329.93 63.70
Rocks 19.675 ns 2.1524 ns 0.5591 ns 5.94 0.31

Notice how Rocks is the same order of magnitude as the stub -- Roslyn FTW!

Do you know of any .NET Core mocking libraries that aren't included in these tests? Any other scenarios that would be useful to get numbers for?

Hacktoberfest is upon us!

The README has clear instructions on how to add these, so feel free to submit a PR and have it count towards your 5 (yes, it's up to five this year) to get a t-shirt.

Happy mocking!