About a year and a half ago, I was trying to figure out what mocking library to use for testing an app I was writing. I had been burned in the past by an enterprise system with thousands of unit tests, where the mocking library used was the bottleneck in the test runs, causing them to take several minutes to run.
Setting out to determine the fastest library to avoid this problem, I created the mouthful BenchmarkMockNet to benchmark those which worked with .NET Core, leveraging BenchmarkDotNet under the hood (hence the name).
Fast-forward to the present, BenchmarkDotNet has since become part of the .NET Foundation, and my BenchmarkMockNet results have been updated with the latest versions of everything!
I won't go into the details of the results here, you can check those out in the repo, but here's one table (creating the mock and having one of its methods return a value) to show the overall theme:
|Stub||2.111 ns||0.5955 ns||0.1547 ns||1.00||0.00|
|FakeItEasy||45,354.683 ns||2,775.1171 ns||720.8263 ns||21,574.78||1,449.12|
|Moq||11,737.904 ns||261.7736 ns||67.9947 ns||5,583.61||367.65|
|NSubstitute||19,128.425 ns||544.4572 ns||141.4207 ns||9,099.20||600.30|
|Rocks||8,047.362 ns||4,548.2656 ns||1,181.3951 ns||3,828.05||562.92|
The first thing that pops out is that, compared to the simple Stub class as a baseline, you can really see the cost of these libraries compared to the more static DIY approach. Of course, the flip-side is less code to write yourself, so you'll need to determine your own tipping point there.
Bottom line: if you can afford to maintain your own stubs, do it.
The other thing that pops out is that the fastest mocking library is relative-newcomer Rocks, which uses Roslyn's Compiler API instead of hooking directly into IL (hence its name, R[oslyn M]ocks). This becomes much more apparent when simulating tests with fixture- or assembly-level setup methods, or those with many parts to mock (which, for the record, I would argue is an anti-pattern):
|Stub||3.322 ns||0.6730 ns||0.1748 ns||1.00||0.00|
|FakeItEasy||16,297.547 ns||30,322.5410 ns||7,876.1672 ns||4,916.27||2,139.20|
|Moq||870.410 ns||137.8692 ns||35.8110 ns||262.57||15.44|
|NSubstitute||4,408.761 ns||264.2877 ns||68.6478 ns||1,329.93||63.70|
|Rocks||19.675 ns||2.1524 ns||0.5591 ns||5.94||0.31|
Notice how Rocks is the same order of magnitude as the stub -- Roslyn FTW!
Do you know of any .NET Core mocking libraries that aren't included in these tests? Any other scenarios that would be useful to get numbers for?
Hacktoberfest is upon us!
The README has clear instructions on how to add these, so feel free to submit a PR and have it count towards your 5 (yes, it's up to five this year) to get a t-shirt.