Deep dive: Minitest

23 Jun, 2019

0 views

ruby

Have you ever wondered what happens when you run a Minitest test suite? How does it work?

This article will demystify Minitest’s magic by going deep into its source code. After reading it, you’ll no longer consider Minitest a magical black box and will understand how Minitest discovers your tests methods and executes them.

Reinvent the wheel

Although the standard programming wisdom instructs us to avoid it, reinventing the wheel is a great way to learn the basic principles about the wheel.

So how does Minitest work?

Let’s write a simplest code possible that demonstrates how it’s used:

require "minitest/autorun"
class MathTest < Minitest::Test
 def test_two_plus_two
 assert 2 + 2 == 4
 end
end

When we run it, we get the following output:

Run options: --seed 22395
# Running:
.
Finished in 0.000802s, 1246.8827 runs/s, 1246.8827 assertions/s.
1 runs, 1 assertions, 0 failures, 0 errors, 0 skips

Let’s try to figure out how it works by removing the require line. If we run it without the require, we’ll get the following error:

Traceback (most recent call last):
test.rb:3:in `<main>': uninitialized constant Minitest (NameError)

It looks like we’re going to need to define that class, so let’s do that:

module Minitest
 class Test; end
end
class MathTest < Minitest::Test
 def test_two_plus_two
 assert 2 + 2 == 4
 end
end

If we run it, absolutely nothing will happen, but at least it will not blow up.

The next step is actually running the test. We need to do the following:

figure out which classes have inherited our test class
call instance methods that begin with test_

In order to do 1, we need to add some sort of descendant tracking to our Test class. Let’s add it:

module Minitest
 class Test
 def self.inherited(klass)
 @descendants ||= []
 @descendants << klass
 end

 def self.descendants
 @descendants || []
 end
 end
end
class MathTest < Minitest::Test
 def test_two_plus_two
 assert 2 + 2 == 4
 end
end

Now we’re ready for 2, so let’s call the methods that start with test_. Since these methods are instance methods, it makes sense to instantiate descendant classes we have tracked:

module Minitest
 class Test
 def self.inherited(klass)
 @descendants ||= []
 @descendants << klass
 end
 def self.descendants
 @descendants || []
 end
 end
end
class MathTest < Minitest::Test
 def test_two_plus_two
 assert 2 + 2 == 4
 end
end
Minitest::Test.descendants.each do |klass|
 klass.new.tap do |instance|
 instance.methods.grep(/^test_/).each do |method_name|
 instance.public_send(method_name)
 end
 end
end

If we run this we’ll get another error:

Traceback (most recent call last):
 6: from test.rb:22:in `<main>'
 5: from test.rb:22:in `each'
 4: from test.rb:24:in `block in <main>'
 3: from test.rb:24:in `each'
 2: from test.rb:25:in `block (2 levels) in <main>'
 1: from test.rb:25:in `public_send'
test.rb:18:in `test_two_plus_two': undefined method `assert' for #<MathTest:0x00007f8eeb0f4ef8> (NoMethodError)

This is just what we’ve expected since we didn’t define the assert method yet. Let’s add it:

module Minitest
 class Test
 def self.inherited(klass)
 @descendants ||= []
 @descendants << klass
 end
 def self.descendants
 @descendants || []
 end
 def assert(condition)
 print '.' if condition
 end
 end
end
class MathTest < Minitest::Test
 def test_two_plus_two
 assert 2 + 2 == 4
 end
end
Minitest::Test.descendants.each do |klass|
 klass.new.tap do |instance|
 instance.methods.grep(/^test_/).each do |method_name|
 instance.public_send(method_name)
 end
 end
end

Run it and witness the dot that is printed out in all its glory:

So far so good, but we need to report the number of runs, assertions and failures, in addition to dots.

Let’s count the number of assertions:

module Minitest
 class Test
 def initialize
 @assertions_count = 0
 end
 attr_reader :assertions_count
 def self.inherited(klass)
 @descendants ||= []
 @descendants << klass
 end
 def self.descendants
 @descendants || []
 end
 def assert(condition)
 @assertions_count += 1
 print '.' if condition
 end
 end
end
class MathTest < Minitest::Test
 def test_two_plus_two
 assert 2 + 2 == 4
 end
end
Minitest::Test.descendants.each do |klass|
 klass.new.tap do |instance|
 instance.methods.grep(/^test_/).each do |method_name|
 instance.public_send(method_name)
 end
 end
end

Nice, we’re now counting the assertions. The only problem now is printing it. Let’s start printing a simple report:

module Minitest
 class Test
 def initialize
 @assertions_count = 0
 end
 attr_reader :assertions_count
 def self.inherited(klass)
 @descendants ||= []
 @descendants << klass
 end
 def self.descendants
 @descendants || []
 end
 def assert(condition)
 @assertions_count += 1
 print '.' if condition
 end
 end
end
class MathTest < Minitest::Test
 def test_two_plus_two
 assert 2 + 2 == 4
 end
end
Minitest::Test.descendants.map do |klass|
 klass.new.tap do |instance|
 instance.methods.grep(/^test_/).each do |method_name|
 instance.public_send(method_name)
 end
 end
end.each_with_object(runs: 0, assertions: 0) do |instance, counter|
 counter[:assertions] += instance.assertions_count
 counter[:runs] += instance.methods.grep(/^test_/).count
end.tap do |counter|
 puts "\n\n#{counter[:runs]} runs, " \
 "#{counter[:assertions]} assertions, " \
 '0 failures, 0 errors, 0 skips'
end

Run this code, and you’ll get the report printed out:

.
1 runs, 1 assertions, 0 failures, 0 errors, 0 skips

Great!

However, this is not precisely how Minitest works. Minitest doesn’t magically add code to the bottom of our tests to print its report. Remember, we require Minitest before our test code, so we need to keep our "test library" code before our tests.

We need to print the report when everything else has finished executing, and luckily Ruby comes with the Kernel#at_exit method we can use for this:

module Minitest
 class Test
 def initialize
 @assertions_count = 0
 end
 attr_reader :assertions_count
 def self.inherited(klass)
 @descendants ||= []
 @descendants << klass
 end
 def self.descendants
 @descendants || []
 end
 def assert(condition)
 @assertions_count += 1
 print '.' if condition
 end
 end
end
def do_the_wizardry
 Minitest::Test.descendants.map do |klass|
 klass.new.tap do |instance|
 instance.methods.grep(/^test_/).each do |method_name|
 instance.public_send(method_name)
 end
 end
 end.each_with_object(runs: 0, assertions: 0) do |instance, counter|
 counter[:assertions] += instance.assertions_count
 counter[:runs] += instance.methods.grep(/^test_/).count
 end.tap do |counter|
 puts "\n\n#{counter[:runs]} runs, " \
 "#{counter[:assertions]} assertions, " \
 '0 failures, 0 errors, 0 skips'
 end
end

at_exit { do_the_wizardry }

class MathTest < Minitest::Test
 def test_two_plus_two
 assert 2 + 2 == 4
 end
end

Excellent! Our library code is now above the test code, which is also how Minitest works and does its magic.

A peek under the hood

Since we’re now total experts when it comes to test libraries, we are no longer afraid to look under the hood of Minitest^[1].

git clone https://github.com/seattlerb/minitest.git
cd minitest
git checkout 1f2b132

Our assumption is that at_exit is used somewhere, to print the report we get when running our test suite. Let’s confirm it:

grep at_exit -nR .

This will bring us to:

lib/minitest.rb#L52-L66

def self.autorun
 at_exit {
 next if $! and not ($!.kind_of? SystemExit and $!.success?)

 exit_code = nil
 at_exit {
 @@after_run.reverse_each(&:call)
 exit exit_code || false
 }
 exit_code = Minitest.run ARGV
 } unless @@installed_at_exit
 @@installed_at_exit = true
end

Which uses at_exit to instruct Minitest to run after all the other code has been executed and the program is exiting. The first line skips invoking Minitest if the exception is raised and it’s not SystemExit with zero status. ^[2]

This hook is only set up if @@installed_at_exit has not been set, ensuring the hook will only be set up once. This allows requiring minitest/autorun multiple times and not having to worry about what will happen with the at_exit hook (imagine multiple test files each requiring minitest/autorun).

This brings us to the inside of the block:

lib/minitest.rb#L52-L66

def self.autorun
 at_exit {
 next if $! and not ($!.kind_of? SystemExit and $!.success?)
 exit_code = nil

 at_exit {
 @@after_run.reverse_each(&:call)
 exit exit_code || false
 }

 exit_code = Minitest.run ARGV
 } unless @@installed_at_exit
 @@installed_at_exit = true
end

Notice the little trick with exit_code being set to nil first, before assigning it to the result of Minitest.run. This is used to ensure that the at_exit block runs regardless of the result of Minitest.run.^[3]

This second at_exit hook will be executed after Minitest finishes its execution, so this is setting up the second layer of code that’s going to be run when the program is exiting.

In its block @@after_run is being called in reverse order. Where is it coming from?

lib/minitest.rb#L10-L15

module Minitest
 VERSION = "5.11.3" # :nodoc:
 ENCS = "".respond_to? :encoding # :nodoc:
 @@installed_at_exit ||= false
 @@after_run = []

Ah, an ordinary array, but we’re calling .call on its items. What does it store?

lib/minitest.rb#L68-L76

##
# A simple hook allowing you to run a block of code after everything
# is done running. Eg:
#
# Minitest.after_run { p $debugging_info }
def self.after_run &block
 @@after_run << block
end

Nice, so this is how Minitest keeps track of its after_run callbacks, it appends blocks passed to Minitest.after_run to an ordinary array - nothing magical.

Now that we’ve demystified other parts of the at_exit hook that Minitest uses to deploy its magic, let’s take a look at the one thing that’s left:

lib/minitest.rb#L52-L66

def self.autorun
 at_exit {
 next if $! and not ($!.kind_of? SystemExit and $!.success?)
 exit_code = nil
 at_exit {
 @@after_run.reverse_each(&:call)
 exit exit_code || false
 }
 exit_code = Minitest.run ARGV
 } unless @@installed_at_exit
 @@installed_at_exit = true
end

This is the most important line of that method since it actually runs the tests. Let’s dig into it!

Starting the engine

Searching for it, we discover these methods:

lib/minitest.rb#L103-L144

##
# This is the top-level run method. Everything starts from here. It
# tells each Runnable sub-class to run, and each of those are
# responsible for doing whatever they do.
#
# The overall structure of a run looks like this:
#
# Minitest.autorun
# Minitest.run(args)
# Minitest.__run(reporter, options)
# Runnable.runnables.each
# runnable.run(reporter, options)
# self.runnable_methods.each
# self.run_one_method(self, runnable_method, reporter)
# Minitest.run_one_method(klass, runnable_method)
# klass.new(runnable_method).run

def self.run args = []
 self.load_plugins unless args.delete("--no-plugins") || ENV["MT_NO_PLUGINS"]
 options = process_args args
 reporter = CompositeReporter.new
 reporter << SummaryReporter.new(options[:io], options)
 reporter << ProgressReporter.new(options[:io], options)
 self.reporter = reporter # this makes it available to plugins
 self.init_plugins options
 self.reporter = nil # runnables shouldn't depend on the reporter, ever
 self.parallel_executor.start if parallel_executor.respond_to?(:start)
 reporter.start
 begin
 __run reporter, options
 rescue Interrupt
 warn "Interrupted. Exiting..."
 end
 self.parallel_executor.shutdown
 reporter.report
 reporter.passed?
end

lib/minitest.rb#L146-L161

##
# Internal run method. Responsible for telling all Runnable
# sub-classes to run.
def self.__run reporter, options
 suites = Runnable.runnables.
 reject { |s| s.runnable_methods.empty? }.
 shuffle
 parallel, serial = suites.partition { |s| s.test_order == :parallel }
 serial.map { |suite| suite.run reporter, options } +
 parallel.map { |suite| suite.run reporter, options }
end

So far so good, but we’re little confused now about this Runnable.runnables invocation. It looks like it’s an array, but where did it come from?

We track it down:

lib/minitest.rb#L373-L378

##
# Returns all subclasses of Runnable.
def self.runnables
 @@runnables
end

This doesn’t clear things up, so we keep searching and find this code:

lib/minitest.rb#L977-L982

class Runnable # re-open
 def self.inherited klass # :nodoc:
 self.runnables << klass
 super
 end
end

Ah! It uses the Class#inherited to track all the classes that have inherited Runnable. It’s a bit weird to discover that @@runnables is being appended to, but not assigned to an array yet. Here’s the missing piece:

lib/minitest.rb#L291-L295

def self.reset # :nodoc:
 @@runnables = []
end
reset

This clears things up about Runnable.runnables being an array of classes that inherit Runnable, but we still know nothing about the nature of those classes.

Doing a quick grep for < Runnable gives us a suspect:

lib/minitest.rb#L496

class Result < Runnable

Remember that we’re still stuck at this line:

lib/minitest.rb#L146-L161

##
# Internal run method. Responsible for telling all Runnable
# sub-classes to run.
def self.__run reporter, options
 suites = Runnable.runnables.
 reject { |s| s.runnable_methods.empty? }.
 shuffle
 parallel, serial = suites.partition { |s| s.test_order == :parallel }
 serial.map { |suite| suite.run reporter, options } +
 parallel.map { |suite| suite.run reporter, options }
end

But we now know ^{(or do we, cough, cough?)} that Runnable.runnables is [Result]. Looks like we’re rejecting classes from that array that have empty runnable_methods.

Tracking this method down leads us to the dead end:

lib/minitest.rb#L365-L371

##
# Each subclass of Runnable is responsible for overriding this
# method to return all runnable methods. See #methods_matching.
def self.runnable_methods
 raise NotImplementedError, "subclass responsibility"
end

What!?

Turns out, this nifty re-opening of Runnable class has a purpose. The Runnable.inherited is defined on line 977, while Result < Runnable happens on line 496. Looks like the order is important here. Who knew!?

# This happens first
class Runnable
 def self.reset # :nodoc:
 @@runnables = []
 end
 reset
 def self.runnables
 @@runnables
 end
end
class Result < Runnable; end
# And then we re-open the class
class Runnable # re-open
 def self.inherited klass # :nodoc:
 self.runnables << klass
 super
 end
end
# > Runnable.runnables
# => []

Our suspect is free to go. We have found another one:

lib/minitest/test.rb#L10

 class Test < Runnable

Interesting. But where do we even require that class?

lib/minitest.rb#L987

require "minitest/test"

Ah! That’s the last line of that file, so the inherited will kick in and catch this class. This suspect is now confirmed, and we can happily declare that Runnable.runnables contains [Test].

We now track down Test.runnable_methods:

lib/minitest/test.rb#L60-L77

##
# Returns all instance methods starting with "test_". Based on
# #test_order, the methods are either sorted, randomized
# (default), or run in parallel.
def self.runnable_methods
 methods = methods_matching(/^test_/)
 case self.test_order
 when :random, :parallel then
 max = methods.size
 methods.sort.sort_by { rand max }
 when :alpha, :sorted then
 methods.sort
 else
 raise "Unknown test_order: #{self.test_order.inspect}"
 end
end

This returns all the instance methods of this class, that start with test_ by using the Runnable#methods_matching:

lib/minitest.rb#L284-L289

##
# Returns all instance methods matching the pattern +re+.
def self.methods_matching re
 public_instance_methods(true).grep(re).map(&:to_s)
end

We can now finally move a couple of lines:

lib/minitest.rb#L146-L161

##
# Internal run method. Responsible for telling all Runnable
# sub-classes to run.
def self.__run reporter, options
 suites = Runnable.runnables.
 reject { |s| s.runnable_methods.empty? }.
 shuffle
 parallel, serial = suites.partition { |s| s.test_order == :parallel }

 serial.map { |suite| suite.run reporter, options } +
 parallel.map { |suite| suite.run reporter, options }
end

It looks like some partitioning is happening, but we don’t care because:

lib/minitest/test.rb#L79-L85

##
# Defines the order to run tests (:random by default). Override
# this or use a convenience method to change it for your tests.
def self.test_order
 :random
end

Which will put everything in serial (parallel will be empty array) and this leads us to:

lib/minitest.rb#L146-L161

##
# Internal run method. Responsible for telling all Runnable
# sub-classes to run.
def self.__run reporter, options
 suites = Runnable.runnables.
 reject { |s| s.runnable_methods.empty? }.
 shuffle
 parallel, serial = suites.partition { |s| s.test_order == :parallel }
 serial.map { |suite| suite.run reporter, options } +
 parallel.map { |suite| suite.run reporter, options }
end

Which finally calls Test.run, which Test inherits from Runnable:

lib/minitest.rb#L297-L324

##
# Responsible for running all runnable methods in a given class,
# each in its own instance. Each instance is passed to the
# reporter to record.

def self.run reporter, options = {}
 filter = options[:filter] || "/./"
 filter = Regexp.new 1ドル if filter =~ %r%/(.*)/%

 filtered_methods = self.runnable_methods.find_all { |m|
 filter === m || filter === "#{self}##{m}"
 }

 exclude = options[:exclude]
 exclude = Regexp.new 1ドル if exclude =~ %r%/(.*)/%

 filtered_methods.delete_if { |m|
 exclude === m || exclude === "#{self}##{m}"
 }

 return if filtered_methods.empty?
 with_info_handler reporter do
 filtered_methods.each do |method_name|
 run_one_method self, method_name, reporter
 end
 end
end

This filters methods by the --name and --exclude options provided from the command line. Let’s get to the juicy part:

lib/minitest.rb#L319-L323

##
# Responsible for running all runnable methods in a given class,
# each in its own instance. Each instance is passed to the
# reporter to record.
def self.run reporter, options = {}
 filter = options[:filter] || "/./"
 filter = Regexp.new 1ドル if filter =~ %r%/(.*)/%
 filtered_methods = self.runnable_methods.find_all { |m|
 filter === m || filter === "#{self}##{m}"
 }
 exclude = options[:exclude]
 exclude = Regexp.new 1ドル if exclude =~ %r%/(.*)/%
 filtered_methods.delete_if { |m|
 exclude === m || exclude === "#{self}##{m}"
 }
 return if filtered_methods.empty?
 with_info_handler reporter do
 filtered_methods.each do |method_name|
 run_one_method self, method_name, reporter
 end
 end
end

Which brings us to Runnable.run_one_method:

lib/minitest.rb#L326-L335

##
# Runs a single method and has the reporter record the result.
# This was considered internal API but is factored out of run so
# that subclasses can specialize the running of an individual
# test. See Minitest::ParallelTest::ClassMethods for an example.
def self.run_one_method klass, method_name, reporter
 reporter.prerecord klass, method_name
 reporter.record Minitest.run_one_method(klass, method_name)
end

Which passes the potato to Minitest.run_one_method:

lib/minitest.rb#L959-L963

def self.run_one_method klass, method_name # :nodoc:
 result = klass.new(method_name).run
 raise "#{klass}#run _must_ return a Result" unless Result === result
 result
end

Remember that klass here is Minitest::Test and method_name is each method defined in that class that starts with test_ and has survived filtering.

Let’s repeat this line since it’s perhaps the most elegant line in the entire project.

lib/minitest.rb#L960

def self.run_one_method klass, method_name # :nodoc:
 result = klass.new(method_name).run
 raise "#{klass}#run _must_ return a Result" unless Result === result
 result
end

This line creates a new instance of Minitest::Test class and passes the current method name as the first argument. Minitest::Test inherits initialize from Minitest::Runnable which looks like this:

lib/minitest.rb#L400-L404

def initialize name # :nodoc:
 self.name = name
 self.failures = []
 self.assertions = 0
end

So every test method we’ve created in our original test file gets its own Minitest::Test instance, which stores the name, failures, and number of assertions. A much better approach, compared to our hack from the beginning, but there are some similarities.

The final layer

We still have one layer before we reach the end since we’re calling run on an instance of Minitest::Test.

Here is that final layer in its entirety:

lib/minitest/test.rb#L89-L110

##
# Runs a single test with setup/teardown hooks.
def run
 with_info_handler do
 time_it do
 capture_exceptions do
 before_setup; setup; after_setup
 self.send self.name
 end
 TEARDOWN_METHODS.each do |hook|
 capture_exceptions do
 self.send hook
 end
 end
 end
 end
 Result.from self # per contract
end

Let’s highlight the most important line in that method.

lib/minitest/test.rb#L98

##
# Runs a single test with setup/teardown hooks.
def run
 with_info_handler do
 time_it do
 capture_exceptions do
 before_setup; setup; after_setup
 self.send self.name
 end
 TEARDOWN_METHODS.each do |hook|
 capture_exceptions do
 self.send hook
 end
 end
 end
 end
 Result.from self # per contract
end

This calls our original test method named test_two_plus_two. The name was stored in Minitest::Runnable#initialize, as explained earlier.

Other parts of this method are pretty self-explanatory, and I will not go into details. The goal of this article is achieved since we have tracked down and found out exactly what happens when we run our Minitest tests.

Hooray!

Notes and examples

The commit 1f2b132 is no longer the latest commit in the repository. The first draft of this article was written 8 weeks ago, and I apologize for not publishing it sooner.
This means it will be skipped for any exceptions except exit 0 which raises SystemExit with success? set to true. You can test this with the following code:
```
 require 'minitest/autorun'
 exit 0
```
Which will result in the standard Minitest report being printed out. Try replacing the exit status like this:
```
 require 'minitest/autorun'
 exit 1
```
And notice that the Minitest report was not printed out. The same thing happens if you raise an exception.

The simplest example to confirm this behavior is:

 exit_code = nil
 at_exit do
 puts 'inside at_exit'
 exit exit_code
 end
 exit_code = raise

Compare that with:

 exit_code = raise
 at_exit do
 puts 'inside at_exit'
 exit exit_code
 end

⁂

Thank you for reading this far.
For more writing like this subscribe to the RSS feed.
You can also always send me an email.
Have a wonderful day.