test.check is a property-based testing library for clojure, inspired by QuickCheck.
This guide, which is based on version 0.10.0, will briefly introduce
property-based testing using test.check examples, and then cover basic
usage of the different parts of the API.
Property-based tests are often contrasted with "example-based tests", which are tests which test a function by enumerating specific inputs and the expected outputs (i.e., "examples"). This guide is written in terms of testing pure functions, but for testing less pure systems you can imagine a function that wraps the test, which uses the arguments to set up the context for the system, runs the system, and then queries the environment to measure the effects, and returns the result of those queries.
Property-based testing, in contrast, describes properties that should be true for all valid inputs. A property-based test consists of a method for generating valid inputs (a "generator"), and a function which takes a generated input and combines it with the function under test to decide whether the property holds for that particular input.
A classic first example of a property is one that tests the sort
function by checking that it’s idempotent. In test.check, this could
be written like this:
(require '[clojure.test.check :as tc])
(require '[clojure.test.check.generators :as gen])
(require '[clojure.test.check.properties :as prop])
(def sort-idempotent-prop
(prop/for-all [v (gen/vector gen/int)]
(= (sort v) (sort (sort v)))))
(tc/quick-check 100 sort-idempotent-prop)
;; => {:result true,
;; => :pass? true,
;; => :num-tests 100,
;; => :time-elapsed-ms 28,
;; => :seed 1528580707376}Here the (gen/vector gen/int) expression is the generator for inputs
to the sort function; it specifies that an input is a vector of
integers. In reality, sort can take any collection of compatibly
Comparable objects; there’s often a tradeoff between the simplicity
of a generator and the completeness with which it describes the actual
input space.
The name v is bound to a particular generated vector of integers,
and the expression in the body of the prop/for-all determines
whether the trial passes or fails.
The tc/quick-check call "runs the property" 100 times, meaning it
generates one hundred vectors of integers and evaluates
(= (sort v) (sort (sort v))) for each of them; it reports success
only if each of those trials passes.
If any of the trials fails, then test.check attempts to "shrink" the input to a minimal failing example, and then reports the original failing example and the shrunk one. For example, this faulty property claims that after sorting a vector of integers, the first element should be less than the last element:
(def prop-sorted-first-less-than-last
(prop/for-all [v (gen/not-empty (gen/vector gen/int))]
(let [s (sort v)]
(< (first s) (last s)))))If we run this property with tc/quick-check, it returns something
like this:
{:num-tests 5,
:seed 1528580863556,
:fail [[-3]],
:failed-after-ms 1,
:result false,
:result-data nil,
:failing-size 4,
:pass? false,
:shrunk
{:total-nodes-visited 5,
:depth 2,
:pass? false,
:result false,
:result-data nil,
:time-shrinking-ms 1,
:smallest [[0]]}}The original failing example [-3] (given at the :fail key) has
been shrunk to [0] (under [:shrunk :smallest]), and a variety of
other data is provided as well.
The different parts of test.check are cleanly separated by namespace. We will proceed from the bottom up, starting with generators, then properties, and then two methods for running tests.
Generators are supported by the clojure.test.check.generators
namespace.
The built-in generators fall into three categories: scalars (basic data types), collections, and combinators.
scalars (basic data types: numbers, strings, etc.)
collections (lists, maps, sets, etc.)
combinators
The combinators are general enough to support creating generators for arbitrary custom types.
Additionally, there are several development functions for experimenting with generators. We’ll introduce those first so we can use them to demonstrate the rest of the generator functionality.
The gen/sample function takes a generator and returns a collection
of small sample elements from that generator:
user=> (gen/sample gen/boolean)
(true false true true true false true true false false)The gen/generate function takes a generator and returns a single
generated element, and additionally allows specifying the size of
the element. size is an abstract parameter, that is generally an
integer ranging from 0 to 200.
user=> (gen/generate gen/large-integer 50)
-165175test.check comes with generators for booleans, numbers, characters, strings, keywords, symbols, and UUIDs. E.g.:
user=> (gen/sample gen/double)
(-0.5 ##Inf -2.0 -2.0 0.5 -3.875 -0.5625 -1.75 5.0 -2.0)
user=> (gen/sample gen/char-alphanumeric)
(\G \w \i 1円 \V \U 8円 \U \t \M)
user=> (gen/sample gen/string-alphanumeric)
("" "" "e" "Fh" "w46H" "z" "Y" "7" "NF4e" "b0")
user=> (gen/sample gen/keyword)
(:. :Lx :x :W :DR :*- :j :g :G :_)
user=> (gen/sample gen/symbol)
(+ kI G uw jw M9E ?23 T3 * .q)
user=> (gen/sample gen/uuid)
(#uuid "c4342745-9f71-42cb-b89e-e99651b9dd5f"
#uuid "819c3d12-b45a-4373-a307-5943cf17d90b"
#uuid "c72b5d34-255f-408f-8d16-4828ed740904"
#uuid "d342d515-b297-4ed4-91cc-8cd55007e2c2"
#uuid "6d09c6f3-12d4-4e5e-9de5-0ed32c9fef20"
#uuid "a572178c-5460-44ee-b992-9d3d26daf8c0"
#uuid "572cc48e-b3a8-40ca-9449-48af08c617d3"
#uuid "5f6ed50b-adef-4e7f-90d0-44511900491e"
#uuid "ddbbfd07-d580-4638-9858-57a469d91727"
#uuid "c32b7788-70de-4bf5-b24f-1e7cb564a37d")The collection generators are generally functions with arguments for generators of their elements.
For example:
user=> (gen/generate (gen/vector gen/boolean) 5)
[false false false false](note that the second argument to gen/generate here is not specifying
the size of the collection, but the abstract size parameter mentioned
earlier; the default value for gen/generate is 30)
There are also generators for heterogeneous collections, the most
important of which is gen/tuple:
user=> (gen/generate (gen/tuple gen/boolean gen/keyword gen/large-integer))
[true :r -85718]Some collection generators can also be customized further:
user=> (gen/generate (gen/vector-distinct (gen/vector gen/boolean 3)
{:min-elements 3 :max-elements 5}))
[[true false false]
[true true false]
[false false true]
[false true true]]The scalar and collection generators can generate a variety of structures, but creating nontrivial custom generators requires using the combinators.
gen/one-ofgen/one-of takes a collection of generators and returns a generator
that can generate values from any of them:
user=> (gen/sample (gen/one-of [gen/boolean gen/double gen/large-integer]))
(-1.0 -1 true false 3 true true -24 -0.4296875 3)There is also gen/frequency, which is similar but allows specifying
a weight for each generator.
gen/such-thatgen/such-that restricts an existing generator to a subset of its
values, using a predicate:
user=> (gen/sample (gen/such-that odd? gen/large-integer))
(3 -1 -1 -1 -3 5 -11 1 -1 -5)However, there’s no magic here: the only way to generate values that
match the predicate is to generate values repeatedly until one happens
to match. This means gen/such-that can randomly fail if the predicate
doesn’t match too many times in a row:
user=> (count (gen/sample (gen/such-that odd? gen/large-integer) 10000))
ExceptionInfo Couldn't satisfy such-that predicate after 10 tries. clojure.core/ex-info (core.clj:4754)This call to gen/sample (asking for 10000 odd numbers) fails because
gen/large-integer returns even numbers about half the time, so
seeing ten even numbers in a row isn’t extraordinarily unlikely.
gen/such-that should be avoided unless the predicate is highly
likely to succeed. In other cases, there is often an alternative way
to build the generator, as we’ll see with gen/fmap.
gen/fmapgen/fmap allows you to modify any generator by supplying a function
to modify the values it generates. You can use this to construct
arbitrary structures or custom objects by generating the pieces they
need and then combining them in the gen/fmap function:
user=> (gen/generate (gen/fmap (fn [[name age]]
{:type :humanoid
:name name
:age age})
(gen/tuple gen/string-ascii
(gen/large-integer* {:min 0}))))
{:type :humanoid, :name ".o]=w2hZ", :age 14}Another use of gen/fmap is to restrict or skew the distribution of
another generator using targeted transformations. For example, to
turn a general integer generator into a generator of odd numbers, you
could either use the gen/fmap function #(+ 1 (* 2 %)) (which also
has the effect of doubling the range of the distribution) or
#(cond-> % (even? %) (+ 1)) (which doesn’t).
Here’s a generator that only generates upper-case strings:
user=> (gen/sample (gen/fmap #(.toUpperCase %) gen/string-ascii))
("" "" "JT" "" ">Y1@" "" "]-" "XCJ@C" "<ANF.\"|" "I@O\"M")gen/bindThe most advanced combinator allows generating things in multiple stages, with the generators in later stages constructed using values generated in earlier stages.
While this may sound complicated, the signature is hardly different
from gen/fmap: the argument order is reversed, and the function
is expected to return a generator instead of a value.
As an example, suppose you want to generate a random list of numbers
in two different orders (e.g., to test a function that should be
agnostic to collection ordering). This is hard to do using gen/fmap
or any other combinator, since generating two collections directly
will generally give you collections with different elements, and if
you just generate one you don’t have the opportunity to use the
generated list with another generator (e.g. gen/shuffle) that might
be able to reorder it.
gen/bind gives us exactly the two-phase structure we need:
user=> (gen/generate (gen/bind (gen/vector gen/large-integer)
(fn [xs]
(gen/fmap (fn [ys] [xs ys])
(gen/shuffle xs)))))
[[-5967 -9114 -2 -4 68583042 223266 540 3 -100]
[223266 -9114 -2 -100 3 540 -5967 -4 68583042]]The structure here is a bit obtuse, as the function we passed
to gen/bind couldn’t simply call (gen/shuffle xs) — if it
had, the whole generator would have simply returned the one
collection generated by (gen/shuffle xs); in order to both
generate a second collection with gen/shuffle and also return
the original collection, we use gen/fmap to combine the two
into a vector.
Here’s another structure that’s a bit simpler at the expense of doing an extra shuffle:
user=> (gen/generate (gen/bind (gen/vector gen/large-integer)
(fn [xs] (gen/vector (gen/shuffle xs) 2))))
[[-4 254202577 -27512 1596863 0 6] [-4 6 254202577 1596863 -27512 0]]However, an option with arguably even better readability is to use
the gen/let macro, which uses a let-like syntax to describe
uses of gen/fmap and gen/bind:
user=> (gen/generate
(gen/let [xs (gen/vector gen/large-integer)
ys (gen/shuffle xs)]
[xs ys]))
[[0 47] [0 47]]A property is an actual test — it combines a generator with a function you want to test, and checks that the function behaves as expected given the generated values.
Properties are created using the clojure.test.check.properties/for-all
macro.
The property in the first example generates a
vector and then calls the function being tested (sort) three times.
Properties can also combine several generators, for example:
(def +-is-commutative
(prop/for-all [a gen/large-integer
b gen/large-integer]
(= (+ a b) (+ b a))))There are two ways to actually run properties, which is what the next two sections are about.
quick-checkThe standalone and functional method of running tests is via the
quick-check function in the clojure.test.check namespace.
It takes a property and a number of trials, and runs the property up to that many times, returning a map describing success or failure.
See the examples above.
defspecdefspec is a macro for writing property-based-tests that are
recognized and run by clojure.test.
The difference from quick-check is partly just syntactic, and
partly that it defines a test instead of running it.
For example, the first quick-check example
in this guide could also be written like this:
(require '[clojure.test.check.clojure-test :refer [defspec]])
(defspec sort-is-idempotent 100
(prop/for-all [v (gen/vector gen/int)]
(= (sort v) (sort (sort v)))))Given this, calling (clojure.test/run-tests) in the same namespace
produces the following output:
Testing my.test.ns
{:result true, :num-tests 100, :seed 1536503193939, :test-var "sort-is-idempotent"}
Ran 1 tests containing 1 assertions.
0 failures, 0 errors.
For additional documentation, see the test.check README.
Original author: Gary Fredericks