Getting started with tcltest
When you install Tcl from the offical sources, included with the distribution is a test harness named tcltest [1]. Tcltest makes it remarkably easy to write automated regression tests. This article briefly describes how to get started with tcltest, and includes links to resources for more information. For additional help, see the tcltest page on the Tcler's wiki [2]
Why tcltest?
If you don't test your software rigorsly, how do you know that it works? It's easy to write code that solves an immediate problem, but in doing so we often write the code with blinders on. "We'll always call this proc with three arguments". "This value will always be an integer". "I'll rewrite this when I have more time". And so on. Then, a new programmer is added to the team who doesn't know about these assumptions; she calls a proc with a floating point number rather than an integer and everything breaks. It happens. A lot.
One solution to this problem is to think about testing up front. Many people believe that tests should be written before a single line of executable code is written. To be sure, that is an excellent way to develop reusable library components. I won't go so far as to say it is the way, but it is certainly a good way. Even if you don't subscribe to the test driven development model, tests can be an extremely important tool for developing quality code.
No matter when you write the tests, though, it can be a full time job. In fact, many people believe it must be a full time job; that is, programmers shouldn't be writing their own tests, that's the job for QA. The reality is, though, programmers should test their own code. In an ideal word a QA tester will come behind and give us better coverage but we are our own first line of defense. Obviously, every bug we catch early on is a bug our customers will never see.
As Tcl programmers we are lucky to have an extremely powerful testing harness that comes free with the product. It makes writing tests so simple there is almost no excuse not to write tests every time we create a new proc.
Do you have tcltest?
Many distributions of Tcl and Tk come with tcltest included. However, it is not guaranteed that you will have it available. You can determine whether you have it available or not by doing the following simple test in an interactive tcl shell:
$ tclsh % package require tcltest 2.2.8
If instead of a version number like "2.2.8" you see "can't find package tcltest", that means you don't have tcltest installed where the interpreter can find it. If you wish to follow along with the examples in this article you need to first download and install tcltest. For help, see my article titled Installing tcltest.
A simple example
Tcltest is remarkably simple to use. As a first example, consider the following code fragment which defines a test for the built-in command 'string toupper':
package require tcltest 2
namespace import tcltest::*
test example-1.1 {an example test} -body {
string toupper "hello world"
} -result "HELLO WORLD"
cleanupTests
This code defines a single test (created by the test command) with the following attributes:
- The test is named example-1.1
- The test has a description of "an example test"
- the test code is {string toupper "hello world"}
- the expected result is the string "HELLO WORLD"
If you copy that code into a file and run it with a tcl interpreter, you should see output similar to the following (assuming you have tcltest 2.x properly installed):
$ tclsh test.tcl test.tcl: Total 1 Passed 1 Skipped 0 Failed 0
As you can see, it is extremely easy to create a simple test which executes an arbitrary block of code and then compares it to an expected result. If the result was not what we expect an error will be printed. For example, if you change the expected output from "HELLO WORLD" to "Hello World" you'll get the following output:
$ tclsh test.tcl ==== example-1.1 an example test FAILED ==== Contents of test case: string toupper "hello world" ---- Result was: HELLO WORLD ---- Result should have been (exact matching): Hello World ==== example-1.1 FAILED test.tcl: Total 1 Passed 0 Skipped 0 Failed 1
In this case we not only got a summary that told us one test failed, we got a very useful block of output that describes what failed and why.
A more complex example
As the first example shows, it is quite simple to create basic tests. Pick a name, write a block of code, and define the expected outcome. However, tcltest is considerably more powerful than that first example shows. Here is a somewhat more complex example:
test example-1.2 {a more complex example} -constraints {
unix
} -setup {
set f [open /tmp/testdata w]
puts $f "hello, world"
close $f
} -body {
set f [open /tmp/testdata r]
set data [read $f 5]
close $f
set data
} -cleanup {
file delete -force /tmp/testdata
} -result "hello"
cleanupTests
The above test has the following attributes:
- The test name is "example-1.2"
- The test will only run on unix systems; it will be skipped on mac an windows platforms
- Before the test is run it will execute a block of code to create a test data file
- The test itself will read the test data file and return what it read
- The test will delete the test data file when it is finished
- The test will pass if the data "hello" was read from the file
As you can see, it is possible to have additional code execute both before and after the actual test in order to set up the context for the test. It is also possible to have tests that are unique to certain platforms. With version 2.2.8 of tcltest there are nearly 30 different constraints covering a large range of variances between platforms and environments.
At this point it's worth noting that the above test is a bit contrived. Tcltest includes convenience procedures for creating test data files so it's not necessary to hard-code references to /tmp, for example. For a complete list of commands included in tcltest, consult the man page [1].
In total, a test may be made up of nine sections, not counting the name and description:
- -constraints
- a list of constraints that must be met for the test to run
- -setup
- A block of code to be executed before the test is run
- -body
- The actual test script to run
- -match
- The method of comparing the actual to expected results. The examples in this article have used exact matching but it is possible to also do regexp and glob matching for greater flexibility.
- -result
- The expected result of the test script
- -output
- Data that is expected to be written to stdout when the test runs.
- -errorOutput
- Similar to -output, this defines that that is expected to be written to stderr when the test runs.
- -returnCodes
- The expected return codes of the test script
Style matters
The format of the test command is familiar to anybody who has ever used tk. Most options are preceeded by an option name such as -setup, -body, etc. This is still Tcl code, though, so every command needs to span a single logical line. And, because this is just pure Tcl code, curly braces are not required [3].
Tests are rather verbose since they almost always will include several lines of executable code in addition to all the other metadata. You might be tempted to just continue options on a line until you run out of room, add a backslash, and continue on to the next line. This is certainly possible since Tcl imposes no formatting restrictions on us. However, over time a common style has been adopted by most users of tcltest that removes almost all need for backslashes.
If you'll notice in the examples above, each option begins on a line, followed by a curly brace, then the data for that option is on the following line. The closing curly brace begins the next line, and immediately following is another option. In this way your test scripts will alternate between lines of option names and option values, making the code easy to read and easy to maintain.
Again, this style isn't required, strictly speaking. Neither is the use of curly braces. However, experience has shown that this particular style makes for test files that are easy to read and easy to maintain. Do yourself a favor and stick to this style.
Building test suites
Typically, a single test script will contain many tests. Exactly how you organize the tests is up you. It often makes sense to have a single test file for each functional group you are testing. Within each file you use a test naming convention that emphasizes the logical groupings. For example, you may have a test file named "io.test" in which you name the tests "io-1", "io-2", etc. You might have another test file named "translations.test" that have tests named "translation-1", "translation-2", etc.
The tcltest man page has an entire section describing how to create test suites so I will point you there rather than repeat roughly the same information. Suffice it to say that the developers of tcltest have written it with the intention of creating large hierarchies of tests.
Summary
This article only scratches the surface of what tcltest can do. It is a very mature and robust testing harness that can be used in a variety of situations. Obviously it can be used to test libraries of tcl code, but because of the power of Tcl as a glue language you can use tcltest as a harness for testing external processes as well. You can also use it to test GUI applications written in Tk since Tk has such rich introspection capabilities.
If you don't yet use tcltest or any other testing harness in your development environment, you owe it to yourself and your customers to spend an afternoon learning how to use tcltest. You can literally master it in a single afternoon and it can give you an edge in creating robust, maintainable code.