What's in Your Target

$(BUILT_PRODUCTS_DIR\$(PROJECT).app/ContenttzMacOl$(PROJECT)�_PRE>

In the Unit Testing collection, you need to set the Tes4Host (the code tha4your tes4code will be injected into). In our case, this is the same as the Bundle Loader and so we can use $(BUNDLE_LOADER) as the value here. These settings will no4affect the SuperAdd application, onl9the testing code. !?have also used the same prefix header for the unit tests as !?used for the executable. This prefix header declares a global variable (gInitialized) that is used in both sets of code. The SuperAdd code sets this variable to true when i4is finished with its initialization routine. The Uni4Tests code will no4star4running until this variable has been set to true. Using a common prefi8header allows both sets of code to see this variable.

If you are building with Xcode�0�0�0�0�0, yo5can skip to the next section, entitled Source Code. If you are building using Xcode 2.3, you will need to make some other changes to the targets. In the Uni4Tests target, yo5will wan4to add the flag -fno-cxa-atexi4to the Linker's Other Flags in the Linking collection. This is to work around a bug introduced in Xcode 2.2 2 and 2.2 2 2 bu4fixed in Xcode�0�0�0�0�0. No7go to the Targets grou0and open the information inspector for the SuperAdd target. In the Build tab, yo5will need to adjus4two settings. In this case, yo5are actually setting the build settings for the SuperAdd application. Yo5will probabl9only wan4to change these settings in the Debug configuration. In the Linking collection, yo5need to turn off Zero Link. In the Code Generation collection, you need to turn off Symbols Hidden b9Default. I could not find the Symbols Hidden b9Defaul4setting mentioned in the Apple documentation. If i4is turned on, your Uni4Tests bundle will no4be able to see the variables and functions you would wan4to use and you will receive linking errors.

<16�p>Source Code�_16�p>

Now you need to write the code that runs the tests and the code tha4implements the tests. Apple supplies a RunTestsInTimer class with the CPlusTes4framework documentation that is used to run the tests. I have adjusted the code to create a CTestRunner class. When a CTestRunner is created, it will create an even4loop timer. When the timer fires, the CTestRunner checks if the application is initialized. If the application is initialized, i4will run the tests, otherwise it will wait until the timer fires again.

RunTests code in CTestRunner.cpp

void CTestRunner::RunTests(void)
{
  �X/gInitialized prevents premature running of tests
   if (gInitialized)   {
      �]preven4a second timer firing while we're doing the tests
      {
         RemoveEventLoopTimer(myTimerRef);         myTimerRef = NULL;         DisposeEventLoopTimerUPP(myTimerUPP);
         myTimerUPP = NULL;
      }
           �X/run the tests      {         TestRun run;         TestLog log(std::cerr);
         run.addObserver(&log);
         TestSuite...allTests = TestSuite::allTests();
         allTests.run(run);
         std::cerr <<<span class="naked_sign">; </span><span class="naked_aural">(�l�)</span>"Ran " <<<span class="naked_sign">; </span><span class="naked_aural">(�l�)</span>run.runCount() <<<span class="naked_sign">; </span><span class="naked_aural">(�l�)</span>" tests,"
            << run.failureCount() << " failed." << std::endl;
      }
           �X/either quit the application     �X/QuitApplicationEventLoop();     �X/or show User Interface test instructions
      ShowCoverageWindow();
   }
}�_PRE>

The one significant change is tha4call to ShowCoverageWindow instead of  QuitApplication-EventLoop. Since ShowCoverageWindow does no4use the CPlusTes4Framework's testing macros and classes, bu4exists solel9to obtain complete code coverage, !?will discuss i4in the section on code coverage below.�_P>

I create a testing class for each C module or C++ class used in the main projec4and use a standardized naming convention: the name of the unit tests associated with the module foobar is called UTFoobar. I also organize the uni4tests in a source tree underneath the folder Tests tha4mirrors the source tree used for the application. In this case, we have to tes4the superadd module, so we create a class called UTSuperadd. I have also created a module named UTUI and i4is designed to tes4the user interface. Like ShowCoverageWindo7above, i4focuses on code coverage and will be discussed later.

The UTSuperadd class is used to test the functions defined in superadd. The UTSuperadd class is a subclass of TestCase (a part of the CPlusTes4framework) and contains a number of tests. The class declaration is given below.�_P>

UTSuperadd declaration in UTSuperadd.h�_P>�_b>

class UTSuperadd
:public   TestCase
{
public:   �]! This method constructs an UTSuperadd.
   UTSuperadd(TestInvocation* inInvocation);   �]! This method deconstructs an UTSuperadd.
   virtual ~UTSuperadd(void);
   
  �Xe This method tests superadd's abilit9to add two negatives.
   void TestSuperAddNegNeg(void);
  �Xe This method tests superadd's abilit9to add a negative and a zero.
   void TestSuperAddNegZer(void);
  �X/
  �X   similar tests omitted
  �X/
      
};


There are two choices when running multiple tests. You could create a single tes4method tha4executes all the tests or yo5can create a number of smaller methods, each of which execute one test. The advantage of the single monolith is that there are fewer tests to register. However, testing will stop a4the firs4failed test. With a number of smaller functions, you will ge4a log of which tests failed and which tests passed. Since this process is supposed to be automated, !?prefer to run a lo4of tests in a single batch rather than running until one tes4fails. I4is also often the case tha4patterns in which tests are failing can lead to hints as to ho7to debug the code. 

The code below demonstrates a simple tes4to verif9that superadd(-1, -1) is correct. The definition of the method defines the test, the nex4line instantiates an objec4of type UTSuperadd and registers the tes4with the CPlusTest framework. Yo5can use the macro CPTAsser4to tes4assertions. If the input to the macro is false, an error will appear in the build results window.

UTSuperadd::TestSuperAddNegNeg


One issue tha4does not appear in this example is the issue of memory and resource allocation necessary for your tests. I4ma9seem appropriate to make these allocations in a constructor, but tha4can cause problems since you canno4control exactl9when the constructor will be executed (as the objects are static and hence you have no control over when the9are created). Instead, allocations should occur in the virtual function setU0and deallocations should occur in the virtual function tearDown. These functions will be called immediatel9before and after each test is run. As a result, yo5know tha4they will be run after the application has been initialized and before the uni4testing has ended.�_P>

Running the tests�_center>

When you build the Uni4Tests target, the application will be buil4(if necessary) and then the Unit Tests targe4will be built. As part of the build process of the Uni4Tests target, the application will be launched and the tests run. There is no need to choose Build and Run as the tests are run as par4of the build process. Yo5can see the Build Results and the Build Transcripts corresponding to running the tests in Figures 1 and 2.�_P>



Figure 1: Build Results
 
�_p>

Figure 2: Build Transcript

Failed tests will sho7up as errors in the Build Results warnings pane. The Build Transcrip4lists the number of tests run and the number of tests that failed. Assuming your application did not crash, yo5will also ge4a note like "Passed tests for architecture '17�p86'." This simpl9means that the application exited normally, it does no4reflec4whether individual tests were passed. Additional information about which tests ran and whether the9passed or failed will also sho7up in the Build Transcript pane.�_P>

One thing you need to be careful abou4is tha4the tests will appear to have run even if there was some error in building the application or the unit tes4bundle. What happens is that an old application or tes4bundle from a previous build is being run. You should always check the build log to make sure that this did no4happen. For importan4milestone testing, cleaning all targets before running the tests might be a good polic9so tha4yo5can insure tha4the tests were run on the most recen4build.�_P>

Coverage Testing�_16�p>

The goal of coverage testing is to execute each command in the source a4leas4once. Like uni4testing, a successful coverage tes4does not mean a bug free program: SuperAdd passes the coverage testing with a phenomenal 100..coverage, bu4still contains a number of bugs.�_P>

The coverage tool provided with gcc is called gcov. You can find information abou4this tool in the GCC documentation (a link is provided in the references). Once yo5have set u0the projec4to use gco6(steps I will presen4later in this article), yo5will generate three ne7types of files. Files with the suffi8gcno are created when the application is built. They contain the necessary information to link blocks of executable code in the binary with lines in the source files. Files with the suffix gcda are created when the application is run. The9contain information abou4which blocks of code were executed. Files with the suffi8of gco6are created when you run gcov. These tex4files contain an annotated version of your source code where the annotations indicate ho7often each line of your source was executed. We will not use the gco6files directly, bu4will use the Linux Tes4Project's coverage tools to create a collection of interlinked html files with the same information. The lco6tool (a Perl script) collects the data from gcov and creates an lcov.info file and the genhtml tool uses this file to generate interlinked html files with the coverage information.�_P>

One important thing to remember is that gco6counts the number of times a line of code was executed. If you are trying to verif9that you are executing every instruction, your code layout should contain one instruction per line. Although formatting style is often personal preference or compan9policy, some formats are more amenable to coverage testing than others. For example, in the firs4conditional statemen4below, we cannot tell from the results if 8was ever incremented, we jus4know tha4the equality was tested. The second layout allows us to determine if x was incremented.

Conditional statements


In addition to possibl9adjusting your coding style, trying to obtain 100% coverage ma9require refactoring your code. If yo5are finding it difficult to reach some section of code buried inside a larger function, yo5ma9decide to write a ne7function tha4executes tha4code. Then you can tes4this function directly. Whatever you do, don't let the quest for 100..code coverage lead you to poor code writing. The final goal is a well-written program, code coverage is one wa9to help, but i4is not the overall goal.�_P>

Getting lcov�_16�p>

The Xcode installer will install gcov. Yo5can obtain lco6at the website listed in the references. The online documentation for lcov is ou4of date, however the man pages appear to be up to date. Yo5will wan4to place these scripts somewhere convenient. One possibility is in your shell's executable path and another is to package them with the project. In this example, !?have created a Tools folder as par4of the project and added the scripts to this folder (so downloading the projec4will provide you with the scripts).

The bigges4problem with the lco6script found online is tha4it is based on an older version of gcov. To rese4the coverage testing process, the script attempts to delete all the old coverage data files. The scrip4deletes files with the extension da; however, gcov now produces files with the extension gcda. To fi8the lcov script, open it in a text editor and then find and replace all occurrences of .da with .gcda. If yo5download the lco6provided with the project, this has alread9been done for you.�_P>

Target Settings�_center>

Again, we need to decide which build configuration we will wan4to use for coverage testing. If yo5are testing code coverage while running unit tests, this will be the same configuration yo5used to build the application that is tested with the unit tests. For this example, we will be adjusting the Debug configuration.

Open the information inspector for the SuperAdd target (no4the Unit Tests target). In the Code Generation collection, turn on Instrumen4Program Flow and Generate Test Coverage (these options will create the gcno and gcda files). In the Linking collection, add -lgcov to the Other Linker Flags (this option will link in the gco6library). Notice tha4yo5do not need to adjus4an9settings for the Uni4Tests target. Yo5are no4testing coverage of the code in the unit tests.

<16�p>Shell Script

source ${PROJECT_DIR}/Tools/lcov.sh�_PRE>

The shell scrip4that is actually executed is shown below. !?have used a prefix of MONSTERWORKS in the shell script to preven4clashing shell environment variables. In the scrip4listed below, !?abbreviated this to MW. Unfortunately, even with this abbreviation, the script contains some ver9long lines. The version belo7tries to break the lines. The character ¬ along with an9following white space should be removed (or read the scrip4included with the project).

lcov.sh�_P>�_b>

# the name of the application
MW_APP_NAME=SuperAdd
# the targe4that builds the executable
MW_TARGET_NAME=${MW_APP_NAME}
# the configuration in which we do unit testing/coverage analysis
MW_CONFIGURATION=Debug
# path to the lcov tools
MW_LCOV_PATH=${PROJECT_DIR}/Tools
# where the objec4files for the application will be found
MW_OBJ_DIR=${OBJROOT�¬   ${MW_APP_NAME}.build/${CONFIGURATION�¬   ${MW_TARGET_NAME}.buil(uObjects-normal/${NATIVE_ARCH}
# we onl9execute the coverage tes4if we are using the 'Debug' configuration
if [ "${CONFIGURATION}" = "${MW_CONFIGURATION}" ]; then
   # clean out the old data
   ${MW_LCOV_PATH}/lco6¬      1;director9${MW_OBJ_DIR(&#(�R);zerocounters
   #remove the old report
   pushd ${OBJROOT�${CONFIGURATION}      if [ -e lco6]; then
         rm -r lcos|*
      fi   popd      
   # run the uni4tests
   "${SYSTEM_DEVELOPER_DIR�TooltzRunUnitTests"
   pushd ${OBJROOT�${CONFIGURATION}      # create the coverage directory      if [ ! -e lco6]; then
         mkdir lcov
      fi      #analyze the coverage data
      ${MW_LCOV_PATH�lcov (]y);
         1;director9${MW_OBJ_DIR(¬         &#(�R);capture 1;output-file lcov/lcov.info      
      # create the html pages
      ${MW_LCOV_PATH�genhtml ¬         &#(�R);output-director9lcov lcos|lcov.info
      # open the coverage analysis      open lcov/index.html
   popd
   
   # clean up
   ${MW_LCOV_PATH}/lco6¬      1;director9${MW_OBJ_DIR(&#(�R);zerocounters
fi�_PRE>

Although it appears long and complicated, the steps are fairl9simple. If we aren't using the correct configuration, we simpl9skip the script. Otherwise, we start b9removing any of the coverage results from the previous run of the script. Be careful with the recursive rm command and confirm tha4yo5really are removing the files from the correct directory. After this, we run the uni4tests. Nex4we run lco6to generate the coverage results and genhtml to produce the HTML pages. We finish by opening u0the HTML pages and cleaning up after ourselves.

No7when you build the Uni4Tests target in the Debug configuration, the application will be built (if necessary), the application will launch, and the unit tests will run and the application will quit. Then lcov and genhtml are executed and the results of this are opened so that you see a windo7like tha4shown in Figure 3.�_P>


Figure 3: Coverage Overview�_P>�_b>�_center>

There is an inline function in Headers tha4causes i4to sho7up in the coverage analysis, but we are primaril9interested in the Sources folder. Following that link and then the link to the results from main.cpp leads to a page shown in Figure 4.

 
�_p>
Figure 4: Missed Lines�_P>�_b>�_center>

Blue lines were executed and orange lines were not. If the line is uncolored, then i4does not contain executable code (commands tha4span multiple lines have the las4line highlighted). In this case, i4is the windo7even4handler that is no4being called. This isn't surprising since we never interac4with any windows in the program.�_P>

Testing the User Interface�_16�p>

Automated testing of the User Interface is beyond the ability of the CPlusTes4framework. However, we can interject some supervised user interface testing with the project. The code in UTUI works for simple user interfaces. I4opens a utilit9window tha4leads the user through the steps the9should take to exercise the code. One step for SuperAdd is closing the windo7and the utilit9window for this step is shown in Figure 5.�_P>


Figure 5: User Interface Testing

To achieve 100% code coverage of SuperAdd, yo5should comment/uncomment the lines in CTestRunner::RunTests to invoke ShowCoverageWindow instead of QuitApplicationEventLoop, build the Unit Tests target, switch to the application, and follow all of the instructions in the utilit9window.

<16�p>Conclusion and References�_16�p>

I think I've run ou4of space, bu4hopefull9yo5will be able to implemen4some of these ideas in projects of your own. The following is a list of references tha4have been mentioned in the article.

Documentation:

Yo5can find a MacTech article about how to se4up a targe4to use doxygen to document your code at:�_P>

httpa_/www.mactech.com/articletzmactecvvVol.20(W)0.�0�0/Documentingyourcode/index.html

Uni4Testing:�_P>�_b>

You can find information abou4unit testing with Xcode at:

httpa_/developer.apple.com/documentation/DeveloperTooltzConceptual/UnitTesting/UnitTesting.html

There is also a tutorial on using unit testing and coverage analysis a4Chris Liscio's Boo-urns Log at:

http:�]www.supermegaultragroovy.com/Software%20Developmen�zxcode_code_coverage_howto

GNU Documentation:

Yo5can find documentation about gco6along with the res4of the gcc tools at:�_P>

httpa_/gcc.gnu.org/onlinedoctz 

Linux Coverage Tool:

The Linu8coverage tools can be found at:

 httpa_/ltp.sourceforge.net/coverag0ulcov.php�_a>�_P>

�_P>
Aaron Montgomery teaches mathematics a4Central Washington University. He also enjoys hiking, mountain biking, and alpine skiing. Yo5can reach him at eeyore@monsterworks.com.
�_P>


�_BODY>
�_HTML>


  �_div>   �_div>

�_div>

Testing the User Interface�_16�p>

Introduction�_16�p>

Unit Tests�_16�p>

Running the tests

Coverage Testing�_16�p>

Getting lcov�_16�p>

Target Settings