Script to automate testing solutions

Question

Here's a script I wrote to automate testing for multiple sample files across two solutions, aiming to streamline the process. The script, provided below, compares expected results with the output of the solutions.

#!/usr/bin/bash
test_results() {
 expected=("${@}")
 i=0
 for file in *.txt
 do
 result=`./a.out $file`
 if [ ${expected[$i]} != $result ]; then
 echo "Test case $i: FAILED."
 else 
 echo "Test case $i: PASSED."
 fi 
 i=$((i + 1))
 done
}
cc sol.c
expected_part1=("142" "209" "54605" "142")
echo "Testing part 1."
test_results "${expected_part1[@]}"
cc sol2.c
expected_part2=("142" "281" "55429" "142")
echo -e "\nTesting part 2."
test_results "${expected_part2[@]}"

How can it be written better, and more generically?

Answer 1

two names for two programs

 result=`./a.out $file`

While this does hold a certain traditional charm to it, this is probably not the best name for that program. Prefer sol or sol2, in the interest of clarity. Pass it in as a parameter:

test_results() {
 cmd="1ドル"; shift
 expected=("${@}")
 ...
 result=`"${cmd}" "${file}"`

shellcheck

When authoring in any language, take advantage of the free advice offered by a relevant linter, e.g. $ cc -Wall -pedantic *.c ....

In bash, run $ shellcheck my_script.sh, and heed its advice.

Perhaps the most common warning will be to "${quote}" variable interpolations, in case the variable contains embedded whitespace. I know, I know, it's annoying, and you might believe that in this situation it will never make a difference. Follow the advice anyway, for several reasons:

1. Quoting issues really are a common source of latent bugs that bite folks.
2. It's good to get in the habit of writing code carefully the first time, so the linter never triggers.
3. As with all linters, it's worthwhile to make it run clean so the next (possibly serious) issue it reports will be noticed and addressed prior to a production run.

---

backtick

In modern usage we prefer

$(./a.out "${file}")

over

`./a.out "${file}"`

For one thing, after deep nesting it's easier to pair up the matching punctuation. Shellcheck will offer this advice. Recommend you take it.

(Kudos, nice arithmetic when incrementing i BTW.)

---

TAP

expected_part1=("142" "209" "54605" "142")
...
test_results "${expected_part1[@]}"

Hmmm. That array notation, together with iterating within the function, is a slightly complex approach to a simple problem.

Consider adopting the Test Anything Protocol. It is much like what you are doing, very simple text output to stdout. But it has the advantage of being well documented and having libraries for interacting with it. It tends to be less chatty in the boring "success" case.

Consider putting the "expected" figures into a text file, and then ./a.out ${file} | diff -u expected.txt - suffices. Whether or not the files exactly matched can be read from the $? status variable.

Consider using a structured format for your numeric output, such as .CSV or perhaps JSON (which jq can help with).

---

make

cc sol.c
...
cc sol2.c

Ok, we're back to the "two names" topic.

Recommend you create a short Makefile, so these two become make sol and make sol2.

You could even go further, writing a dependency rule so sol2_output.txt depends on the sol2.c source code, and will be regenerated whenever the source is edited. And then sol2_diffs.txt could be generated as the (hopefully zero byte!) differences between expected and actual output.

---

printf

echo -e "\nTesting part 2."

Yeah, sometimes portable shell code is just painful, as we see here. The -e switch has a long and sordid history. In some environments it's not recognized and we will literally see DASH e SPACE at the beginning of output.

The usual way to avoid such nonsense is to just give up on echo and use /usr/bin/printf, which always works as expected: printf "\nTesting part 2.\n"

---

This codebase achieves its design goals.

I would be willing to delegate or accept maintenance tasks on it.

Answer 2

Separate responsibilities

The script has too many responsibilities:

• Knows how to compile
• Knows how to validate test execution, including the input file paths and expected results
• The above for two distinct programs

I would remedy the above with:

• Leave compilation out of the script. For example you could use Makefile to only recompile the programs when they have changed since the last run of its tests.
• Make the tester script take parameters: the expected result and the command line to run.

With the above changes, the usage of a verify.sh script might look something like this:

./verify.sh 142 ./sol input1.txt
./verify.sh 209 ./sol input2.txt
...

This will be a smaller script and far more reusable. Something like:

expected=1ドル
shift
printf "Executing: '%s' -> " "$*"
actual=$("$@")
if [ "$actual" = "$expected" ]; then
 echo "OK"
else
 echo "FAILED: expected '$expected'; got: '$actual'"
fi

Avoid hardcoding and assumptions

The test_results function runs a counter and loops over *.txt files in the current directory.

It assumes that the files will appear in a certain order to match the expected result values, and that may not always be the case, and will also break if you accidentally create an unrelated text file at some point, say for example a readme.txt or todo.txt. When that happens the counter values will be misleading too, and the output gives no clue about which file was used in the test.

It would be better if the expected values and the input files were both parameters. These should come in pairs because they are tightly related and critical for the correct functioning of the test.

Enclose variables in double-quotes

This is fragile, because it's subject to globbing and word splitting:

if [ ${expected[$i]} != $result ]; then

The fix:

if [ "${expected[$i]}" != "$result" ]; then

Avoid echo -e, and often it's very easy

echo is just fine when used without any flags. And often that's easy enough to do.

Instead of this:

echo -e "\nTesting part 2."

This is equivalent and safer:

echo
echo "Testing part 2."