Running test script scenarios in python

Question

I'm developing a script to test another program. Basically I replicated a class with all the functions that the program I am testing have. To make things simple, assume I have only 2 functions: set(value) and add(value). set(value) sets an accumulator value and add(value) adds value to it; and I'm testing a program that should do the same thing.

I also have a check() function that verifies that the accumulator value in which set(value) and add(value) operates have the same value as the value retrieved by the program I'm testing; so, after EACH operation, I want to run this check() function and make sure they match.

To try to make this clear, here's a (non-tested) example:

def set(self, value):
 self.acc = value
 sendCommandToMyProgram('set', value)
def add(self, value):
 self.acc += value
 sendCommandToMyProgram('add', value)
def check(self):
 return self.acc == sendCommandToMyProgram('check', value)

and I want to do "test scenarios" like:

set(1)
check()
add(2)
check()
add(-2)
check()
add(3)

and many others

What would be the best way to make this "test scenario" script? It would be nice if it could work with functions that takes different parameters (in this example, both set() and add() received just one parameter - value -, but this isn't always the case).

thanks

Answer 1

Assuming that you're not wanting to check the value all the time during normal operation, the thing that pops to mind first would be the built-in unit test module.

If you are needing to do more complex things that you cannot cleanly isolate the test as shown below, check out a some of my issues I had to work around. It's worth noting that I ended up rolling my own test runner that behaves very similar to unit-test, but with greater control. I'll be pushing it up to py-pi in the next month or so.

Working Unit Test Examples

import unittest
class RealAccumlator(object):
 def __init__(self):
 self.acc = 0
 def add(self, val):
 self.acc += val
 def set(self, val):
 self.acc = val
 def get(self):
 return self.acc
# Using a global variable as I dont know how "sendCommandToMyProgram"
# is used
real_acc = RealAccumlator()
# Test Script
class MyAccumlator(object):
 def __init__(self):
 self.acc = 0
 def add(self, val):
 self.acc += val
 real_acc.add(val)
 def set(self, val):
 self.acc = val
 real_acc.set(val)
 def check(self):
 return self.acc == real_acc.get()
class MockedAccumlatorTests(unittest.TestCase):
 def setUp(self):
 self.acc = MyAccumlator()
 def test_SetFunction(self):
 test_values = range(-10,10)
 # Test the set commands
 for val in test_values:
 self.acc.set(val)
 self.assertTrue(self.acc.check())
 def test_AddFunction(self):
 test_values = range(-10,10)
 # Set the acc to a known value and to a quick test
 self.acc.set(0) 
 self.assertTrue(self.acc.check())
 # Test the add commands
 for val in test_values:
 self.acc.add(val)
 self.assertTrue(self.acc.check())
class RealAccumlatorTests(unittest.TestCase):
 def test_SetFunction(self):
 test_values = range(-10,10)
 # Test the set commands
 for val in test_values:
 real_acc.set(val)
 self.assertEqual(val, real_acc.get())
 def test_AddFunction(self):
 test_values = range(-10,10)
 # Set the acc to a known value and to a quick test
 real_acc.set(0) 
 self.assertEqual(0, real_acc.get())
 # Test the add commands
 expected_value = 0
 for val in test_values:
 expected_value += val
 real_acc.add(val)
 self.assertEqual(expected_value, real_acc.get())
if __name__ == '__main__':
 unittest.main()

Update based on accepted answer

If this is only going to be a test script and your mock accumulator isn't used out side this tests consider the following mod. I still believe writing the unittests will serve you better in the long run

class MyAccumlator(object):
 def __init__(self):
 self.acc = 0
 def add(self, val):
 self.acc += val
 real_acc.add(val)
 assert(self.check())
 return self.check()
 def set(self, val):
 self.acc = val
 real_acc.set(val)
 assert(self.check())
 return self.check()
 def check(self):
 return self.acc == real_acc.get()

This would allow you to iterate over whatever lists you want without having to think about calling the check function. There are two methods provided that you could check.

1. Leaving the assert call would throw an exception if they ever didn't match (only suggesting under the assumption this will live as a test script).

2. Removing the assert and checking the return status from your calling script (rather than an explicit call to check() might clean things up as well.

Answer 2

I needed to run the "check()" after each operation.

I ended up creating "test scenarios" like:

test_scenarios =
 [
 [
 (obj.set, {1}),
 (obj.add, {5}),
 (obj.add, {-2})
 ],
 [
 (obj.set, {-2}),
 (obj.add, {3}),
 (obj.add, {-5})
 (obj.add, {1})
 ]
 ]

then I iterate over the test scenarios basically calling test[0](**test[1]) and then check()

maybe not the best solution, but the best I could think of

• kirbyfan64sos: Try using a list of numbers to add instead, as I stated in my answer.
• Adam Lewis: Check out my edit, it might make your implementation easier to manage.

Answer 3

Set the value. Create a list with the values to add. Then, loop through that list, running check and add with each loop. i.e.: numlist = [1, 2, -2, 3]; for num in numlist: obj.check(); obj.add(num)

• Lem0n: This wouldn't work very well because "It would be nice if it could work with functions that takes different parameters (in this example, both set() and add() received just one parameter - value -, but this isn't always the case)."
• kirbyfan64sos: @Lem0n: Reflection is an option. You could create a multi-dimensional list, each item containing the arguments. The second dimension is the arguments. Then, you can use reflection to call the correct parameters. i.e. mylist = [[firstarg, secondarg], [firstarg, secondarg]]. Then you would get the number of args in each one and use reflection to correctly call the functions.