How to structure tests where one test is another test's setup?

Question

I'm integration testing a system, by using only the public APIs. I have a test that looks something like this:

def testAllTheThings():
 email = create_random_email()
 password = create_random_password()
 ok = account_signup(email, password)
 assert ok
 url = wait_for_confirmation_email()
 assert url
 ok = account_verify(url)
 assert ok
 token = get_auth_token(email, password)
 a = do_A(token)
 assert a
 b = do_B(token, a)
 assert b
 c = do_C(token, b)
 # ...and so on...

Basically, I'm attempting to test the entire "flow" of a single transaction. Each step in the flow depends on the previous step succeeding. Because I'm restricting myself to the external API, I can't just go poking values into the database.

So, either I have one really long test method that does `A; assert; B; assert; C; assert...", or I break it up into separate test methods, where each test method needs the results of the previous test before it can do its thing:

def testAccountSignup():
 # etc.
 return email, password
def testAuthToken():
 email, password = testAccountSignup()
 token = get_auth_token(email, password)
 assert token
 return token
def testA():
 token = testAuthToken()
 a = do_A(token)
 # etc.

I think this smells. Is there a better way to write these tests?

Answer 1

If this test is intended to run frequently, your concerns would rather be focused on how to present test results in a way convenient to those expected to work with these results.

From this perspective, testAllTheThings raises a huge red flag. Imagine someone running this test every hour or even more frequently (against buggy codebase of course, otherwise there would be no point to re-run), and seeing every time all the same FAIL, without a clear indication of what stage failed.

Separate methods look much more appealing, because results of re-runs (assuming steady progress in fixing bugs in code) could look like:

 FAIL FAIL FAIL FAIL
 PASS FAIL FAIL FAIL -- 1st stage fixed
 PASS FAIL FAIL FAIL
 PASS PASS FAIL FAIL -- 2nd stage fixed
 ....
 PASS PASS PASS PASS -- we're done

---

Side note, in one of my past projects, there were so many re-runs of dependent tests that users even began complaining about not willing to see repeated expected failures at later stage "triggered" by a failure at the earlier one. They said this garbage makes it harder to them to analyze test results "we know already that the rest will fail by test design, don't bother us repeating".

As a result, test developers were eventually forced to extend their framework with additional SKIP status and add a feature in test manager code to abort execution of dependent tests and an option to drop SKIPped test results from the report, so that it looked like:

 FAIL -- the rest is skipped
 PASS FAIL -- 1st stage fixed, abort after 2nd test
 PASS FAIL
 PASS PASS FAIL -- 2nd stage fixed, abort after 3rd test
 ....
 PASS PASS PASS PASS -- we're done

• Javier: as i read it, it sounds like it would've been better to actually write a testAllTheThings, but with clear reporting of where it failed.
• gnat: @Javier clear reporting of where it failed sounds nice in theory, but in my practice, whenever tests are executed frequently, those working with these strongly prefer seeing dumb PASS-FAIL tokens

Answer 2

I'd separate the testing code from the setup code. Perhaps:

# Setup
def accountSignup():
 email = create_random_email()
 password = create_random_password()
 ok = account_signup(email, password)
 url = wait_for_confirmation_email()
 verified = account_verify(url)
 return email, password, ok, url, verified
def authToken():
 email, password = accountSignup()[:2]
 token = get_auth_token(email, password)
 return token
def getA():
 token = authToken()
 a = do_A()
 return a
def getB():
 a = getA()
 b = do_B()
 return b
# Testing
def testAccountSignup():
 ok, url, verified = accountSignup()[2:]
 assert ok
 assert url
 assert verified
def testAuthToken():
 token = authToken()
 assert token
def testA():
 a = getA()
 assert a
def testB():
 b = getB()
 assert b

Remember, all random information that is generated must be included in the assertion in case it fails, otherwise your test may not be reproducible. I might even record the random seed used. Also any time a random case does fail, add that specific input as a hard-coded test to prevent regression.

• DougM: +1 for you! Tests are code, and DRY applies as much in testing as it does in production.

Answer 3

Not much better, but you can at least separate setup code from asserting code. Write a separate method that tells the entire story step by step, and take a parameter controlling how many steps it should take. Then each test can say something like simulate 4 or simulate 10 and then assert whatever it tests.

Answer 4

Well, I might not get the Python syntax right here by "air coding", but I guess you get the idea: you can implement a general function like this:

def asserted_call(create_random_email,*args):
 result=create_random_email(*args)
 assert result
 return result

which will allow you to write your tests like this:

 asserted_call(account_signup, email, password)
 url = asserted_call(wait_for_confirmation_email)
 asserted_call(account_verify,url)
 token = asserted_call(get_auth_token,email, password)
 # ...

Of course, it is debatable if the loss in readability of this approach is worth to use it, but it reduces the boilerplate code a little bit.