Python coverage and external executables

Question

I have a project that has almost complete unit test coverage. There's only one statement that isn't covered and I can't work out a good way to do so. Say the project is called foo, I have foo/commands.py:

#!/usr/bin/python
[...]
def main(argv):
 [...]
if __name__ == '__main__':
 return main(sys.argv) # This line allegedly not covered by tests

I do have a test for it! foo/tests/test_commands.py:

import unittest
class CommandsTest(unittest.TestCase):
 def test_direct_call(self):
 proc = subprocess.Popen(['python', 'foo/commands.py', 'help'], stdout=PIPE)
 stdout, stderr = proc.communicate()
 self.assertTrue(stdout.startswith('usage:'))
 self.assertEquals(0, proc.returncode)

(That's not my actual test, but it's along those lines.)

I use nose as my test runner:

nosetests --cover-erase --cover-tests --with-coverage --cover-package=foo -vs foo

The problem is that this particular statement isn't considered "covered", because I use plain python instead of coverage.py (or python-coverage as its called on Ubuntu). I could change it to python-coverage, but I don't really want to do that:

• It may not be available.
• If it is available, it may be named differently.
• It's ok if someone wants to run the test suite without the coverage stuff.

Right now, I have a dreadful hack to do it:

interpreter = 'coverage' in sys.modules ['python-coverage', 'run'] or ['python']

Not exactly the most readable code, but imagine the irony of using the more readable approach:

if 'coverage' in sys.modules:
 interpreter = ['python-coverage', 'run']
else:
 interpreter = ['python']

...only to realise that the "else:" branch ends up not getting called when coverage is enabled, so I just have another line of code that's not exercised by unit tests.

My goal is to fail the test suite if there isn't 100% coverage. I "need" the statement in foo/commands.py because it's a very convenient way to test things during development (in actual installs, it'll use distutils console_scripts magic to call into this file).

Ideas? Prior art? Anything?

Note: 100% coverage is by no means a guarantee that everything is great. Less than 100% is a guarantee, though, that there's something you're not testing. I can easily verify the 100% coverage, so I see no reason not to.

• testerab: Why is your goal to fail the test suite if there isn't 100% coverage? What benefits will 100% coverage give you over 95% coverage, but more thorough testing around more critical areas?
• Soren: 100% coverage is by no means a guarantee that everything is great. Less than 100% is a guarantee, though, that there's something you're not testing. I can easily verify the 100% coverage, so I see no reason not to.

Answer 1

The simplest option is to just mark the line as ignored by your coverage tests. You know more than coverage.py does, you can just excuse the line from the measurements:

if __name__ == '__main__': # pragma: no cover
 return main(sys.argv) 

You can also use some tricks with coverage.py to get it to measure code in launched subprocesses. This sounds like the thing you are really looking for.

Answer 2

Remove the uncovered line from your .py file and use

python -c "import foo; foo.main(args);"

You can wrap that in an alias or a bash function.

Answer 3

Solution 1

Add the following code to your test runner:

import coverage
coverage.process_startup()

For a complete picture, see this sample runtests.py file (which is the entry point for running the tests).

#!/usr/bin/env python
import os
import sys
import pytest
import coverage
def main():
 coverage.process_startup()
 sys.path.insert(0, os.path.abspath('src'))
 return pytest.main()
if __name__ == '__main__':
 sys.exit(main())

Solution 2

In the root of your project add a single file named .pth with the following content:

import coverage; coverage.process_startup()

Solution 3

In the root of your project add a single file named sitecustomize.py with the following content:

import coverage
coverage.process_startup()