Pytest reports missing lines in coverage for imports and constant assignments

Question

I'm baffled by differences in coverage behavior for different files in my project and hope someone will please suggest how to debug this more effectively than I am doing now. This is basically the same question as Pytest incorrectly reports missing lines in coverage

I'm using Python 3.11, tox 4.11.3, coverage 7.4.4. Working on a mbp laptop if that matters any. My python code files all look like this, this one in my package directory myproj/ is called coverme.py:

import logging
logger = logging.getLogger(__name__)
some_obj = SomeObject(foo='bar')
CONST = 123
def myfunc() -> bool:
 return True
def otherfunc() -> bool:
 """
 nice pydoc goes here
 """
 return CONST

My test files all look like this, this one in the project's tests/ directory is called test_coverme.py:

import logging
from myproj import coverme
logger = logging.getLogger(__name__)
def test_myfunc():
 logger.debug('test myfunc')
 assert coverme.myfunc()

Here's my tox.ini file:

[tox]
envlist = code,flake8
minversion = 2.0
[pytest]
testpaths = tests
[testenv:code]
basepython = python3
deps=
 coverage
 pytest
 pytest-cov
 pytest-mock
commands =
 pytest --cov myproj --cov-report term-missing --cov-fail-under=70 {posargs}
[testenv:flake8]
basepython = python3
skip_install = true
deps = flake8
commands = flake8 setup.py myproj tests

I can run the test via tox -- tests/test_coverme.py and see these results; I removed results for those files I cannot post here:

---------- coverage: platform darwin, python 3.11.7-final-0 ----------
Name Stmts Miss Cover Missing
-------------------------------------------------------------
myproj/coverme.py 7 1 86% 16

The simple coverme example shown above works perfectly, pytest-cov reports all lines covered except for line 16; I agree that that otherfunc is never called. Same goes for most code files in my project, the reported coverage is as I expect.

Yet for some code files which of course I cannot post here, the coverage report lists uncovered (missed) lines that are import statements and constant definitions! For example, pytest claims lines like CONST = 123 (that appears in the example I posted above) are not covered by any test. It seems impossible that a constant assignment line in a file imported by a test, was not executed.

I see in the coverage FAQ advice to invoke coverage from the command line, which I believe tox is doing for me. I ran coverage erase to clean old data, that made no difference. I ran tox -r to reinstall all dependencies, also no difference. I know that I can run single test files or single tests with tox (not the full suite), I have not found that to make a difference either.

Please suggest other ways to figure out what I am doing wrong, or just possibly pytest is not quite doing correctly, thanks in advance.

• larsks: "Yet for some code files which of course I cannot post here..." Please update this question to include a minimal reproducible example -- if you can't post your actual code, then post example code that reproduces the problem.
• chrisinmtown: @larsks I have been pulling my hair out trying to do exactly that, I know how impossible this looks at first glance. I don't see any structural difference of any kind between code files where all the imports and constant-assignment statements are covered, and files where half the imports and constant-assignment statements are missed. I am looking for suggestions of how to run tox, coverage etc. that will yield more information.
• chrisinmtown: Grasping at straws here, I duplicated one of the problem code files that has missed assignment lines, then duplicated & updated the test file to call that renamed (but otherwise identical) file. Ran tox again and Pytest reports appropriate coverage for the copied file, no missed variable-assignment lines. So I see different behavior on byte-for-byte identical files. This ignores whether those files are imported in other modules.
• chrisinmtown: I don't yet have an answer here, but the --trace option for pytest is proving very helpful. I modified my tox.ini file so the command line starts pytest --trace --cov .. and that lets me see some warnings that might hold the key.

Answer 1

The lack of similar questions/answers made me pretty certain that I was doing something wrong, or at least a little unusual. I noticed that the first line missed was always the line following this line:

api_client = ApiClient(client_config)

A lot of tracing in Pdb confirmed this was the offending line of code. That class ApiClient was generated by swagger-codegen from an OpenAPI specification YAML file. The ApiClient class creates a regular Python3 thread pool within its __init__ method like this:

self.pool = ThreadPool()

The library Pool class's __init__ method calls the _repopulate_pool method which creates a process and calls .start() on the process. At that point the Python debugger stops tracing and just continues execution. I suspect (but don't know how to prove) that something similar happens when the coverage tool is watching the execution, which is why it fails to keep track of the executed lines.

My solution is a workaround. Instead of initializing a variable with the API client at the top of the module, I added a getter function that creates an instance of ApiClient lazily:

_api_client = None
def _get_api_client() -> ApiClient:
 """
 Initialize an API object lazily to simplify tox testing.
 :return: ApiClient
 """
 global _api_client
 if _api_client is None:
 client_config = Configuration()
 _api_client = ApiClient(client_config)
 return _api_client

For tox tests, I use mocker.patch to return a fake value from this getter function, fake bcos my test suite does not simulate the remote REST services accessed by these generated clients.

With all that in place, I again see coverage results that match my expectation, and all the import and assignment statements at the top of modules are no longer missed.

Maybe this will help someone out there who is using Swagger-generated code.