Python monkeypatching best practices

Question

I am testing an application that has several external dependencies and I have used monkeypatching techniques to patch the functions of external libraries with a custom implementation to help my tests. It works as expected.

But the problem I currently have is that this makes my test file really messy. I have several tests and each test requires its own implementation of the patched function.

For instance, let us say I have a GET function from an external library, my test_a() needs GET() to be patched so that it returns False and test_b() needs GET() to be patched so that it returns True.

What is the preferred way to handle such a scenario. Currently I do the following:

def test_a(monkeypatch):
 my_patcher(monkeypatch, patch_get_to_return_true = True, patch_get_to_return_false = False, patch_get_to_raise_exception = False)
def test_b(monkeypatch)
 my_patcher(monkeypatch, patch_get_to_return_true = True, patch_get_to_return_false = False, patch_get_to_raise_exception = False)
def test_c(monkeypatch)
 my_patcher(monkeypatch, patch_get_to_return_true = False, patch_get_to_return_false = False, patch_get_to_raise_exception = True)
def my_patcher(monkeypatch, patch_get_to_return_true = False, patch_get_to_return_false = False, patch_get_to_raise_exception = False):
 def patch_func_pos():
 return True
 patch_func_neg():
 return False
 patch_func_exception():
 raise my_exception
 if patch_get_to_return_true:
 monkeypatch.setattr(ExternalLib, 'GET', patch_func_pos)
 if patch_get_to_return_false:
 monkeypatch.setattr(ExternalLib, 'GET', patch_func_neg)
 if patch_get_to_raise_exception:
 monkeypatch.setattr(ExternalLib, 'GET', patch_func_exception)

The above sample has just three tests that patch one function. My actual test file has around 20 tests and each test will further patch several functions.

Can someone suggest me a better way of handling this? Is it recommended to move monkeypatching part to a separate file?

Answer 1

Without knowing further details, I would suggest splitting my_patcher into several small fixtures:

@pytest.fixture
def mocked_GET_pos(monkeypatch):
 monkeypatch.setattr(ExternalLib, 'GET', lambda: True)
@pytest.fixture
def mocked_GET_neg(monkeypatch):
 monkeypatch.setattr(ExternalLib, 'GET', lambda: False)
@pytest.fixture
def mocked_GET_raises(monkeypatch):
 def raise_():
 raise Exception()
 monkeypatch.setattr(ExternalLib, 'GET', raise_)

Now use pytest.mark.usefixtures to autoapply the fixture in test:

@pytest.mark.usefixtures('mocked_GET_pos')
def test_GET_pos():
 assert ExternalLib.GET()
@pytest.mark.usefixtures('mocked_GET_neg')
def test_GET_neg():
 assert not ExternalLib.GET()
@pytest.mark.usefixtures('mocked_GET_raises')
def test_GET_raises():
 with pytest.raises(Exception):
 ExternalLib.GET()

However, there is room for improvements, depending on the actual context. For example, when the tests logic is the same and the sole thing that varies is some test precondition (like different patching of GET in your case), tests or fixtures parametrization often saves a lot of code duplication. Imagine you have an own function that calls GET internally:

# my_lib.py
def inform():
 try:
 result = ExternalLib.GET()
 except Exception:
 return 'error'
 if result:
 return 'success'
 else:
 return 'failure'

and you want to test whether it returns a valid result no matter what GET behaves:

# test_my_lib.py
def test_inform():
 assert inform() in ['success', 'failure', 'error']

Using the above approach, you would need to copy test_inform three times, the only difference between the copies being a different fixture used. This can be avoided by writing a parametrized fixture that will accept multiple patch possibilities for GET:

@pytest.fixture(params=[lambda: True,
 lambda: False,
 raise_],
 ids=['pos', 'neg', 'exception'])
def mocked_GET(request):
 monkeypatch.setattr(ExternalLib, 'GET', request.param)

Now when applying mocked_GET to test_inform:

@pytest.mark.usefixtures('mocked_GET')
def test_inform():
 assert inform() in ['success', 'failure', 'error']

you get three tests out of one: test_inform will run three times, once with each mock passed to mocked_GET parameters.

test_inform[pos]
test_inform[neg]
test_inform[exception]

Tests can be parametrized too (via pytest.mark.parametrize), and when applied correctly, parametrization technique saves a lot of boilerplate code.

• sjaymj62: Thanks for your detailed answer. I knew about monkeypatching, fixtures and parameterized tests. But I never thought about combining all these concepts to avoid code duplication. Your answer really helped me. Too bad I can only give you one vote.
• hoefling: Glad I could help! Reading the official docs helped me a lot when I first started working with pytest, lots of great examples there.