<pre lang="python">def pytest_runtest_makereport(__multicall__, item, call):
    if call.when == "call":
        try:
            assert([] == item._testcase.verificationErrors)
        except AssertionError:
            call.excinfo = py.code.ExceptionInfo()
    rep = __multicall__.execute()
    return rep

This ‘clever’ function took me almost two days to figure out how to do. And meant that I had to learn waaay too much about how py.test behaves behind the scenes so this post is partly to share those findings with others and partly [largely] for my own reference later.

First, background.

I first came across the notion or hard and soft asserts exporting code from Se-IDE. A hard assert is your normal one where if the condition fails the script stops immediately whereas a soft assert will collect failures and ultimately fail the script but will proceed as if nothing happened.

Normally this soft assert pattern is implemented like this where you create a list in the setup, catch any hard assert fails and append them to the list, then check that the list is empty in the teardown.

<pre lang="python">class SomeTest():
  def setUp(self):
    self.verification_errors = []
    
  def testSomething(self):
    try:
      assert(False)
    except AssertionError:
      self.verification_errors.append("Some soft assert failed")
      
  def tearDown(self):
    assert([] == self.verification_errors)

py.test’s workflow is a bit, erm, unique though so that doesn’t really work.

py.test treats each of these three methods as completely separate and atomic calls. Each call is implemented as a hook that can be overridden (pytest_runtest_setup, pytest_runtest_call, pytest_runtest_teardown). Due to implementation, the executing code knows only about its locally scoped information and not some of the meta information you might have access to in a pure UnitTest2 script — like current script status. (Likely explained wrong, but that’s what it ‘feels’ like.) That useful bit of information is calculated in the reporting step (pytest_runtest_makereport) which is called after each of the other three steps which means that we can’t make the pass/fail determination in teardown.

But we also cannot make the determination after teardown is complete, because it then looks like the failure is happening in the teardown — which is wasn’t; the ‘call’ step was what failed. Thus the ‘if call.when == “call”‘ condition. Once in there, we can check if any of our soft asserts failed (the list isn’t empty) and then add it to the ‘item’ (everything in py.test is an item) as if it had happened during the script execution.

(The __multicall__ argument is a bit of deep, dark py.test black magic I borrowed from their mailing list archives.)

If you drop this function in your conftest.py you will be able to make use of the soft assert pattern with py.test and have it report correctly. Just remember not to put the check in the teardown as well…

<pre lang="python">class SomeTest():
  def setUp(self):
    self.verification_errors = []
    
  def testSomething(self):
    try:
      assert(False)
    except AssertionError:
      self.verification_errors.append("Some soft assert failed")