Bugs in unit test frameworks – oh the irony
This last week I’ve been working on my metaframework for selenium (more on that in a different post) which has been pretty fun. It’s been too long since I had to exercise my brain at work. Unfortunately, it has also been pretty frustrating at times. The source of the frustration? Why, 2 bugs in Python’s unittest module — which is included as part of the standard library so in theory should be pretty well tested.
Bug 1 – You cannot load tests from a module (using unittest.loadTestsFromModule) that is not directly inherited from unittest.TestClass. In other words, even though unittest is all nicely divided up into classes, you cannot leverage inheritance to organize your test code. Here is an example:
test_module_A.py
import unittest
class ParentTestClass(unittest.TestCase):
pass
test_module_B.py
import test_module_A
class ChildTestClass(test_module_A.ParentTestClasss):
def test_Something(self):
pass
With the way things are currently, unittest.loadTestsFromModule(test_module_B) won’t find any tests. This is because it is checking the classes in the module to see if they are a subclass of ‘TestCase’. Due to some weird scoping rules, this doesn’t work. It should be unittest.TestCase instead.
Crazy enough, in Jython, this has been fixed in one spot, but not another in unittest. CPython is still affected by this in both places. Here are patches for Jython and CPython (2.5).
Bug 2 could be argued either way whether or not it is a bug but I think it is. In Python there are old-style classes, and new-style ones. New-style classes have been around that in a temporal sense they are no longer new; only in comparison to what was before. All modules in the standard library should be (in my opinon) new-style. Surprisingly, the classes inside unittest are not. This means that you cannot use built-in methods like super() to reach into a class’ superclass.
To me, this is the perfect example why open source apps get a bad rep in corporate environments. At work, when I find a bug I look around to see if there are any other ones in the vicinity. Similarly, when developers are fixing it they look to see if it crops up elsewhere too. And when I verify their fix, I try a couple other suspect places as well. In opensource, the itch gets scratched; the problem is that there might be another itch hiding around the corner.
Update: My CPython patch was rejected, by Guido. Seems I need to go back and re-examine how namespaces work. Oh well. My Jython patch makes things work the way I want them to though, which is the only thing that matters. 🙂