This one is from the ‘Practice what you Preach’ category.

One of the things I have done at my current employer is write a full test application and associated test scripts in python. When reporting the results to the appropriate persons recently, I had a couple that were not passing, but I could explain away. To make a moderate length story short, the development manager sent me a note to confirm that those failures were due to ‘flakiness in the test framework’ which caused me one of those lightbulb moments. How can we (I), as (a) QA professional push our developers to create unit tests to robustify their codebase we (I) do not also. So, as of this week, I’m unit testing the testing framework. (‘Who watches the watchers’ / ‘Who tests the tests’).

This led me to the problem of how do I structure, and more importantly, run all my unit tests; below is what I came up with.

Structure – I talked briefly about the structure of my scripting framework in my post about Sync ‘n Run, so you can have a looksee at it to familiarise yourself. Essentially, the guts of what the tests actually do are in modules stored in $FRAMEWORK/lib ($FRAMEWORK in this case refers to wherever it is on disk as it is not tied to a particular directory). I decided to fully segregate the testing code from production code, so any unit tests are stored in their own module prefixed with test_. This leads to the unit tests for the ldap_manipulation module to be stored in test_ldap_manipulation.

Unit Test Module – Once I had where things were going to be, I needed to figure out what the unit tests were going to look so I could run everyone with a single command. This is likely best illustrated with a sample (which was the basis for an actual bunch of unit tests, but I’ve obscured it to remove all product specific stuff).


import unittest
import dummy
import xml.dom.minidom

# this is mandatory for all unit test files every test suite (for which there is one per class) is appended here
suites = []

class custom_props(unittest.TestCase):
    def setUp(self):
        self.defaults = {"foo": "bar=true"}

    def test_missing_custom_props(self):
        """custom_props is not a mandatory item, test_dict should have default values"""
        test_xml = "foo"
        test_dict = {}
        dummy.handle_custom_properties(test_xml, test_dict, self.defaults)
        for default in self.defaults.keys():
            assert_(default in test_dict.keys(), "Missing default tag %s" % default)
            assertEqual(defaults[default], test_dict["url_tag"], "default tag %s has wrong default value" % default)

    def test_has_custom_props_with(self):
        pass

    def test_has_custom_props_without(self):
        pass

# create a test suite based upon all methods of custom_props that start with "test"
suite_custom_props = unittest.makeSuite(custom_props, "test")

# add it to the master list of suites
suites.append(suite_custom_props)

# here we create a module wide suite and call it 'all_suites'; the name is important and must be the same for each module
all_suites = unittest.TestSuite()
for suite in suites:
    all_suites.addTest(suite)

Execution – Now you will see why ‘all_suites’ is important. This is my controller script which will collect all the suites that are stored in the various modules ‘ all_suites, make one massive module then run it. Note that this script resides in $FRAMEWORK/bin, thus the bit of path trickiness for os.walk().


import os, os.path, sys
import unittest

# master suite
suite = unittest.TestSuite()

# dirs that contain (potentially) interesting stuff
interestings = ["lib", "testscripts"]
for interesting in interestings:
    for root, dirs, files in os.walk(os.path.join(os.path.split(os.getcwd())[0], interesting)):
        for f in files:
            if f.startswith("test_") and f.endswith(".py"):
                test_module = os.path.split(os.path.join(root, f))
                if test_module[0] not in sys.path:
                    sys.path.append(test_module[0])
                buncha = __import__(test_module[1][:-3])
                msuite.addTest(buncha.all_suites)
                del(sys.modules[test_module[1][:-3]])

runner = unittest.TextTestRunner()
runner.run(msuite)

Happy unit testing.