Forcing unittest to function
Brandon Rhodes
April 20, 2024
Pre-recorded
Python + DS fwdays'24
Kyiv — Ukraine
Let’s tackle an example problem.
When I check out a small package that
has zero runtime dependencies,
I like to be able to run its tests
with zero testing dependencies.
For a big project, with dependencies:
Heavyweight setup
uv venv
uv pip install -r requirements.txt
3rd party test framework
Zero-dependency project:
Ideally, zero install
Python’s old motto
‘batteries included’
The Python Standard Library does include
a testing framework, named `unittest`
It was originally underpowered.
So third-party testing
frameworks became popular.
But `unittest` started to
quietly improve in Python 3.2.
$ python -m unittest --help
...
-f, --failfast Stop on first failure
...
$ python -m unittest --help
...
-b, --buffer Buffer stdout/stderr
...
Test discovery!
package/__init__.py
# A simple package.
package/tests.py
from unittest import TestCase
class Tests(TestCase):
def test_one(self):
self.assertEqual(1+1, 2)
def test_two(self):
self.assertEqual(2*2, 4)
$ python -m unittest package
------------------------------------------
Ran 0 tests in 0.000s
NO TESTS RAN
`unittest` forced you to list
every module on the command line.
$ python -m unittest -v package.tests
test_one (package.tests.Tests.test_one) ... ok
test_two (package.tests.Tests.test_two) ... ok
------------------------------------------
Ran 2 tests in 0.000s
OK
Test discovery fixes this!
$ python -m unittest discover -v
test_one (package.tests.Tests.test_one) ... ok
test_two (package.tests.Tests.test_two) ... ok
------------------------------------------
Ran 2 tests in 0.000s
OK
Test discovery made
`unittest` nearly useable!
And it’s pretty fast.
`unittest` runs our tiny `test.py` pretty quickly:
$ time python -m unittest discover project
...
0.104 total
Compare that to the third-party `pytest` framework:
$ time pytest project
...
0.808 total
‘all unit tests combined
should run in 1 second’
But `unittest` has a big remaining problem
It makes you write tests as ugly methods,
rather than as beautiful functions.
Which means we have to:
• Import a class we don’t want
• Build a subclass we don’t like
• Lose 4 columns to useless indentation
package/tests.py
from unittest import TestCase
class Tests(TestCase):
def test_one(self):
self.assertEqual(1+1, 2)
def test_two(self):
self.assertEqual(2*2, 4)
Q: Could my own package add this missing
feature to `unittest`? And let me start
using `unittest` again?
A: This is Python! We can do anything!
Q: Can I add test function support
in a way that is simple and elegant
enough that I’m willing to use it?
A: Let’s try!
Q: How does `unittest` find tests?
A: Through Python’s powers
of introspection.
Most Python programmers know how to
access the items in a `list` or `dict`.
But how do we loop over the names in
a namespace — a module or class?
Two builtins:
1. `dir()`
2. `getattr()`
package/tests.py
from unittest import TestCase
class Tests(TestCase):
def test_one(self):
self.assertEqual(1+1, 2)
def test_two(self):
self.assertEqual(2*2, 4)
example.py
from package import tests
names = dir(tests)
print(names)
$ python example.py
['TestCase', 'Tests', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__']
After learning a name from `dir()`,
we can get its object with `getattr()`.
example.py
from package import tests
obj = getattr(tests, 'TestCase')
print(obj)
$ python example.py
<class 'unittest.case.TestCase'>
`dir()` and `getattr()` are an
elegant part of Python’s design.
Some of the older dynamic
languages offered only `eval()`
eval('tests.' + name)
• Slow
• Overpowered
• Dangerous
So, instead of forcing you to call `eval()`,
Python offers a clean, limited, dynamic
operator for each namespace operation.
'a.b' == getattr(a, 'b')
'a.b = x' == setattr(a, 'b', x)
'del a.b' == delattr(a, 'b')
What steps does `unittest` follow
when looking for tests in a module?
‘The `unittest` contract’
1. Collect `TestCase` classes.
2. Call `load_tests()`.
# `unittest/loader.py` to find classes:
for name in dir(module):
obj = getattr(module, name)
if issubclass(obj, case.TestCase): ...
# `unittest/loader.py` to find methods:
dir(testCaseClass)
if attrname.startswith('test'):
...
getattr(testCaseClass, attrname)
Then, things get weird: `unittest`
instantiates your test class n times,
once for each test method that it found.
This is easier to show than describe.
So here’s our test module again:
package/tests.py
from unittest import TestCase
class Tests(TestCase):
def test_one(self):
self.assertEqual(1+1, 2)
def test_two(self):
self.assertEqual(2*2, 4)
And here are exactly the steps
that `unittest` will perform:
example.py
from unittest import TestSuite, TextTestRunner
from package.tests import Tests
cases = [Tests('test_one'), Tests('test_two')]
suite = TestSuite(cases)
TextTestRunner().run(suite)
$ python example.py
..
------------------------------------------
Ran 2 tests in 0.000s
OK
So that’s how `unittest` normally loads
tests: from `TestCase` methods.
But there’s also an escape hatch!
After building the test suite,
`unittest` does one last check:
# From `unittest/loader.py`:
getattr(module, 'load_tests', None)
If `load_tests()` is defined,
then `unittest` calls it.
package/tests.py
from unittest import TestCase
class Tests(TestCase):
def test_one(self): self.assertEqual(1+1, 2)
def test_two(self): self.assertEqual(2*2, 4)
def load_tests(loader, suite, pattern):
# ...add or remove tests from `suite`...
return suite
Here, for example, is how
to add a test to the suite:
package/tests.py
from unittest import TestCase
class Tests(TestCase):
def test_one(self): self.assertEqual(1+1, 2)
def test_two(self): self.assertEqual(2*2, 4)
def division(self): self.assertEqual(10/5, 2)
$ python -m unittest discover -v
test_one (package.tests.Tests.test_one) ... ok
test_two (package.tests.Tests.test_two) ... ok
------------------------------------------
Ran 2 tests in 0.000s
OK
package/tests.py
from unittest import TestCase
class Tests(TestCase):
def test_one(self): self.assertEqual(1+1, 2)
def test_two(self): self.assertEqual(2*2, 4)
def division(self): self.assertEqual(10/5, 2)
def load_tests(loader, suite, pattern):
suite.addTest(Tests('division'))
return suite
$ python -m unittest discover -v
test_one (package.tests.Tests.test_one) ... ok
test_two (package.tests.Tests.test_two) ... ok
division (package.tests.Tests.division) ... ok
------------------------------------------
Ran 3 tests in 0.000s
OK
The Standard Library `doctest` documentation
recommends this exact mechanism for adding
doctests to your package’s test suite.
# Example from `doctest` documentation:
import doctest
import my_module_with_doctests
def load_tests(loader, suite, ignore):
suite.addTests(doctest.DocTestSuite(
my_module_with_doctests,
))
return tests
So, if we want `unittest` to load our test
functions, we have two chances to do it!
‘The `unittest` contract’
1. Collect `TestCase` classes.
2. Call `load_tests()`.
Design Decision!
Which mechanism should we use
for enrolling test functions?
Let’s code both solutions and use
that experience to help us decide!
[ ] `TestCase` class
↗
↘
[ ] `load_tests()`
package/tests.py
def test_one():
assert 1+1 == 2
def test_two():
assert 2*2 == 4
$ python -m unittest discover -v
------------------------------------------
Ran 0 tests in 0.000s
NO TESTS RAN
Q: How should we get ourselves started?
A: Let’s do API-driven development!
Let’s write a call to an API we wish existed
and then see if we can make it exist!
package/tests.py
from .utils import find_tests
def test_one():
assert 1+1 == 2
def test_two():
assert 2*2 == 4
Tests = find_tests()
Next, let’s try implementing this API!
package/utils.py
def find_tests():
...
# What can we put here to build
# a `Tests` class that will convince
# `unittest` to run our functions?
...
An essential technique:
hard-coding!
I’m going to write exactly the code
that’s needed for this one specific case
so we reach a working proof-of-concept
as quickly as possible.
package/utils.py
from unittest import TestCase
def find_tests():
from package import tests
class Tests(TestCase):
def test_one(self):
tests.test_one()
return Tests
$ python -m unittest discover -v
test_one (package.utils.find_tests.<locals>.Tests.test_one) ... ok
------------------------------------------
Ran 1 test in 0.000s
OK
That’s why I love hard-coding!
Now that the code works,
I can iterate quickly towards
a full solution, enjoying working
code at every step.
Let’s iterate!
[ ] How it finds the module.
[ ] How it finds the tests.
[ ] How it adds the tests to the class.
package/utils.py
from unittest import TestCase
def find_tests():
from package import tests
class Tests(TestCase):
def test_one(self):
tests.test_one()
return Tests
package/utils.py
from unittest import TestCase
def find_tests():
from package import tests
class Tests(TestCase): pass
f = tests.test_one
Tests.test_one = f
return Tests
$ python -m unittest discover -v
test_one (package.utils.find_tests.<locals>.Tests.test_one) ... ERROR
==========================================
ERROR: test_one (package.utils.find_tests.<locals>.Tests.test_one)
------------------------------------------
TypeError: test_one() takes 0 positional arguments but 1 was given
------------------------------------------
Ran 1 test in 0.000s
FAILED (errors=1)
What did we do wrong?
Always commit working code to
version control, so you know exactly
what changed when you break something!
Let’s back up to our previous working code:
package/utils.py
from unittest import TestCase
def find_tests():
from package import tests
class Tests(TestCase):
def test_one(self):
tests.test_one()
return Tests
So we need a wrapper.
package/utils.py
from unittest import TestCase
def wrap(function):
def method(self):
function()
return method
def find_tests():
from package import tests
class Tests(TestCase): pass
f = tests.test_one
Tests.test_one = wrap(f)
return Tests
$ python -m unittest discover -v
test_one (package.utils.find_tests.<locals>.Tests.test_one) ... ok
------------------------------------------
Ran 1 test in 0.000s
OK
[ ] How it finds the module.
[ ] How it finds the tests.
[✔] How it adds the tests to the class.
It turns out that `wrap()`
should look very familiar.
It already a builtin!
`staticmethod`
package/utils.py
from unittest import TestCase
# def wrap(function):
# def method(self):
# function()
# return method
def find_tests():
from package import tests
class Tests(TestCase): pass
f = tests.test_one
Tests.test_one = staticmethod(f)
return Tests
$ python -m unittest discover -v
test_one (package.utils.find_tests.<locals>.Tests.test_one) ... ok
------------------------------------------
Ran 1 test in 0.000s
OK
[ ] How it finds the module.
[ ] How it finds the tests.
[✔] How it adds the tests to the class.
package/utils.py
from unittest import TestCase
# def wrap(function):
# def method(self):
# function()
# return method
def find_tests():
from package import tests
class Tests(TestCase): pass
f = tests.test_one
Tests.test_one = staticmethod(f)
return Tests
package/utils.py
from unittest import TestCase
def find_tests():
from package import tests
class Tests(TestCase): pass
for name in dir(tests):
if name.startswith('test'):
f = getattr(tests, name)
setattr(Tests, name, staticmethod(f))
return Tests
$ python -m unittest discover -v
test_one (package.utils.find_tests.<locals>.Tests.test_one) ... ok
test_two (package.utils.find_tests.<locals>.Tests.test_two) ... ok
------------------------------------------
Ran 2 tests in 0.000s
OK
[ ] How it finds the module.
[✔] How it finds the tests.
[✔] How it adds the tests to the class.
Well — how do you find the module?
Q: Where do modules live?
A: They live in the `sys.modules` dictionary.
If we knew the module’s name, we
could retrieve it from `sys.modules`.
package/utils.py
import sys
from unittest import TestCase
def find_tests():
module_name = 'package.tests'
module = sys.modules[module_name]
class Tests(TestCase): pass
for name in dir(module):
if name.startswith('test'):
f = getattr(module, name)
setattr(Tests, name, staticmethod(f))
return Tests
$ python -m unittest discover -v
test_one (package.utils.find_tests.<locals>.Tests.test_one) ... ok
test_two (package.utils.find_tests.<locals>.Tests.test_two) ... ok
------------------------------------------
Ran 2 tests in 0.000s
OK
So now we face a simpler problem:
Q: How do I get the module’s name?
A: Adjust the API so it’s passed in!
package/tests.py
from .utils import find_tests
def test_one():
assert 1+1 == 2
def test_two():
assert 2*2 == 4
Tests = find_tests()
package/tests.py
from .utils import find_tests
def test_one():
assert 1+1 == 2
def test_two():
assert 2*2 == 4
Tests = find_tests(__name__)
package/utils.py
import sys
from unittest import TestCase
def find_tests(module_name):
module = sys.modules[module_name]
class Tests(TestCase): pass
for name in dir(module):
if name.startswith('test'):
f = getattr(module, name)
setattr(Tests, name, staticmethod(f))
return Tests
$ python -m unittest discover -v
test_one (package.utils.find_tests.<locals>.Tests.test_one) ... ok
test_two (package.utils.find_tests.<locals>.Tests.test_two) ... ok
------------------------------------------
Ran 2 tests in 0.000s
OK
[✔] How it finds the module.
[✔] How it finds the tests.
[✔] How it adds the tests to the class.
[✔] `TestCase` class: 10 lines of code + 1 per module
↗
↘
[ ] `load_tests()`
Tests = find_tests(__name__)
Let’s now turn to the
other big alternative.
Once again, API-Driven Design!
package/tests.py
def test_one():
assert 1+1 == 2
def test_two():
assert 2*2 == 4
def load_tests(loader, suite, pattern):
...
return suite
package/tests.py
def test_one():
assert 1+1 == 2
def test_two():
assert 2*2 == 4
def load_tests(loader, suite, pattern):
suite.addTests(...)
return suite
package/tests.py
from .utils import find_tests
def test_one():
assert 1+1 == 2
def test_two():
assert 2*2 == 4
def load_tests(loader, suite, pattern):
suite.addTests(find_tests(...))
return suite
package/tests.py
from .utils import find_tests
def test_one():
assert 1+1 == 2
def test_two():
assert 2*2 == 4
def load_tests(loader, suite, pattern):
suite.addTests(find_tests(__name__))
return suite
Time to write a new `utils.py` module!
Let’s again start by hard-coding the test,
to quickly reach a working solution!
package/utils.py
import sys
from unittest import TestCase
def find_tests(package_name):
module = sys.modules[package_name]
class Tests(TestCase):
def test_one(self):
module.test_one()
return [Tests('test_one')]
$ python -m unittest discover -v
test_one (package.utils.find_tests.<locals>.Tests.test_one) ... ok
------------------------------------------
Ran 1 test in 0.000s
OK
[✔] `TestCase` class: 10 lines of code + 1 per module
↗
↘
[…] `load_tests()`
Now, we enjoy the luxury of
iterating on a working solution.
Let’s make the class fully generic.
package/utils.py
import sys
from unittest import TestCase
def find_tests(package_name):
module = sys.modules[package_name]
class Test(TestCase):
def test_one(self):
f()
f = module.test_one
return [Test('test_one')]
$ python -m unittest discover -v
test_one (package.utils.find_tests.<locals>.Test.test_one) ... ok
------------------------------------------
Ran 1 test in 0.000s
OK
package/utils.py
import sys
from unittest import TestCase
def find_tests(package_name):
module = sys.modules[package_name]
class Test(TestCase):
def test(self):
f()
f = module.test_one
return [Test('test')]
$ python -m unittest discover -v
test (package.utils.find_tests.<locals>.Test.test) ... ok
------------------------------------------
Ran 1 test in 0.000s
OK
package/utils.py
import sys
from unittest import TestCase
def find_tests(package_name):
module = sys.modules[package_name]
class Test(TestCase):
def __init__(self):
super().__init__('test')
def test(self):
f()
f = module.test_one
return [Test()]
$ python -m unittest discover -v
test (package.utils.find_tests.<locals>.Test.test) ... ok
------------------------------------------
Ran 1 test in 0.000s
OK
package/utils.py
import sys
from unittest import TestCase
def find_tests(package_name):
module = sys.modules[package_name]
class Test(TestCase):
def __init__(self, f):
super().__init__('test')
self.f = f
def test(self):
self.f()
return [Test(module.test_one)]
$ python -m unittest discover -v
test (package.utils.find_tests.<locals>.Test.test) ... ok
------------------------------------------
Ran 1 test in 0.000s
OK
package/utils.py
import sys
from unittest import TestCase
class Test(TestCase):
def __init__(self, f):
super().__init__('test')
self.f = f
def test(self):
self.f()
def find_tests(package_name):
module = sys.modules[package_name]
return [Test(module.test_one)]
$ python -m unittest discover -v
test (package.utils.Test.test) ... ok
------------------------------------------
Ran 1 test in 0.000s
OK
Then I noticed something.
This `Tests` class?
It already exists!
package/utils.py
import sys
from unittest import FunctionTestCase
# class Tests(TestCase):
# def __init__(self, f):
# super().__init__('test')
# self.f = f
# def test(self):
# self.f()
def find_tests(package_name):
module = sys.modules[package_name]
return [FunctionTestCase(module.test_one)]
$ python -m unittest discover -v
unittest.case.FunctionTestCase (test_one) ... ok
------------------------------------------
Ran 1 test in 0.000s
OK
package/utils.py
import sys
from unittest import FunctionTestCase
def find_tests(package_name):
module = sys.modules[package_name]
return [FunctionTestCase(module.test_one)]
$ python -m unittest discover -v
unittest.case.FunctionTestCase (test_one) ... ok
------------------------------------------
Ran 1 test in 0.000s
OK
package/utils.py
import sys
from unittest import FunctionTestCase
def find_tests(package_name):
module = sys.modules[package_name]
tests = []
for name in dir(module):
if name.startswith('test'):
f = getattr(module, name)
tests.append(FunctionTestCase(f))
return tests
$ python -m unittest discover -v
unittest.case.FunctionTestCase (test_one) ... ok
unittest.case.FunctionTestCase (test_two) ... ok
------------------------------------------
Ran 2 tests in 0.000s
OK
[✔] `TestCase` class: 11 lines of code + 1 per module
↗
↘
[✔] `load_tests()`: 10 lines of code + 3 per module
def load_tests(loader, suite, pattern):
suite.addTests(find_tests(__name__))
return suite
Verdict? I choose `load_tests()`!
• Explicit rather than implicit.
• Uses a standard `unittest` feature.
• Creates only instances at runtime, not classes.
• Composes well with the `doctest` instructions.
<aside>
I’m sure you have a question:
‘But what about asserts?!’
Because you noticed me quietly pivot
from `assertEqual()` to plain `assert`.
class Tests(TestCase):
def test_one(self):
self.assertEqual(1+1, 2)
# vs
def test_one():
assert 1+1 == 2
How do we get pretty asserts back?
If you don’t have `self`, how
do we call `self.assertEqual()`?
The Prebound Method Pattern
project/asserts.py
from unittest import TestCase
_c = TestCase()
assertEqual = _c.assertEqual
assertNotEqual = _c.assertNotEqual
assertIsNone = _c.assertIsNone
...
from asserts import assertEqual
def test_one():
assertEqual(1+1, 2)
def test_two():
assertEqual(2*2, 4)
</aside>
Takeaways
• How introspection works.
• How to use introspection in developer tools.
• How hard-coded solutions let us start fast.
• How prototypes help us decide between approaches.
• And, of course, you now know how to plug test
functions into `unittest` if you ever want to.
Thank you very much!
I’m Brandon Rhodes.
I hope you enjoy the rest
of Python+DS fwdays’24.
