Testing is an important part of software development: there's no point writing code if it doesn't work, or do what you want. Testing is how you find out what your code actually does.
But testing, especially with lots of code, is laborious to do by hand. That's why there are all manner of automated testing tools you can use with Python.
In this post, I'll look at two of the simplest, which have the advantage of being included in the Python standard library: doctest and unittest.
There are many other testing frameworks around, and most of them are (arguably) better than either doctest or unittest in some way: more powerful, easier to write tests, or something.
The examples in this post draw on the cipher-tools project. You might want to look at the post describing how it's organised.
Doctest
Docstrings are a very useful of part of Python. If the first line of a procedure definition is a string, it's referred to as a docstring; it should say something useful about the function.
For instance, the definition of unpos is:
def unpos(number):
"""Return the letter in the given position in the alphabet (mod 26)"""
return chr(number % 26 + ord('a'))
And the string Return the letter in the given position in the alphabet (mod 26) is the docstring.
If you ask for help on this function, Python will return the docstring:
>>> help(unpos)
Help on function unpos in module __main__:
unpos(number)
Return the letter in the given position in the alphabet (mod 26)
Doctest extends this idea by putting simple test cases in the docstring. They act as both documentation (showing someone how to use the function) but Python can automatically test the function against the test cases.
The format is simple: after the "documentation" chunk of the docstring, you put the test cases. Each test case has the case to test (preceeded by >>>, just like the Python command line), followed by the return value (exactly as it would print in the Python REPL). For instance, the letters function contains two tests.
def letters(text):
"""Remove all non-alphabetic characters from a text
>>> letters('The Quick')
'TheQuick'
>>> letters('The Quick BROWN fox jumped! over... the (9lazy) DOG')
'TheQuickBROWNfoxjumpedoverthelazyDOG'
"""
return ''.join([c for c in text if c in string.ascii_letters])
These say that letters('The Quick') should return 'TheQuick', and that letters('The Quick BROWN fox jumped! over... the (9lazy) DOG') should return 'TheQuickBROWNfoxjumpedoverthelazyDOG'.
To call the test cases, you run the doctext.testmod() procedure. Generally, the tests go in Python files which are used as library modules, so you won't run the file directly. Therefore, if you add these few magic lines that the end of the file:
if __name__ == "__main__":
import doctest
doctest.testmod()
and then ask Python to run the file, it will run the tests.
If you just import the file as a module, no tests will run.
For instance, the file utilities.py is meant to be used by importing it as a library. But, with the magic lines above, I can run the tests when I execute the file directly:
neil@desktop:~/Documents/cipher-tools/support$ python3 utilities.py
neil@desktop:~/Documents/cipher-tools/support$
By default, doctest only tells you if a test fails. But you can ask doctest to be verbose with the -v flag:
neil@desktop:~/Documents/cipher-tools/support$ python3 utilities.py -v
Trying:
letters('The Quick')
Expecting:
'TheQuick'
ok
Trying:
letters('The Quick BROWN fox jumped! over... the (9lazy) DOG')
Expecting:
'TheQuickBROWNfoxjumpedoverthelazyDOG'
ok
Trying:
sanitise('The Quick')
Expecting:
'thequick'
ok
neil@desktop:~/Documents/cipher-tools/support$
It's all very simple, but there are a few occasions where you'll need to do help doctest a bit. They stem from the fact that doctest is really simple and just sends text as input to and compares the result as text.
For instance, if you return a dict or something where the elements can be in different orders, you'll need to sort the output in the test case to ensure it always looks the same.
Long input lines
If you input has really long lines and you want to wrap them in the source file, you can't just put in additional newlines like you would in the rest of the docstring. Instead, you need to specifically tell Python that what looks like several lines in the source code is actually one long line. You do that in the standard way, of making sure the last character on the line is a backslash; the next line will continue on from there.
You can see this in the example of testing the affine cipher, below. Note that this example also uses Python's standard approach of combining two strings that are defined next to each other, so 'hours passed during which jerico tried every ' and 'trick he could think of' are combined into the one string 'hours passed during which jerico tried every trick he could think of'.
def affine_encipher(message, multiplier=1, adder=0, one_based=True):
"""Encipher a message
>>> affine_encipher('hours passed during which jerico tried every ' \
'trick he could think of', 15, 22, True)
'lmyfu bkuusd dyfaxw claol psfaom jfasd snsfg jfaoe ls omytd jlaxe mh'
"""
enciphered = [affine_encipher_letter(l, multiplier, adder, one_based)
for l in message]
return cat(enciphered)
Long output lines
If you want to wrap the output, append # doctest: NORMALIZE_WHITESPACE to the test case. That tells doctest to take the expected and actual test results, and collapse all the sequences of whitespace characters in both to single spaces.
In the example below, NORMALIZE_WHITESPACE is used to make doctest ignore the newlines in the expected test outputs.
def ngrams(text, n):
"""Returns all n-grams of a text
>>> ngrams(sanitise('the quick brown fox'), 2) # doctest: +NORMALIZE_WHITESPACE
['th', 'he', 'eq', 'qu', 'ui', 'ic', 'ck', 'kb', 'br', 'ro', 'ow', 'wn',
'nf', 'fo', 'ox']
>>> ngrams(sanitise('the quick brown fox'), 4) # doctest: +NORMALIZE_WHITESPACE
['theq', 'hequ', 'equi', 'quic', 'uick', 'ickb', 'ckbr', 'kbro', 'brow',
'rown', 'ownf', 'wnfo', 'nfox']
"""
return [text[i:i+n] for i in range(len(text)-n+1)]
Numbers and ellipsis
Depending on software versions, computation history, and the direction of the wind, the text representation of floating point numbers can vary in the smaller decimal places. To make sure the tests still pass, you can make doctest skip some of the output.
If you specify # doctest: +ELLIPSIS after the test case, three dots in the expected result will be matched against anything in the actual result. In the exmaple below, the second case will pass if the result is [(1, 0.333), (2, 0.333), (3, 0.333)], [(1, 0.3334), (2, 0.3333, (3, 0.3333)], [(1, 0.3333975), (2, 0.33357676), (3, 0.3332756], or something similar.
def l1_scale(f):
"""Scale a set of frequencies so they sum to one
>>> sorted(normalise({1: 1, 2: 0}).items())
[(1, 1.0), (2, 0.0)]
>>> sorted(normalise({1: 1, 2: 1}).items())
[(1, 0.5), (2, 0.5)]
>>> sorted(normalise({1: 1, 2: 1, 3: 1}).items()) # doctest: +ELLIPSIS
[(1, 0.333...), (2, 0.333...), (3, 0.333...)]
>>> sorted(normalise({1: 1, 2: 2, 3: 1}).items())
[(1, 0.25), (2, 0.5), (3, 0.25)]
"""
return scale(f, l1)
+NORMALIZE_WHITESPACE and +ELLIPSIS are examples of doctest directives. There are many more doctest directives.
extraglobs
It took me a while to work out that extraglobs means "extra global variables." If there are additional global variables you want to refer to in your doctests, you can set them up when you call the tests and they're available for all the tests.
For instance, at the bottom of the file, if you add the pe extrablob:
if __name__ == "__main__":
import doctest
doctest.testmod(extraglobs={'pe': PocketEnigma(1, 'a')})
you can use pe as a variable in tests, like this:
def set_position(self, position):
"""Sets the position of the wheel, by specifying the letter the arrow
points to.
>>> pe.set_position('a')
0
>>> pe.set_position('m')
12
>>> pe.set_position('z')
25
"""
self.position = pos(position)
return self.position
(These examples come from the pocket enigma cipher machine simulation in the ciper-tools library.)
Unit Test
While doctest is very simple to both define test cases and to use, it doesn't have the sophistication needed for complex tests. For those situations, we should turn to unittest.
The basic idea is that you write tests in a separate file, and then run them. unitttest has some advantages over doctest:
- you can test more features, such as checking for exceptions being raised
- you can deal with importing several files
- you can do a more complex setup than just
extraglobs - it's more like standard Python, so you don't need to faff around with various directives
It's best to keep all your tests in a separate test directory, and have one test file for each collection of tests that makes sense. Each test file needs to be called test_something.py. For instance, with the cipher-tools library, I use one test file per source file, such as test_affine.py and test_engima.py.
Within the test file, you need to import the unittest module, along with any other imports you need to do.
Create a new Test object for each group of tests you want to run. Within that, have one procedure for each feature you want to test. These procedures can be as simple or as complex as necessary for the tests you want to run.
import unittest
import string
from cipher.affine import *
from support.utilities import *
class AffineTest(unittest.TestCase):
def test_encipher_letter(self):
...
def test_decipher_letter(self):
...
The testing itself comes in the shape of various assert… calls within the test cases. For instance, the test_encipher_letter procedure below does 112 tests, 56 tests in each of the for loops. Inside each loop, the assertEqual method just checks that the result of the affine_encipher_letter is the same as the expected output.
def test_encipher_letter(self):
for p, c in zip(
string.ascii_letters,
'hknqtwzcfiloruxadgjmpsvybeHKNQTWZCFILORUXADGJMPSVYBE'):
self.assertEqual(affine_encipher_letter(p, 3, 5, True), c)
for p, c in zip(
string.ascii_letters,
'filoruxadgjmpsvybehknqtwzcFILORUXADGJMPSVYBEHKNQTWZC'):
self.assertEqual(affine_encipher_letter(p, 3, 5, False), c)
When it comes to comparing real values, theassertAlmostEqual function allows its two real-number arguments to vary a bit. To deal with dicts and sets, which can be ordered, the assertIn function tests if a given value is in a collection of values.
Running tests
In the root directory of the project, run the command
neil@desktop:~/Documents/cipher-tools/$ python3 -m unittest discover test
The -m unittest tells Python to load the unittest module. discover tells unittest to look for tests, and to look for them in the directory test.
(This is where the naming of the files in the test directory is important: when unittest discovers tests, it looks for files with names starting with test.
But I can never remember that syntax, so I tend to create a little shell script file to run the tests, called run_tests:
#!/bin/bash
python3 -m unittest discover test
Combining doctest and unittest
You can also get unittest to call all the tests you've defined as doctests. so there's no need to rewrite all of them! In the tests directory, create a file called test_doctests.py and have it contain something like this:
import unittest
import doctest
import cipher.caesar
import cipher.affine
import cipher.pocket_enigma
# and so on...
def load_tests(loader, tests, ignore):
tests.addTests(doctest.DocTestSuite(cipher.caesar))
tests.addTests(doctest.DocTestSuite(cipher.affine))
tests.addTests(doctest.DocTestSuite(cipher.pocket_enigma,
extraglobs={'pe': cipher.pocket_enigma.PocketEnigma(1, 'a')}))
# and so on...
return tests
unittest will then pick up all the doctests and run them when it runs all the others.
See also
- Standard library
doctestdocumentation - Standard library
unittestdocumentation - Python Testing's introduction to
doctest - Python Testing's introduction to
unittest
Acknowledgements
Post cover photo by Chris Liverani on Unsplash.