Unit Testing

Tests are an essential part of good software. They ensure that your implementation does what it is supposed to do. They are also invaluable when making changes to existing code to see whether things break unexpectedly. Whenever you contribute code to ProbNum, make sure it is covered by appropriate tests.

If you’re unsure of what tests are or have never written a test before, check out this basic guide on testing in Python.

Running Tests

ProbNum’s tests can be found in the folder ./tests. To run the entire test suite, we recommend using tox.

Tox

To run the tests using Tox, simply execute

tox -e py3

This runs the tests for your version of Python 3. You can also run the tests against a specific Python version, e.g. tox -e py36 for Python 3.6.

For more information on tox, check out the general development instructions.

PyTest

ProbNum uses pytest as a test runner. You can also run the tests directly by installing (pip install pytest) and executing

pytest

in the ./tests directory.

Using an IDE

If you are using an IDE such as PyCharm or VSCode, you can for example use the graphic interface to run single tests. This can be very convenient when writing tests. To set up your IDE to run pytest, check out the following guides:

• PyCharm: https://www.jetbrains.com/help/pycharm/pytest.html

• VSCode: https://code.visualstudio.com/docs/python/testing#_enable-a-test-framework

from IPython.display import Image

display(Image(url='https://code.visualstudio.com/assets/docs/python/testing/editor-adornments-pytest.png',
              embed=True))

The unittest Framework

ProbNum uses unittest as its testing framework.

Writing a Unit Test

Tests are organized into folders and classes which roughly mirror the structure of the repository. The main components of a test case are the test fixtures, which set up and tear down any required resources and the test cases which are the actual functions implementing tests. Below is a basic example of a test case written using unittest.

import unittest
import numpy as np


class ExampleTestCase(unittest.TestCase):

    def setUp(self):
        np.random.seed(42)
        # Set up a variable which will be reused across multiple tests.
        self.A = np.array([1, 2, 3])

    def test_first_entry(self):
        """Check that the first entry of A is 1."""
        self.assertEqual(self.A[0], 1)

    def test_all_entries(self):
        """Check whether A's entries are 1, 2 and 3."""
        for i in [1, 2, 3]:
            # Iterate over multiple similar tests
            with self.subTest():
                self.assertEqual(self.A[i], i)

Assertions

Assertion statements defined in unittest.TestCase define the actual “test” of a test_* function. If they fail, the associated test fails. Some commonly used statements are:

Method	Checks that
assertEqual(a, b)	a == b
assertNotEqual(a, b)	a != b
assertTrue(x)	bool(x) is True
assertFalse(x)	bool(x) is False
assertIn(a, b)	a in b
assertNotIn(a, b)	a not in b
assertIsInstance(a, b)	isinstance(a, b)
assertNotIsInstance(a, b)	not isinstance(a, b)
assertRaises(exc)	exception exc is raised.

For a complete list see the unittest.TestCase documentation.

NumPy Assertions

Often assertions which compare arrays are needed for tests. The class tests.testing.NumpyAssertions wraps assert statements used in NumPy, e.g.

Method	Checks that
assertApproxEqual(a, b, significant=7)	scalars are equal up to significant digits.
assertAllClose(a, b, rtol, atol, equal_nan=True)	arrays are the same within a desired tolerance.
assertArrayEqual(a, b)	arrays have the same shape and elements.
assertArrayLess(a, b)	all elements of a are strictly smaller than those of b.

Common Types of Tests

We collect some common types of tests here, in particular for probabilistic numerical methods.

In- and Output

• Deterministic input: Does your PN method accept parameters / problem definitions which are not random variables?

• Output Random Variables: Does your PN method output a random variable?

• Shape: Does your method return consistent shapes for differently shaped inputs?

• Expected errors: Are appropriate errors raised for invalid input? Do these match the Raises keyword in the docstring?

• Random state: Does fixing a random seed result in deterministic behaviour?

Testing Probabilistic Numerical Methods

• Perfect information: Does your method converge instantly for a prior encoding the solution of the problem?

• Convergence criteria: Are all convergence criteria covered by at least one test?

• Theoretical Results: Are all available theoretical results about the method checked via tests?