Writing Tests

htf includes a state of the art test runner that supports three different types of tests: functions, methods in simple classes or subclasses of htf.TestCase.

Each function or method that starts with test or that is decorated with htf.test is assumed to be a test and is collected.

We recommend to use functions or simple classes to build tests.

All tests can use Fixtures to set up their test environment.

Tests use Assertions to assert conditions of your tests.

Test Functions

The simplest way to create a test is to define a function. The function is assumed to be a test if the name starts with test, or the function is decorated with htf.test.

Example:

def test_function():
    pass

@htf.test
def function():
    pass

if __name__ == "__main__":
    htf.main()

Test Classes

A class is a container for multiple test methods. The method is assumed to be a test if the name starts with test, or the method is decorated with htf.test.

Example:

class Tests:
    def test_method(self):
        pass

    @htf.test
    def method(self):
        pass

Legacy Tests: TestCase

Unittest-like tests are also supported by subclassing htf.TestCase.

class LegacyTests(htf.TestCase):
    def test_method(self):
        pass

    @htf.test
    def method(self):
        pass

htf.TestCase.setUp, htf.TestCase.tearDown, htf.TestCase.setUpClass and htf.TestCase.tearDownClass are also supported. These tests cannot be run with unittest as a test runner.

class htf.TestCase(method_name='run_test')

TestCase is the basic test case class for the HILSTER Testing Framework that should be used using inheritance.

Parameters

method_name='run_test' (str) – the name of the test method to be run by the test runner.

static assert_almost_equal(a, b, places=None, delta=None, message=None)

Assert that a and b are almost equal. Useful to compare floating point numbers. Comparison is realized either using a maximum delta or the number of decimal places

Parameters

  • a (float) – number

  • b (float) – number

  • places=None (int) – the number of decimal places for comparison. If None 7 decimal places are used.

  • delta=None (float) – the maximum delta

  • message=None (str) – the message to prepend to the error message

Raises

AssertionError – if a and b are not almost equal.

static assert_dict_equal(a, b, message=None)

Assert that a equals b and that both are of type dict.

Parameters

  • a – a dict

  • b – a dict

  • message=None (str) – the message to prepend to the error message

Raises

AssertionError – if a and b are not equal.

static assert_equal(a, b, message=None)

Assert that a equals b. Tries to find a registered equality asserter and uses the base assert if no special assert is registered.

Parameters

  • a – an object

  • b – an object

  • message=None (str) – the message to prepend to the error message

Raises

AssertionError – if a and b are not equal.

static assert_false(expression, message=None)

Assert that expression is False.

Parameters

  • expression (bool) – the boolean expression to be validated

  • message=None (str) – the message to prepend to the error message

Raises

AssertionError – if expression is not False

static assert_greater(a, b, message=None)

Assert that a is greater than b.

Parameters

  • a – a comparable object

  • b – a comparable object

  • message=None (str) – the message to prepend to the error message

Raises

AssertionError – if a is not greater than b

static assert_greater_equal(a, b, message=None)

Assert that a is greater or equal to b.

Parameters

  • a – a comparable object

  • b – a comparable object

  • message=None (str) – the message to prepend to the error message

Raises

AssertionError – if a is not greater or equal to b

static assert_in(member, container, message=None)

Assert that container contains member.

Parameters

  • member – the object that shall be included in container

  • container (iterable) – an iterable to search for member

  • message=None (str) – the message to prepend to the error message

Raises

AssertionError – if member is not included in container

static assert_is(a, b, message=None)

Assert that id(a) equals id(b).

Parameters

  • a – an object

  • b – another object

  • message=None (str) – the message to prepend to the error message

Raises

AssertionError – if id(a) does not equal id(b)

static assert_is_instance(obj, cls, message=None)

Assert that obj is an instance of cls.

Parameters

  • obj – an instance

  • cls – a class obj shall be an instance of

  • message=None (str) – the message to prepend to the error message

Raises

AssertionError – if obj is not an instance of cls.

static assert_is_none(obj, message=None)

Assert that obj is None.

Parameters

  • obj – an object

  • message=None (str) – the message to prepend to the error message

Raises

AssertionError – if obj is not None

static assert_is_not(a, b, message=None)

Assert that id(a) does not equal id(b).

Parameters

  • a – an object

  • b – another object

  • message=None (str) – the message to prepend to the error message

Raises

AssertionError – if id(a) equals id(b)

static assert_is_not_instance(obj, cls, message=None)

Assert that obj is an instance of cls.

Parameters

  • obj – an instance

  • cls – a class obj shall be an instance of

  • message=None (str) – the message to prepend to the error message

Raises

AssertionError – if obj is not an instance of cls.

static assert_is_not_none(obj, message=None)

Assert that obj is not None.

Parameters

  • obj – an object

  • message=None (str) – the message to prepend to the error message

Raises

AssertionError – if obj is None

static assert_less(a, b, message=None)

Assert that a is less than b.

Parameters

  • a – a comparable object

  • b – a comparable object

  • message=None (str) – the message to prepend to the error message

Raises

AssertionError – if a is not less than b

static assert_less_equal(a, b, message=None)

Assert that a is less or equal to b.

Parameters

  • a – a comparable object

  • b – a comparable object

  • message=None (str) – the message to prepend to the error message

Raises

AssertionError – if a is not less or equal to b

static assert_list_equal(a, b, message=None)

Assert that a equals b and that both are of type list.

Parameters

  • a – a list

  • b – a list

  • message=None (str) – the message to prepend to the error message

Raises

AssertionError – if a and b are not equal.

static assert_multi_line_equal(a, b, message=None)

Assert that a equals b and that both are of type str.

Parameters

  • a – a string

  • b – a string

  • message=None (str) – the message to prepend to the error message

Raises

AssertionError – if a and b are not equal.

static assert_not_almost_equal(a, b, places=None, delta=None, message=None)

Assert that a and b are not almost equal. Useful to compare floating point numbers. Comparison is realized either using a maximum delta or the number of decimal places

Parameters

  • a (float) – number

  • b (float) – number

  • places=None (int) – the number of decimal places for comparison. If None 7 decimal places are used.

  • delta=None (float) – the maximum delta

  • message=None (str) – the message to prepend to the error message

Raises

AssertionError – if a and b are almost equal.

static assert_not_equal(a, b, message=None)

Assert that a does not equal b.

Parameters

  • a – an object

  • b – an object

  • message=None (str) – the message to prepend to the error message

Raises

AssertionError – if a and b are equal.

static assert_not_in(member, container, message=None)

Assert that container does not contain member.

Parameters

  • member – the object that shall not be included in container

  • container (iterable) – an iterable to search for member

  • message=None (str) – the message to prepend to the error message

Raises

AssertionError – if member is included in container

static assert_not_is_instance(obj, cls, message=None)

Assert that obj is an instance of cls.

Parameters

  • obj – an instance

  • cls – a class obj shall be an instance of

  • message=None (str) – the message to prepend to the error message

Raises

AssertionError – if obj is not an instance of cls.

static assert_not_regex(text, unexpected_regex, message=None)

Assert that text does not match regular expression unexpected_regex.

Parameters

  • text – a text to be matched

  • unexpected_regex – a regular expression to be used

  • message=None (str) – the message to prepend to the error message

  • Raises – AssertionError: if text matches regular expression unexpected_regex.

static assert_raises(expected_exception, callable_object=None, *args, **kwargs)

Assert that expected_exception is raised. Can be used as a context manager or with a callable object that is run with *args and **kwargs.

Parameters

  • expected_exception (type) – the expected exception that shall be raised

  • callable_object=None (callable) – the callable object if not used as a context manager

  • *args – variable positional arguments passed to callable_object

  • **kwargs – variable keyword arguments passed to callable_object

Returns

a context manager if callable_object is None else None.

static assert_raises_regex(expected_exception, expected_regex, callable_object=None, *args, **kwargs)

Assert that expected_exception is raised. Can be used as a context manager or with a callable object that is run with *args and **kwargs.

Parameters

  • expected_exception (type) – the expected exception that shall be raised

  • expected_regex (regular expression) – the regular expression that shall match the stringified representation of the exception

  • callable_object=None (callable) – the callable object if not used as a context manager

  • *args – variable positional arguments passed to callable_object

  • **kwargs – variable keyword arguments passed to callable_object

Returns

a context manager if callable_object is None else None.

static assert_regex(text, expected_regex, message=None)

Assert that text matches regular expression expected_regex.

Parameters

  • text – a text to be matched

  • expected_regex – a regular expression to be used

  • message=None (str) – the message to prepend to the error message

  • Raises – AssertionError: if text does not match regular expression expected_regex.

static assert_sequence_equal(a, b, message=None, sequence_type=None)

Assert that a equals b and that both are sequences.

Parameters

  • a – a sequence

  • b – a sequence

  • sequence_type=None (type) – the sequence type to be used (if not None)

  • message=None (str) – the message to prepend to the error message

Raises

AssertionError – if a and b are not equal.

static assert_set_equal(a, b, message=None)

Assert that a equals b and that both are of type set.

Parameters

  • a – a set

  • b – a set

  • message=None (str) – the message to prepend to the error message

Raises

AssertionError – if a and b are not equal.

static assert_str_equal(a, b, message=None)

Assert that a equals b and that both are of type str.

Parameters

  • a – a string

  • b – a string

  • message=None (str) – the message to prepend to the error message

Raises

AssertionError – if a and b are not equal.

static assert_true(expression, message=None)

Assert that expression is True.

Parameters

  • expression (bool) – the boolean expression to be validated

  • message=None (str) – the message to prepend to the error message

Raises

AssertionError – if expression is not True

static assert_tuple_equal(a, b, message=None)

Assert that a equals b and that both are of type tuple.

Parameters

  • a – a list

  • b – a list

  • message=None (str) – the message to prepend to the error message

Raises

AssertionError – if a and b are not equal.

attach_file(filename, title)

Attach a file to the current test that is put into the test report.

Parameters

  • filename (str) – the filename to be attached

  • title (str) – the title

attach_url(url, title)

Attach an url to the current test that is put into the test report. This results in an external link and should be used for huge files.

Parameters

  • url (str) – the url to be attached

  • title (str) – the title

delay(duration)

Delay execution for a given duration and print progress.

Parameters

duration (float) – sleep duration

expect(expected_value, observed_value, error_message=None)

Expect that expectedValue and observedValue are the same. If expectedValue and observedValue differ an AssertionError is raised.

Parameters

  • expected_value – the expected value.

  • observed_value – the observed value that is compared to expectedValue.

  • errorMessage=None (str) – an optional message.

expect_step(title, expected_value, observed_value, error_message=None)

Creates a test step in test report and on stdout using title. Expect that expectedValue and observedValue are the same. If expectedValue and observedValue differ an AssertionError is raised.

Parameters

  • title (str) – the step’s title.

  • expected_value – the value that is expected.

  • observed_value – the current observed value that is compared to expectedValue.

  • errorMessage=None (str) – an optional message.

run_background(func, name=None, stop=None, force_stop=True, join_timeout=1, args=None, kwargs=None)

With this method you can run another method in background.

Parameters

  • func (callable) – the method to be run in background.

  • name=None (str) – a name for the background task. Default: None. If name is None func.__name__ is used.

  • stop=None (callable) – callable to stop the thread

  • force_stop=True (bool) – force the thread to stop when leaving its scope (can impact performance)

  • join_timeout=1 (float) – the timeout to be used when calling join on the thread.

  • args=None (tuple) – a tuple of positional arguments that are passed to the background task when it is called.

  • kwargs=None (dict) – a dictionary of keyword arguments that are passed to the background task when it is called.

Returns

a callable object is returned that stops

execution when called.

Return type

stop-trigger (callable)

run_periodic(func, period, maximum_period=None, name=None, raise_exception=True, args=None, kwargs=None)

With this method you can run another method periodically in background with a given period.

Parameters

  • func (callable) – the method to be called periodically in background.

  • period (float) – the period in second.

  • maximum_period=None (float) – if set to a float >= period func may take up to maximum_period time without raising an exception. A warning is printed out to stdout instead.

  • name=None (str) – a name for the background task.

  • raise_exceptions=True – if set to True exceptions are put into the exception queue to let a test fail if a background thread fails.

  • args=None (tuple) – a tuple of positional arguments that are passed to the background task when it is called.

  • kwargs=None (dict) – a dictionary of keyword arguments that are passed to the background task when it is called.

Returns

a callable object is returned that stops

execution when called.

Return type

stop-trigger (callable)

setUp()

This method is called from test runner before a test is executed to create the test-fixtures.

classmethod setUpClass()

This method is called for each class instance.

set_metadata(name, value)

Set metadata for a running test.

Parameters

  • name (str) – the name of the metadata

  • value (str) – the value of the metadata (value is casted to a string automatically)

Raises

AssertionError – if name was already set to non accidently overwrite existing metdata

set_output_capture_enable(enabled=True)

Enables or disables capturing of stdout and stderr for later use in test reports.

Parameters

enabled=True (bool) – the new enabled state

skip_step(message)

Skip the current test step.

skip_test(message)

Skip the current test.

sleep(duration)

Delay execution for a given duration and print progress.

Parameters

duration (float) – sleep duration

step(title, *args, **kwargs)

Run a test step that is documented in the test report.

Parameters

  • title (str) – the step’s title.

  • *args (tuple) – a tuple of positional arguments for the step’s context.

  • **kwargs (dict) – a dictionary of keyword arguments for the step’s context.

stop_thread(name_or_function)

This method stops a specific thread with the name or function described by name_or_function by calling join with join timeout supplied by run_periodic or run_background or on it. It does not set a stop condition so the thread must end and must not include an endless loop. If the thread is still alive after the first join messages are printed to inform the user.

Parameters

name_or_function (str, callable) – the name or the function (reference) of the thread to be stopped.

Returns

the number of stopped threads.

Return type

int

Raises

Exception – if the thread was not found

tearDown()

This method is called from test runner after a test was executed regardless of it’s result.

This method shall destroy all used fixtures.

classmethod tearDownClass()

This method is called for each class instance.