Writing Tests

htf includes a state of the art test runner that supports three different types of tests: functions, coroutines and methods in simple classes or subclasses of htf.TestCase.

Each function or method or coroutine that starts with test or that is decorated with htf.test is assumed to be a test and is collected.

We recommend to use functions, coroutines or simple classes to build tests.

All tests can use Fixtures to set up their test environment.

Tests use Assertions to assert conditions of your tests.

Test Functions

The simplest way to create a test is to define a function. The function is assumed to be a test if the name starts with test, or the function is decorated with htf.test.

Example:

def test_function():
    pass

async def test_async_function():
    pass

@htf.test
def function():
    pass

@htf.test
def async_function():
    pass

if __name__ == "__main__":
    htf.main()

Test Classes

A class is a container for multiple test methods. The method is assumed to be a test if the name starts with test, or the method is decorated with htf.test.

Example:

class Tests:
    def test_method(self):
        pass

    async def test_async_method(self):
        pass

    @htf.test
    def method(self):
        pass

    @htf.test
    async def async_method(self):
        pass

Legacy Tests: TestCase

Unittest-like tests are also supported by subclassing htf.TestCase.

class LegacyTests(htf.TestCase):
    def test_method(self):
        pass

    async def test_async_method(self):
        pass

    @htf.test
    def method(self):
        pass

    @htf.test
    async def async_method(self):
        pass

htf.TestCase.setUp, htf.TestCase.tearDown, htf.TestCase.setUpClass and htf.TestCase.tearDownClass are also supported. These tests cannot be run with unittest as a test runner.

class htf.TestCase(method_name: str = 'run_test')

TestCase is the basic test case class for the HILSTER Testing Framework that should be used with inheritance.

Parameters:: method_name='run_test' – the name of the test method to be run by the test runner.

static assert_almost_equal(a: float, b: float, places: int | None = None, delta: float | None = None, message: str | None = None) → None

Assert that a and b are almost equal. Useful to compare floating point numbers. Comparison is realized either using a maximum delta or the number of decimal places

Parameters:

a – number
b – number
places=None – the number of decimal places for comparison. If None 7 decimal places are used.
delta=None – the maximum delta
message=None – the message to prepend to the error message

Raises:

AssertionError – if a and b are not almost equal.

static assert_dict_equal(a: Dict, b: Dict, message: str | None = None) → None

Assert that a equals b and that both are of type dict.

Parameters:

a – a dict
b – a dict
message=None – the message to prepend to the error message

Raises:

AssertionError – if a and b are not equal.

static assert_equal(a: Any, b: Any, message: str | None = None) → None

Assert that a equals b. Tries to find a registered equality asserter and uses the base assert if no special assert is registered.

Parameters:

a – an object
b – an object
message=None – the message to prepend to the error message

Raises:

AssertionError – if a and b are not equal.

static assert_false(expression: bool, message: str | None = None) → None

Assert that expression is False.

Parameters:

expression – the boolean expression to be validated
message=None – the message to prepend to the error message

Raises:

AssertionError – if expression is not False

static assert_greater(a: Any, b: Any, message: str | None = None) → None

Assert that a is greater than b.

Parameters:

a – a comparable object
b – a comparable object
message=None – the message to prepend to the error message

Raises:

AssertionError – if a is not greater than b

static assert_greater_equal(a: Any, b: Any, message: str | None = None) → None

Assert that a is greater or equal to b.

Parameters:

a – a comparable object
b – a comparable object
message=None – the message to prepend to the error message

Raises:

AssertionError – if a is not greater or equal to b

static assert_in(member: Any, container: Iterable, message: str | None = None) → None

Assert that container contains member.

Parameters:

member – the object that shall be included in container
container – an iterable to search for member
message=None – the message to prepend to the error message

Raises:

AssertionError – if member is not included in container

static assert_is(a: Any, b: Any, message: str | None = None) → None

Assert that id(a) equals id(b).

Parameters:

a – an object
b – another object
message=None – the message to prepend to the error message

Raises:

AssertionError – if id(a) does not equal id(b)

static assert_is_instance(obj: Any, cls: Type, message: str | None = None) → None

Assert that obj is an instance of cls.

Parameters:

obj – an instance
cls – a class obj shall be an instance of
message=None – the message to prepend to the error message

Raises:

AssertionError – if obj is not an instance of cls.

static assert_is_none(obj: Any, message: str | None = None) → None

Assert that obj is None.

Parameters:

obj – an object
message=None – the message to prepend to the error message

Raises:

AssertionError – if obj is not None

static assert_is_not(a: Any, b: Any, message: str | None = None) → None

Assert that id(a) does not equal id(b).

Parameters:

a – an object
b – another object
message=None – the message to prepend to the error message

Raises:

AssertionError – if id(a) equals id(b)

static assert_is_not_instance(obj: Any, cls: Type, message: str | None = None) → None

Assert that obj is an instance of cls.

Parameters:

obj – an instance
cls – a class obj shall be an instance of
message=None – the message to prepend to the error message

Raises:

AssertionError – if obj is not an instance of cls.

static assert_is_not_none(obj: Any, message: str | None = None) → None

Assert that obj is not None.

Parameters:

obj – an object
message=None – the message to prepend to the error message

Raises:

AssertionError – if obj is None

static assert_less(a: Any, b: Any, message: str | None = None) → None

Assert that a is less than b.

Parameters:

a – a comparable object
b – a comparable object
message=None – the message to prepend to the error message

Raises:

AssertionError – if a is not less than b

static assert_less_equal(a: Any, b: Any, message: str | None = None) → None

Assert that a is less or equal to b.

Parameters:

a – a comparable object
b – a comparable object
message=None – the message to prepend to the error message

Raises:

AssertionError – if a is not less or equal to b

static assert_list_equal(a: List, b: List, message: str | None = None) → None

Assert that a equals b and that both are of type list.

Parameters:

a – a list
b – a list
message=None – the message to prepend to the error message

Raises:

AssertionError – if a and b are not equal.

static assert_multi_line_equal(a: str, b: str, message: str | None = None) → None

Assert that a equals b and that both are of type str.

Parameters:

a – a string
b – a string
message=None – the message to prepend to the error message

Raises:

AssertionError – if a and b are not equal.

static assert_not_almost_equal(a: float, b: float, places: int | None = None, delta: float | None = None, message: str | None = None) → None

Assert that a and b are not almost equal. Useful to compare floating point numbers. Comparison is realized either using a maximum delta or the number of decimal places

Parameters:

a – number
b – number
places=None – the number of decimal places for comparison. If None 7 decimal places are used.
delta=None – the maximum delta
message=None – the message to prepend to the error message

Raises:

AssertionError – if a and b are almost equal.

static assert_not_equal(a: Any, b: Any, message: str | None = None) → None

Assert that a does not equal b.

Parameters:

a – an object
b – an object
message=None – the message to prepend to the error message

Raises:

AssertionError – if a and b are equal.

static assert_not_in(member: Any, container: Iterable, message: str | None = None) → None

Assert that container does not contain member.

Parameters:

member – the object that shall not be included in container
container – an iterable to search for member
message=None – the message to prepend to the error message

Raises:

AssertionError – if member is included in container

static assert_not_is_instance(obj: Any, cls: Type, message: str | None = None) → None

Assert that obj is an instance of cls.

Parameters:

obj – an instance
cls – a class obj shall be an instance of
message=None – the message to prepend to the error message

Raises:

AssertionError – if obj is not an instance of cls.

static assert_not_regex(text: str, unexpected_regex: str, message: str | None = None) → None

Assert that text does not match regular expression unexpected_regex.

Parameters:

text – a text to be matched
unexpected_regex – a regular expression to be used
message=None – the message to prepend to the error message
Raises – AssertionError: if text matches regular expression unexpected_regex.

static assert_raises(expected_exception: Type[BaseException], callable_object: Any | None = None, *args: Any, **kwargs: Any) → _AssertRaisesContext | None

Assert that expected_exception is raised. Can be used as a context manager or with a callable object that is run with *args and **kwargs.

Parameters:

expected_exception – the expected exception that shall be raised
callable_object=None – the callable object if not used as a context manager
*args – variable positional arguments passed to callable_object
**kwargs – variable keyword arguments passed to callable_object

Returns:

a context manager if callable_object is None else None.

static assert_raises_regex(expected_exception: Type[BaseException], expected_regex: str, callable_object: Any | None = None, *args: Any, **kwargs: Any) → _AssertRaisesContext | None

Assert that expected_exception is raised. Can be used as a context manager or with a callable object that is run with *args and **kwargs.

Parameters:

expected_exception – the expected exception that shall be raised
expected_regex – the regular expression that shall match the stringified representation of the exception
callable_object=None – the callable object if not used as a context manager
*args – variable positional arguments passed to callable_object
**kwargs – variable keyword arguments passed to callable_object

Returns:

a context manager if callable_object is None else None.

static assert_regex(text: str, expected_regex: str, message: str | None = None) → None

Assert that text matches regular expression expected_regex.

Parameters:

text – a text to be matched
expected_regex – a regular expression to be used
message=None – the message to prepend to the error message
Raises – AssertionError: if text does not match regular expression expected_regex.

static assert_sequence_equal(a: Sequence, b: Sequence, message: str | None = None, sequence_type: Type | None = None) → None

Assert that a equals b and that both are sequences.

Parameters:

a – a sequence
b – a sequence
sequence_type=None – the sequence type to be used (if not None)
message=None – the message to prepend to the error message

Raises:

AssertionError – if a and b are not equal.

static assert_set_equal(a: Set, b: Set, message: str | None = None) → None

Assert that a equals b and that both are of type set.

Parameters:

a – a set
b – a set
message=None – the message to prepend to the error message

Raises:

AssertionError – if a and b are not equal.

static assert_str_equal(a: str, b: str, message: str | None = None) → None

Assert that a equals b and that both are of type str.

Parameters:

a – a string
b – a string
message=None – the message to prepend to the error message

Raises:

AssertionError – if a and b are not equal.

static assert_true(expression: bool, message: str | None = None) → None

Assert that expression is True.

Parameters:

expression – the boolean expression to be validated
message=None – the message to prepend to the error message

Raises:

AssertionError – if expression is not True

static assert_tuple_equal(a: Tuple, b: Tuple, message: str | None = None) → None

Assert that a equals b and that both are of type tuple.

Parameters:

a – a list
b – a list
message=None – the message to prepend to the error message

Raises:

AssertionError – if a and b are not equal.

attach_file(filename: str, title: str) → None

Attach a file to the current test that is put into the test report.

Parameters:

filename – the filename to be attached
title – the title

attach_url(url: str, title: str) → None

Attach an url to the current test that is put into the test report. This results in an external link and should be used for huge files.

Parameters:

url – the url to be attached
title – the title

delay(duration: float) → None

Delay execution for a given duration and print progress.

Parameters:: duration – sleep duration

expect(expected_value: Any, observed_value: Any, error_message: str | None = None) → None

Expect that expectedValue and observedValue are the same. If expectedValue and observedValue differ an AssertionError is raised.

Parameters:

expected_value – the expected value.
observed_value – the observed value that is compared to expectedValue.
errorMessage=None – an optional message.

expect_step(title: str, expected_value: Any, observed_value: Any, error_message: str | None = None) → None

Creates a test step in test report and on stdout using title. Expect that expectedValue and observedValue are the same. If expectedValue and observedValue differ an AssertionError is raised.

Parameters:

title – the step’s title.
expected_value – the value that is expected.
observed_value – the current observed value that is compared to expectedValue.
errorMessage=None – an optional message.

run_background(func: Callable[[...], Any], name: str | None = None, stop: Callable[[...], bool] | None = None, force_stop: bool = True, join_timeout: float = 1.0, args: Tuple[Any] | None = None, kwargs: Dict[str, Any] | None = None) → Callable[[], None]

With this method you can run another method in background.

Parameters:

func – the method to be run in background.
name=None – a name for the background task. Default: None. If name is None func.__name__ is used.
stop=None – callable to stop the thread
force_stop=True – force the thread to stop when leaving its scope (can impact performance)
join_timeout=1 – the timeout to be used when calling join on the thread.
args=None – a tuple of positional arguments that are passed to the background task when it is called.
kwargs=None – a dictionary of keyword arguments that are passed to the background task when it is called.

Returns:

a callable object is returned that stops: execution when called.

Return type:

stop-trigger

run_periodic(func: Callable[[...], Any], period: float, maximum_period: float | None = None, name: str | None = None, raise_exception: bool = True, args: Tuple[Any] | None = None, kwargs: Dict[str, Any] | None = None) → Callable[[], None]

With this method you can run another method periodically in background with a given period.

Parameters:

func – the method to be called periodically in background.
period – the period in second.
maximum_period=None – if set to a float >= period func may take up to maximum_period time without raising an exception. A warning is printed out to stdout instead.
name=None – a name for the background task.
raise_exceptions=True – if set to True exceptions are put into the exception queue to let a test fail if a background thread fails.
args=None – a tuple of positional arguments that are passed to the background task when it is called.
kwargs=None – a dictionary of keyword arguments that are passed to the background task when it is called.

Returns:

a callable object is returned that stops: execution when called.

Return type:

stop-trigger

set_metadata(name: str, value: Any) → None

Set metadata for a running test.

Parameters:

name – the name of the metadata
value – the value of the metadata (value is casted to a string automatically)

Raises:

AssertionError – if name was already set to non accidentally overwrite existing metadata

set_output_capture_enable(enabled: bool = True) → None

Enables or disables capturing of stdout and stderr for later use in test reports.

Parameters:: enabled=True – the new enabled state

set_up() → None: This method is called from test runner before a test is executed to create the test-fixtures.

classmethod set_up_class() → None: This method is called for each class instance.

skip_step(message: str) → None: Skip the current test step.

skip_test(message: str) → None: Skip the current test.

async sleep(duration: float) → None

Delay execution (async) for a given duration and print progress.

Parameters:: duration – sleep duration

step(title: str, expected_result: str | None = None, *args: Any, **kwargs: Any) → TestStep

Run a test step that is documented in the test report.

Parameters:

title – the step’s title.
*args – a tuple of positional arguments for the step’s context.
**kwargs – a dictionary of keyword arguments for the step’s context.

stop_thread(name_or_function: str | Callable) → int

This method stops a specific thread with the name or function described by name_or_function by calling join with join timeout supplied by run_periodic or run_background or on it. It does not set a stop condition so the thread must end and must not include an endless loop. If the thread is still alive after the first join messages are printed to inform the user.

Parameters:: name_or_function – the name or the function (reference) of the thread to be stopped.
Returns:: the number of stopped threads.
Return type:: int
Raises:: Exception – if the thread was not found

tear_down() → None

This method is called from test runner after a test was executed regardless of its result.

This method shall destroy all used fixtures.

classmethod tear_down_class() → None: This method is called for each class instance.