Writing Tests

htf includes a state of the art test runner that supports three different types of tests: functions, coroutines and methods in simple classes or subclasses of htf.TestCase.

Each function or method or coroutine that starts with test or that is decorated with htf.test is assumed to be a test and is collected.

We recommend to use functions, coroutines or simple classes to build tests.

All tests can use Fixtures to set up their test environment.

Tests use Assertions to assert conditions of your tests.

Test Functions

The simplest way to create a test is to define a function. The function is assumed to be a test if the name starts with test, or the function is decorated with htf.test.

Example:

def test_function():
    pass

async def test_async_function():
    pass

@htf.test
def function():
    pass

@htf.test
def async_function():
    pass

if __name__ == "__main__":
    htf.main()

Test Classes

A class is a container for multiple test methods. The method is assumed to be a test if the name starts with test, or the method is decorated with htf.test.

Example:

class Tests:
    def test_method(self):
        pass

    async def test_async_method(self):
        pass

    @htf.test
    def method(self):
        pass

    @htf.test
    async def async_method(self):
        pass

Legacy Tests: TestCase

Unittest-like tests are also supported by subclassing htf.TestCase.

class LegacyTests(htf.TestCase):
    def test_method(self):
        pass

    async def test_async_method(self):
        pass

    @htf.test
    def method(self):
        pass

    @htf.test
    async def async_method(self):
        pass

htf.TestCase.setUp, htf.TestCase.tearDown, htf.TestCase.setUpClass and htf.TestCase.tearDownClass are also supported. These tests cannot be run with unittest as a test runner.

class htf.TestCase(method_name: str = 'run_test')

TestCase is the basic test case class for the HILSTER Testing Framework that should be used with inheritance.

Parameters:

method_name='run_test' – the name of the test method to be run by the test runner.

static assert_almost_equal(a: float, b: float, places: int | None = None, delta: float | None = None, message: str | None = None) None

Assert that a and b are almost equal. Useful to compare floating point numbers. Comparison is realized either using a maximum delta or the number of decimal places

Parameters:
  • a – number

  • b – number

  • places=None – the number of decimal places for comparison. If None 7 decimal places are used.

  • delta=None – the maximum delta

  • message=None – the message to prepend to the error message

Raises:

AssertionError – if a and b are not almost equal.

static assert_dict_equal(a: Dict, b: Dict, message: str | None = None) None

Assert that a equals b and that both are of type dict.

Parameters:
  • a – a dict

  • b – a dict

  • message=None – the message to prepend to the error message

Raises:

AssertionError – if a and b are not equal.

static assert_equal(a: Any, b: Any, message: str | None = None) None

Assert that a equals b. Tries to find a registered equality asserter and uses the base assert if no special assert is registered.

Parameters:
  • a – an object

  • b – an object

  • message=None – the message to prepend to the error message

Raises:

AssertionError – if a and b are not equal.

static assert_false(expression: bool, message: str | None = None) None

Assert that expression is False.

Parameters:
  • expression – the boolean expression to be validated

  • message=None – the message to prepend to the error message

Raises:

AssertionError – if expression is not False

static assert_greater(a: Any, b: Any, message: str | None = None) None

Assert that a is greater than b.

Parameters:
  • a – a comparable object

  • b – a comparable object

  • message=None – the message to prepend to the error message

Raises:

AssertionError – if a is not greater than b

static assert_greater_equal(a: Any, b: Any, message: str | None = None) None

Assert that a is greater or equal to b.

Parameters:
  • a – a comparable object

  • b – a comparable object

  • message=None – the message to prepend to the error message

Raises:

AssertionError – if a is not greater or equal to b

static assert_in(member: Any, container: Iterable, message: str | None = None) None

Assert that container contains member.

Parameters:
  • member – the object that shall be included in container

  • container – an iterable to search for member

  • message=None – the message to prepend to the error message

Raises:

AssertionError – if member is not included in container

static assert_is(a: Any, b: Any, message: str | None = None) None

Assert that id(a) equals id(b).

Parameters:
  • a – an object

  • b – another object

  • message=None – the message to prepend to the error message

Raises:

AssertionError – if id(a) does not equal id(b)

static assert_is_instance(obj: Any, cls: Type, message: str | None = None) None

Assert that obj is an instance of cls.

Parameters:
  • obj – an instance

  • cls – a class obj shall be an instance of

  • message=None – the message to prepend to the error message

Raises:

AssertionError – if obj is not an instance of cls.

static assert_is_none(obj: Any, message: str | None = None) None

Assert that obj is None.

Parameters:
  • obj – an object

  • message=None – the message to prepend to the error message

Raises:

AssertionError – if obj is not None

static assert_is_not(a: Any, b: Any, message: str | None = None) None

Assert that id(a) does not equal id(b).

Parameters:
  • a – an object

  • b – another object

  • message=None – the message to prepend to the error message

Raises:

AssertionError – if id(a) equals id(b)

static assert_is_not_instance(obj: Any, cls: Type, message: str | None = None) None

Assert that obj is an instance of cls.

Parameters:
  • obj – an instance

  • cls – a class obj shall be an instance of

  • message=None – the message to prepend to the error message

Raises:

AssertionError – if obj is not an instance of cls.

static assert_is_not_none(obj: Any, message: str | None = None) None

Assert that obj is not None.

Parameters:
  • obj – an object

  • message=None – the message to prepend to the error message

Raises:

AssertionError – if obj is None

static assert_less(a: Any, b: Any, message: str | None = None) None

Assert that a is less than b.

Parameters:
  • a – a comparable object

  • b – a comparable object

  • message=None – the message to prepend to the error message

Raises:

AssertionError – if a is not less than b

static assert_less_equal(a: Any, b: Any, message: str | None = None) None

Assert that a is less or equal to b.

Parameters:
  • a – a comparable object

  • b – a comparable object

  • message=None – the message to prepend to the error message

Raises:

AssertionError – if a is not less or equal to b

static assert_list_equal(a: List, b: List, message: str | None = None) None

Assert that a equals b and that both are of type list.

Parameters:
  • a – a list

  • b – a list

  • message=None – the message to prepend to the error message

Raises:

AssertionError – if a and b are not equal.

static assert_multi_line_equal(a: str, b: str, message: str | None = None) None

Assert that a equals b and that both are of type str.

Parameters:
  • a – a string

  • b – a string

  • message=None – the message to prepend to the error message

Raises:

AssertionError – if a and b are not equal.

static assert_not_almost_equal(a: float, b: float, places: int | None = None, delta: float | None = None, message: str | None = None) None

Assert that a and b are not almost equal. Useful to compare floating point numbers. Comparison is realized either using a maximum delta or the number of decimal places

Parameters:
  • a – number

  • b – number

  • places=None – the number of decimal places for comparison. If None 7 decimal places are used.

  • delta=None – the maximum delta

  • message=None – the message to prepend to the error message

Raises:

AssertionError – if a and b are almost equal.

static assert_not_equal(a: Any, b: Any, message: str | None = None) None

Assert that a does not equal b.

Parameters:
  • a – an object

  • b – an object

  • message=None – the message to prepend to the error message

Raises:

AssertionError – if a and b are equal.

static assert_not_in(member: Any, container: Iterable, message: str | None = None) None

Assert that container does not contain member.

Parameters:
  • member – the object that shall not be included in container

  • container – an iterable to search for member

  • message=None – the message to prepend to the error message

Raises:

AssertionError – if member is included in container

static assert_not_is_instance(obj: Any, cls: Type, message: str | None = None) None

Assert that obj is an instance of cls.

Parameters:
  • obj – an instance

  • cls – a class obj shall be an instance of

  • message=None – the message to prepend to the error message

Raises:

AssertionError – if obj is not an instance of cls.

static assert_not_regex(text: str, unexpected_regex: str, message: str | None = None) None

Assert that text does not match regular expression unexpected_regex.

Parameters:
  • text – a text to be matched

  • unexpected_regex – a regular expression to be used

  • message=None – the message to prepend to the error message

  • Raises – AssertionError: if text matches regular expression unexpected_regex.

static assert_raises(expected_exception: Type[BaseException], callable_object: Any | None = None, *args: Any, **kwargs: Any) _AssertRaisesContext | None

Assert that expected_exception is raised. Can be used as a context manager or with a callable object that is run with *args and **kwargs.

Parameters:
  • expected_exception – the expected exception that shall be raised

  • callable_object=None – the callable object if not used as a context manager

  • *args – variable positional arguments passed to callable_object

  • **kwargs – variable keyword arguments passed to callable_object

Returns:

a context manager if callable_object is None else None.

static assert_raises_regex(expected_exception: Type[BaseException], expected_regex: str, callable_object: Any | None = None, *args: Any, **kwargs: Any) _AssertRaisesContext | None

Assert that expected_exception is raised. Can be used as a context manager or with a callable object that is run with *args and **kwargs.

Parameters:
  • expected_exception – the expected exception that shall be raised

  • expected_regex – the regular expression that shall match the stringified representation of the exception

  • callable_object=None – the callable object if not used as a context manager

  • *args – variable positional arguments passed to callable_object

  • **kwargs – variable keyword arguments passed to callable_object

Returns:

a context manager if callable_object is None else None.

static assert_regex(text: str, expected_regex: str, message: str | None = None) None

Assert that text matches regular expression expected_regex.

Parameters:
  • text – a text to be matched

  • expected_regex – a regular expression to be used

  • message=None – the message to prepend to the error message

  • Raises – AssertionError: if text does not match regular expression expected_regex.

static assert_sequence_equal(a: Sequence, b: Sequence, message: str | None = None, sequence_type: Type | None = None) None

Assert that a equals b and that both are sequences.

Parameters:
  • a – a sequence

  • b – a sequence

  • sequence_type=None – the sequence type to be used (if not None)

  • message=None – the message to prepend to the error message

Raises:

AssertionError – if a and b are not equal.

static assert_set_equal(a: Set, b: Set, message: str | None = None) None

Assert that a equals b and that both are of type set.

Parameters:
  • a – a set

  • b – a set

  • message=None – the message to prepend to the error message

Raises:

AssertionError – if a and b are not equal.

static assert_str_equal(a: str, b: str, message: str | None = None) None

Assert that a equals b and that both are of type str.

Parameters:
  • a – a string

  • b – a string

  • message=None – the message to prepend to the error message

Raises:

AssertionError – if a and b are not equal.

static assert_true(expression: bool, message: str | None = None) None

Assert that expression is True.

Parameters:
  • expression – the boolean expression to be validated

  • message=None – the message to prepend to the error message

Raises:

AssertionError – if expression is not True

static assert_tuple_equal(a: Tuple, b: Tuple, message: str | None = None) None

Assert that a equals b and that both are of type tuple.

Parameters:
  • a – a list

  • b – a list

  • message=None – the message to prepend to the error message

Raises:

AssertionError – if a and b are not equal.

attach_file(filename: str, title: str) None

Attach a file to the current test that is put into the test report.

Parameters:
  • filename – the filename to be attached

  • title – the title

attach_url(url: str, title: str) None

Attach an url to the current test that is put into the test report. This results in an external link and should be used for huge files.

Parameters:
  • url – the url to be attached

  • title – the title

delay(duration: float) None

Delay execution for a given duration and print progress.

Parameters:

duration – sleep duration

expect(expected_value: Any, observed_value: Any, error_message: str | None = None) None

Expect that expectedValue and observedValue are the same. If expectedValue and observedValue differ an AssertionError is raised.

Parameters:
  • expected_value – the expected value.

  • observed_value – the observed value that is compared to expectedValue.

  • errorMessage=None – an optional message.

expect_step(title: str, expected_value: Any, observed_value: Any, error_message: str | None = None) None

Creates a test step in test report and on stdout using title. Expect that expectedValue and observedValue are the same. If expectedValue and observedValue differ an AssertionError is raised.

Parameters:
  • title – the step’s title.

  • expected_value – the value that is expected.

  • observed_value – the current observed value that is compared to expectedValue.

  • errorMessage=None – an optional message.

run_background(func: Callable[[...], Any], name: str | None = None, stop: Callable[[...], bool] | None = None, force_stop: bool = True, join_timeout: float = 1.0, args: Tuple[Any] | None = None, kwargs: Dict[str, Any] | None = None) Callable[[], None]

With this method you can run another method in background.

Parameters:
  • func – the method to be run in background.

  • name=None – a name for the background task. Default: None. If name is None func.__name__ is used.

  • stop=None – callable to stop the thread

  • force_stop=True – force the thread to stop when leaving its scope (can impact performance)

  • join_timeout=1 – the timeout to be used when calling join on the thread.

  • args=None – a tuple of positional arguments that are passed to the background task when it is called.

  • kwargs=None – a dictionary of keyword arguments that are passed to the background task when it is called.

Returns:

a callable object is returned that stops

execution when called.

Return type:

stop-trigger

run_periodic(func: Callable[[...], Any], period: float, maximum_period: float | None = None, name: str | None = None, raise_exception: bool = True, args: Tuple[Any] | None = None, kwargs: Dict[str, Any] | None = None) Callable[[], None]

With this method you can run another method periodically in background with a given period.

Parameters:
  • func – the method to be called periodically in background.

  • period – the period in second.

  • maximum_period=None – if set to a float >= period func may take up to maximum_period time without raising an exception. A warning is printed out to stdout instead.

  • name=None – a name for the background task.

  • raise_exceptions=True – if set to True exceptions are put into the exception queue to let a test fail if a background thread fails.

  • args=None – a tuple of positional arguments that are passed to the background task when it is called.

  • kwargs=None – a dictionary of keyword arguments that are passed to the background task when it is called.

Returns:

a callable object is returned that stops

execution when called.

Return type:

stop-trigger

set_metadata(name: str, value: Any) None

Set metadata for a running test.

Parameters:
  • name – the name of the metadata

  • value – the value of the metadata (value is casted to a string automatically)

Raises:

AssertionError – if name was already set to non accidentally overwrite existing metadata

set_output_capture_enable(enabled: bool = True) None

Enables or disables capturing of stdout and stderr for later use in test reports.

Parameters:

enabled=True – the new enabled state

set_up() None

This method is called from test runner before a test is executed to create the test-fixtures.

classmethod set_up_class() None

This method is called for each class instance.

skip_step(message: str) None

Skip the current test step.

skip_test(message: str) None

Skip the current test.

async sleep(duration: float) None

Delay execution (async) for a given duration and print progress.

Parameters:

duration – sleep duration

step(title: str, expected_result: str | None = None, *args: Any, **kwargs: Any) TestStep

Run a test step that is documented in the test report.

Parameters:
  • title – the step’s title.

  • *args – a tuple of positional arguments for the step’s context.

  • **kwargs – a dictionary of keyword arguments for the step’s context.

stop_thread(name_or_function: str | Callable) int

This method stops a specific thread with the name or function described by name_or_function by calling join with join timeout supplied by run_periodic or run_background or on it. It does not set a stop condition so the thread must end and must not include an endless loop. If the thread is still alive after the first join messages are printed to inform the user.

Parameters:

name_or_function – the name or the function (reference) of the thread to be stopped.

Returns:

the number of stopped threads.

Return type:

int

Raises:

Exception – if the thread was not found

tear_down() None

This method is called from test runner after a test was executed regardless of its result.

This method shall destroy all used fixtures.

classmethod tear_down_class() None

This method is called for each class instance.