Writing Tests¶
htf includes a state of the art test runner that supports three different types of tests:
functions, coroutines and methods in simple classes or subclasses of htf.TestCase.
Each function or method or coroutine that starts with test or that is decorated with
htf.test is assumed to be a test and is collected.
We recommend to use functions, coroutines or simple classes to build tests.
All tests can use Fixtures to set up their test environment.
Tests use Assertions to assert conditions of your tests.
Test Functions¶
The simplest way to create a test is to define a function.
The function is assumed to be a test if the name starts with test, or the function
is decorated with htf.test.
Example:
def test_function():
pass
async def test_async_function():
pass
@htf.test
def function():
pass
@htf.test
def async_function():
pass
if __name__ == "__main__":
htf.main()
Test Classes¶
A class is a container for multiple test methods.
The method is assumed to be a test if the name starts with test, or the method
is decorated with htf.test.
Example:
class Tests:
def test_method(self):
pass
async def test_async_method(self):
pass
@htf.test
def method(self):
pass
@htf.test
async def async_method(self):
pass
Legacy Tests: TestCase¶
Unittest-like tests are also supported by subclassing htf.TestCase.
class LegacyTests(htf.TestCase):
def test_method(self):
pass
async def test_async_method(self):
pass
@htf.test
def method(self):
pass
@htf.test
async def async_method(self):
pass
htf.TestCase.setUp,
htf.TestCase.tearDown, htf.TestCase.setUpClass and
htf.TestCase.tearDownClass are also supported.
These tests cannot be run with unittest as a test runner.
- class htf.TestCase(method_name: str = 'run_test')¶
TestCaseis the basic test case class for the HILSTER Testing Framework that should be used with inheritance.- Parameters:
method_name='run_test' – the name of the test method to be run by the test runner.
- static assert_almost_equal(a: float, b: float, places: int | None = None, delta: float | None = None, message: str | None = None) None¶
Assert that
aandbare almost equal. Useful to compare floating point numbers. Comparison is realized either using a maximumdeltaor the number of decimalplaces- Parameters:
a – number
b – number
places=None – the number of decimal places for comparison. If
None7decimal places are used.delta=None – the maximum delta
message=None – the message to prepend to the error message
- Raises:
AssertionError – if
aandbare not almost equal.
- static assert_dict_equal(a: Dict, b: Dict, message: str | None = None) None¶
Assert that
aequalsband that both are of typedict.- Parameters:
a – a dict
b – a dict
message=None – the message to prepend to the error message
- Raises:
AssertionError – if
aandbare not equal.
- static assert_equal(a: Any, b: Any, message: str | None = None) None¶
Assert that
aequalsb. Tries to find a registered equality asserter and uses the base assert if no special assert is registered.- Parameters:
a – an object
b – an object
message=None – the message to prepend to the error message
- Raises:
AssertionError – if
aandbare not equal.
- static assert_false(expression: bool, message: str | None = None) None¶
Assert that
expressionisFalse.- Parameters:
expression – the boolean expression to be validated
message=None – the message to prepend to the error message
- Raises:
AssertionError – if
expressionis notFalse
- static assert_greater(a: Any, b: Any, message: str | None = None) None¶
Assert that
ais greater thanb.- Parameters:
a – a comparable object
b – a comparable object
message=None – the message to prepend to the error message
- Raises:
AssertionError – if
ais not greater thanb
- static assert_greater_equal(a: Any, b: Any, message: str | None = None) None¶
Assert that
ais greater or equal tob.- Parameters:
a – a comparable object
b – a comparable object
message=None – the message to prepend to the error message
- Raises:
AssertionError – if
ais not greater or equal tob
- static assert_in(member: Any, container: Iterable, message: str | None = None) None¶
Assert that
containercontainsmember.- Parameters:
member – the object that shall be included in
containercontainer – an iterable to search for
membermessage=None – the message to prepend to the error message
- Raises:
AssertionError – if
memberis not included incontainer
- static assert_is(a: Any, b: Any, message: str | None = None) None¶
Assert that
id(a)equalsid(b).- Parameters:
a – an object
b – another object
message=None – the message to prepend to the error message
- Raises:
AssertionError – if
id(a)does not equalid(b)
- static assert_is_instance(obj: Any, cls: Type, message: str | None = None) None¶
Assert that
objis an instance ofcls.- Parameters:
obj – an instance
cls – a class
objshall be an instance ofmessage=None – the message to prepend to the error message
- Raises:
AssertionError – if
objis not an instance ofcls.
- static assert_is_none(obj: Any, message: str | None = None) None¶
Assert that
objisNone.- Parameters:
obj – an object
message=None – the message to prepend to the error message
- Raises:
AssertionError – if
objis notNone
- static assert_is_not(a: Any, b: Any, message: str | None = None) None¶
Assert that
id(a)does not equalid(b).- Parameters:
a – an object
b – another object
message=None – the message to prepend to the error message
- Raises:
AssertionError – if
id(a)equalsid(b)
- static assert_is_not_instance(obj: Any, cls: Type, message: str | None = None) None¶
Assert that
objis an instance ofcls.- Parameters:
obj – an instance
cls – a class
objshall be an instance ofmessage=None – the message to prepend to the error message
- Raises:
AssertionError – if
objis not an instance ofcls.
- static assert_is_not_none(obj: Any, message: str | None = None) None¶
Assert that
objis notNone.- Parameters:
obj – an object
message=None – the message to prepend to the error message
- Raises:
AssertionError – if
objisNone
- static assert_less(a: Any, b: Any, message: str | None = None) None¶
Assert that
ais less thanb.- Parameters:
a – a comparable object
b – a comparable object
message=None – the message to prepend to the error message
- Raises:
AssertionError – if
ais not less thanb
- static assert_less_equal(a: Any, b: Any, message: str | None = None) None¶
Assert that
ais less or equal tob.- Parameters:
a – a comparable object
b – a comparable object
message=None – the message to prepend to the error message
- Raises:
AssertionError – if
ais not less or equal tob
- static assert_list_equal(a: List, b: List, message: str | None = None) None¶
Assert that
aequalsband that both are of typelist.- Parameters:
a – a list
b – a list
message=None – the message to prepend to the error message
- Raises:
AssertionError – if
aandbare not equal.
- static assert_multi_line_equal(a: str, b: str, message: str | None = None) None¶
Assert that
aequalsband that both are of typestr.- Parameters:
a – a string
b – a string
message=None – the message to prepend to the error message
- Raises:
AssertionError – if
aandbare not equal.
- static assert_not_almost_equal(a: float, b: float, places: int | None = None, delta: float | None = None, message: str | None = None) None¶
Assert that
aandbare not almost equal. Useful to compare floating point numbers. Comparison is realized either using a maximumdeltaor the number of decimalplaces- Parameters:
a – number
b – number
places=None – the number of decimal places for comparison. If
None7decimal places are used.delta=None – the maximum delta
message=None – the message to prepend to the error message
- Raises:
AssertionError – if
aandbare almost equal.
- static assert_not_equal(a: Any, b: Any, message: str | None = None) None¶
Assert that
adoes not equalb.- Parameters:
a – an object
b – an object
message=None – the message to prepend to the error message
- Raises:
AssertionError – if
aandbare equal.
- static assert_not_in(member: Any, container: Iterable, message: str | None = None) None¶
Assert that
containerdoes not containmember.- Parameters:
member – the object that shall not be included in
containercontainer – an iterable to search for
membermessage=None – the message to prepend to the error message
- Raises:
AssertionError – if
memberis included incontainer
- static assert_not_is_instance(obj: Any, cls: Type, message: str | None = None) None¶
Assert that
objis an instance ofcls.- Parameters:
obj – an instance
cls – a class
objshall be an instance ofmessage=None – the message to prepend to the error message
- Raises:
AssertionError – if
objis not an instance ofcls.
- static assert_not_regex(text: str, unexpected_regex: str, message: str | None = None) None¶
Assert that
textdoes not match regular expressionunexpected_regex.- Parameters:
text – a text to be matched
unexpected_regex – a regular expression to be used
message=None – the message to prepend to the error message
Raises – AssertionError: if
textmatches regular expressionunexpected_regex.
- static assert_raises(expected_exception: Type[BaseException], callable_object: Any | None = None, *args: Any, **kwargs: Any) _AssertRaisesContext | None¶
Assert that
expected_exceptionis raised. Can be used as a context manager or with a callable object that is run with*argsand**kwargs.- Parameters:
expected_exception – the expected exception that shall be raised
callable_object=None – the callable object if not used as a context manager
*args – variable positional arguments passed to
callable_object**kwargs – variable keyword arguments passed to
callable_object
- Returns:
a context manager if
callable_objectisNoneelseNone.
- static assert_raises_regex(expected_exception: Type[BaseException], expected_regex: str, callable_object: Any | None = None, *args: Any, **kwargs: Any) _AssertRaisesContext | None¶
Assert that
expected_exceptionis raised. Can be used as a context manager or with a callable object that is run with*argsand**kwargs.- Parameters:
expected_exception – the expected exception that shall be raised
expected_regex – the regular expression that shall match the stringified representation of the exception
callable_object=None – the callable object if not used as a context manager
*args – variable positional arguments passed to
callable_object**kwargs – variable keyword arguments passed to
callable_object
- Returns:
a context manager if
callable_objectisNoneelseNone.
- static assert_regex(text: str, expected_regex: str, message: str | None = None) None¶
Assert that
textmatches regular expressionexpected_regex.- Parameters:
text – a text to be matched
expected_regex – a regular expression to be used
message=None – the message to prepend to the error message
Raises – AssertionError: if
textdoes not match regular expressionexpected_regex.
- static assert_sequence_equal(a: Sequence, b: Sequence, message: str | None = None, sequence_type: Type | None = None) None¶
Assert that
aequalsband that both are sequences.- Parameters:
a – a sequence
b – a sequence
sequence_type=None – the sequence type to be used (if not
None)message=None – the message to prepend to the error message
- Raises:
AssertionError – if
aandbare not equal.
- static assert_set_equal(a: Set, b: Set, message: str | None = None) None¶
Assert that
aequalsband that both are of typeset.- Parameters:
a – a set
b – a set
message=None – the message to prepend to the error message
- Raises:
AssertionError – if
aandbare not equal.
- static assert_str_equal(a: str, b: str, message: str | None = None) None¶
Assert that
aequalsband that both are of typestr.- Parameters:
a – a string
b – a string
message=None – the message to prepend to the error message
- Raises:
AssertionError – if
aandbare not equal.
- static assert_true(expression: bool, message: str | None = None) None¶
Assert that
expressionisTrue.- Parameters:
expression – the boolean expression to be validated
message=None – the message to prepend to the error message
- Raises:
AssertionError – if
expressionis notTrue
- static assert_tuple_equal(a: Tuple, b: Tuple, message: str | None = None) None¶
Assert that
aequalsband that both are of typetuple.- Parameters:
a – a list
b – a list
message=None – the message to prepend to the error message
- Raises:
AssertionError – if
aandbare not equal.
- attach_file(filename: str, title: str) None¶
Attach a file to the current test that is put into the test report.
- Parameters:
filename – the filename to be attached
title – the title
- attach_url(url: str, title: str) None¶
Attach an url to the current test that is put into the test report. This results in an external link and should be used for huge files.
- Parameters:
url – the url to be attached
title – the title
- delay(duration: float) None¶
Delay execution for a given duration and print progress.
- Parameters:
duration – sleep duration
- expect(expected_value: Any, observed_value: Any, error_message: str | None = None) None¶
Expect that
expectedValueandobservedValueare the same. IfexpectedValueandobservedValuediffer anAssertionErroris raised.- Parameters:
expected_value – the expected value.
observed_value – the observed value that is compared to
expectedValue.errorMessage=None – an optional message.
- expect_step(title: str, expected_value: Any, observed_value: Any, error_message: str | None = None) None¶
Creates a test step in test report and on stdout using
title. Expect that expectedValue and observedValue are the same. IfexpectedValueandobservedValuediffer anAssertionErroris raised.- Parameters:
title – the step’s title.
expected_value – the value that is expected.
observed_value – the current observed value that is compared to
expectedValue.errorMessage=None – an optional message.
- run_background(func: Callable[[...], Any], name: str | None = None, stop: Callable[[...], bool] | None = None, force_stop: bool = True, join_timeout: float = 1.0, args: Tuple[Any] | None = None, kwargs: Dict[str, Any] | None = None) Callable[[], None]¶
With this method you can run another method in background.
- Parameters:
func – the method to be run in background.
name=None – a name for the background task. Default:
None. If name isNonefunc.__name__ is used.stop=None – callable to stop the thread
force_stop=True – force the thread to stop when leaving its scope (can impact performance)
join_timeout=1 – the timeout to be used when calling
joinon the thread.args=None – a tuple of positional arguments that are passed to the background task when it is called.
kwargs=None – a dictionary of keyword arguments that are passed to the background task when it is called.
- Returns:
- a callable object is returned that stops
execution when called.
- Return type:
stop-trigger
- run_periodic(func: Callable[[...], Any], period: float, maximum_period: float | None = None, name: str | None = None, raise_exception: bool = True, args: Tuple[Any] | None = None, kwargs: Dict[str, Any] | None = None) Callable[[], None]¶
With this method you can run another method periodically in background with a given period.
- Parameters:
func – the method to be called periodically in background.
period – the period in second.
maximum_period=None – if set to a float >=
periodfuncmay take up tomaximum_periodtime without raising an exception. A warning is printed out to stdout instead.name=None – a name for the background task.
raise_exceptions=True – if set to
Trueexceptions are put into the exception queue to let a test fail if a background thread fails.args=None – a tuple of positional arguments that are passed to the background task when it is called.
kwargs=None – a dictionary of keyword arguments that are passed to the background task when it is called.
- Returns:
- a callable object is returned that stops
execution when called.
- Return type:
stop-trigger
- set_metadata(name: str, value: Any) None¶
Set metadata for a running test.
- Parameters:
name – the name of the metadata
value – the value of the metadata (
valueis casted to a string automatically)
- Raises:
AssertionError – if
namewas already set to non accidentally overwrite existing metadata
- set_output_capture_enable(enabled: bool = True) None¶
Enables or disables capturing of stdout and stderr for later use in test reports.
- Parameters:
enabled=True – the new enabled state
- set_up() None¶
This method is called from test runner before a test is executed to create the test-fixtures.
- async sleep(duration: float) None¶
Delay execution (async) for a given duration and print progress.
- Parameters:
duration – sleep duration
- step(title: str, expected_result: str | None = None, *args: Any, **kwargs: Any) TestStep¶
Run a test step that is documented in the test report.
- Parameters:
title – the step’s title.
*args – a tuple of positional arguments for the step’s context.
**kwargs – a dictionary of keyword arguments for the step’s context.
- stop_thread(name_or_function: str | Callable) int¶
This method stops a specific thread with the name or function described by
name_or_functionby callingjoinwith join timeout supplied byrun_periodicorrun_backgroundor on it. It does not set a stop condition so the thread must end and must not include an endless loop. If the thread is still alive after the first join messages are printed to inform the user.