Writing Tests¶
htf includes a state of the art test runner that supports three different types of tests:
functions, methods in simple classes or subclasses of htf.TestCase
.
Each function or method that starts with test
or that is decorated with
htf.test
is assumed to be a test and is collected.
We recommend to use functions or simple classes to build tests.
All tests can use Fixtures to set up their test environment.
Tests use Assertions to assert conditions of your tests.
Test Functions¶
The simplest way to create a test is to define a function.
The function is assumed to be a test if the name starts with test
, or the function
is decorated with htf.test
.
Example:
def test_function():
pass
@htf.test
def function():
pass
if __name__ == "__main__":
htf.main()
Test Classes¶
A class is a container for multiple test methods.
The method is assumed to be a test if the name starts with test
, or the method
is decorated with htf.test
.
Example:
class Tests:
def test_method(self):
pass
@htf.test
def method(self):
pass
Legacy Tests: TestCase¶
Unittest-like tests are also supported by subclassing htf.TestCase
.
class LegacyTests(htf.TestCase):
def test_method(self):
pass
@htf.test
def method(self):
pass
htf.TestCase.setUp
,
htf.TestCase.tearDown
, htf.TestCase.setUpClass
and
htf.TestCase.tearDownClass
are also supported.
These tests cannot be run with unittest as a test runner.
-
class
htf.
TestCase
(method_name='run_test')¶ TestCase
is the basic test case class for the HILSTER Testing Framework that should be used using inheritance.- Parameters
method_name='run_test' (str) – the name of the test method to be run by the test runner.
-
static
assert_almost_equal
(a, b, places=None, delta=None, message=None)¶ Assert that
a
andb
are almost equal. Useful to compare floating point numbers. Comparison is realized either using a maximumdelta
or the number of decimalplaces
- Parameters
- Raises
AssertionError – if
a
andb
are not almost equal.
-
static
assert_dict_equal
(a, b, message=None)¶ Assert that
a
equalsb
and that both are of typedict
.- Parameters
a – a dict
b – a dict
message=None (str) – the message to prepend to the error message
- Raises
AssertionError – if
a
andb
are not equal.
-
static
assert_equal
(a, b, message=None)¶ Assert that
a
equalsb
. Tries to find a registered equality asserter and uses the base assert if no special assert is registered.- Parameters
a – an object
b – an object
message=None (str) – the message to prepend to the error message
- Raises
AssertionError – if
a
andb
are not equal.
-
static
assert_false
(expression, message=None)¶ Assert that
expression
isFalse
.- Parameters
- Raises
AssertionError – if
expression
is notFalse
-
static
assert_greater
(a, b, message=None)¶ Assert that
a
is greater thanb
.- Parameters
a – a comparable object
b – a comparable object
message=None (str) – the message to prepend to the error message
- Raises
AssertionError – if
a
is not greater thanb
-
static
assert_greater_equal
(a, b, message=None)¶ Assert that
a
is greater or equal tob
.- Parameters
a – a comparable object
b – a comparable object
message=None (str) – the message to prepend to the error message
- Raises
AssertionError – if
a
is not greater or equal tob
-
static
assert_in
(member, container, message=None)¶ Assert that
container
containsmember
.- Parameters
member – the object that shall be included in
container
container (iterable) – an iterable to search for
member
message=None (str) – the message to prepend to the error message
- Raises
AssertionError – if
member
is not included incontainer
-
static
assert_is
(a, b, message=None)¶ Assert that
id(a)
equalsid(b)
.- Parameters
a – an object
b – another object
message=None (str) – the message to prepend to the error message
- Raises
AssertionError – if
id(a)
does not equalid(b)
-
static
assert_is_instance
(obj, cls, message=None)¶ Assert that
obj
is an instance ofcls
.- Parameters
obj – an instance
cls – a class
obj
shall be an instance ofmessage=None (str) – the message to prepend to the error message
- Raises
AssertionError – if
obj
is not an instance ofcls
.
-
static
assert_is_none
(obj, message=None)¶ Assert that
obj
isNone
.- Parameters
obj – an object
message=None (str) – the message to prepend to the error message
- Raises
AssertionError – if
obj
is notNone
-
static
assert_is_not
(a, b, message=None)¶ Assert that
id(a)
does not equalid(b)
.- Parameters
a – an object
b – another object
message=None (str) – the message to prepend to the error message
- Raises
AssertionError – if
id(a)
equalsid(b)
-
static
assert_is_not_instance
(obj, cls, message=None)¶ Assert that
obj
is an instance ofcls
.- Parameters
obj – an instance
cls – a class
obj
shall be an instance ofmessage=None (str) – the message to prepend to the error message
- Raises
AssertionError – if
obj
is not an instance ofcls
.
-
static
assert_is_not_none
(obj, message=None)¶ Assert that
obj
is notNone
.- Parameters
obj – an object
message=None (str) – the message to prepend to the error message
- Raises
AssertionError – if
obj
isNone
-
static
assert_less
(a, b, message=None)¶ Assert that
a
is less thanb
.- Parameters
a – a comparable object
b – a comparable object
message=None (str) – the message to prepend to the error message
- Raises
AssertionError – if
a
is not less thanb
-
static
assert_less_equal
(a, b, message=None)¶ Assert that
a
is less or equal tob
.- Parameters
a – a comparable object
b – a comparable object
message=None (str) – the message to prepend to the error message
- Raises
AssertionError – if
a
is not less or equal tob
-
static
assert_list_equal
(a, b, message=None)¶ Assert that
a
equalsb
and that both are of typelist
.- Parameters
a – a list
b – a list
message=None (str) – the message to prepend to the error message
- Raises
AssertionError – if
a
andb
are not equal.
-
static
assert_multi_line_equal
(a, b, message=None)¶ Assert that
a
equalsb
and that both are of typestr
.- Parameters
a – a string
b – a string
message=None (str) – the message to prepend to the error message
- Raises
AssertionError – if
a
andb
are not equal.
-
static
assert_not_almost_equal
(a, b, places=None, delta=None, message=None)¶ Assert that
a
andb
are not almost equal. Useful to compare floating point numbers. Comparison is realized either using a maximumdelta
or the number of decimalplaces
- Parameters
- Raises
AssertionError – if
a
andb
are almost equal.
-
static
assert_not_equal
(a, b, message=None)¶ Assert that
a
does not equalb
.- Parameters
a – an object
b – an object
message=None (str) – the message to prepend to the error message
- Raises
AssertionError – if
a
andb
are equal.
-
static
assert_not_in
(member, container, message=None)¶ Assert that
container
does not containmember
.- Parameters
member – the object that shall not be included in
container
container (iterable) – an iterable to search for
member
message=None (str) – the message to prepend to the error message
- Raises
AssertionError – if
member
is included incontainer
-
static
assert_not_is_instance
(obj, cls, message=None)¶ Assert that
obj
is an instance ofcls
.- Parameters
obj – an instance
cls – a class
obj
shall be an instance ofmessage=None (str) – the message to prepend to the error message
- Raises
AssertionError – if
obj
is not an instance ofcls
.
-
static
assert_not_regex
(text, unexpected_regex, message=None)¶ Assert that
text
does not match regular expressionunexpected_regex
.- Parameters
text – a text to be matched
unexpected_regex – a regular expression to be used
message=None (str) – the message to prepend to the error message
Raises – AssertionError: if
text
matches regular expressionunexpected_regex
.
-
static
assert_raises
(expected_exception, callable_object=None, *args, **kwargs)¶ Assert that
expected_exception
is raised. Can be used as a context manager or with a callable object that is run with*args
and**kwargs
.- Parameters
expected_exception (type) – the expected exception that shall be raised
callable_object=None (callable) – the callable object if not used as a context manager
*args – variable positional arguments passed to
callable_object
**kwargs – variable keyword arguments passed to
callable_object
- Returns
a context manager if
callable_object
isNone
elseNone
.
-
static
assert_raises_regex
(expected_exception, expected_regex, callable_object=None, *args, **kwargs)¶ Assert that
expected_exception
is raised. Can be used as a context manager or with a callable object that is run with*args
and**kwargs
.- Parameters
expected_exception (type) – the expected exception that shall be raised
expected_regex (regular expression) – the regular expression that shall match the stringified representation of the exception
callable_object=None (callable) – the callable object if not used as a context manager
*args – variable positional arguments passed to
callable_object
**kwargs – variable keyword arguments passed to
callable_object
- Returns
a context manager if
callable_object
isNone
elseNone
.
-
static
assert_regex
(text, expected_regex, message=None)¶ Assert that
text
matches regular expressionexpected_regex
.- Parameters
text – a text to be matched
expected_regex – a regular expression to be used
message=None (str) – the message to prepend to the error message
Raises – AssertionError: if
text
does not match regular expressionexpected_regex
.
-
static
assert_sequence_equal
(a, b, message=None, sequence_type=None)¶ Assert that
a
equalsb
and that both are sequences.- Parameters
- Raises
AssertionError – if
a
andb
are not equal.
-
static
assert_set_equal
(a, b, message=None)¶ Assert that
a
equalsb
and that both are of typeset
.- Parameters
a – a set
b – a set
message=None (str) – the message to prepend to the error message
- Raises
AssertionError – if
a
andb
are not equal.
-
static
assert_str_equal
(a, b, message=None)¶ Assert that
a
equalsb
and that both are of typestr
.- Parameters
a – a string
b – a string
message=None (str) – the message to prepend to the error message
- Raises
AssertionError – if
a
andb
are not equal.
-
static
assert_true
(expression, message=None)¶ Assert that
expression
isTrue
.- Parameters
- Raises
AssertionError – if
expression
is notTrue
-
static
assert_tuple_equal
(a, b, message=None)¶ Assert that
a
equalsb
and that both are of typetuple
.- Parameters
a – a list
b – a list
message=None (str) – the message to prepend to the error message
- Raises
AssertionError – if
a
andb
are not equal.
-
attach_file
(filename, title)¶ Attach a file to the current test that is put into the test report.
-
attach_url
(url, title)¶ Attach an url to the current test that is put into the test report. This results in an external link and should be used for huge files.
-
delay
(duration)¶ Delay execution for a given duration and print progress.
- Parameters
duration (float) – sleep duration
-
expect
(expected_value, observed_value, error_message=None)¶ Expect that
expectedValue
andobservedValue
are the same. IfexpectedValue
andobservedValue
differ anAssertionError
is raised.- Parameters
expected_value – the expected value.
observed_value – the observed value that is compared to
expectedValue
.errorMessage=None (str) – an optional message.
-
expect_step
(title, expected_value, observed_value, error_message=None)¶ Creates a test step in test report and on stdout using
title
. Expect that expectedValue and observedValue are the same. IfexpectedValue
andobservedValue
differ anAssertionError
is raised.
-
run_background
(func, name=None, stop=None, force_stop=True, join_timeout=1, args=None, kwargs=None)¶ With this method you can run another method in background.
- Parameters
func (callable) – the method to be run in background.
name=None (str) – a name for the background task. Default:
None
. If name isNone
func.__name__ is used.stop=None (callable) – callable to stop the thread
force_stop=True (bool) – force the thread to stop when leaving its scope (can impact performance)
join_timeout=1 (float) – the timeout to be used when calling
join
on the thread.args=None (tuple) – a tuple of positional arguments that are passed to the background task when it is called.
kwargs=None (dict) – a dictionary of keyword arguments that are passed to the background task when it is called.
- Returns
- a callable object is returned that stops
execution when called.
- Return type
stop-trigger (callable)
-
run_periodic
(func, period, maximum_period=None, name=None, raise_exception=True, args=None, kwargs=None)¶ With this method you can run another method periodically in background with a given period.
- Parameters
func (callable) – the method to be called periodically in background.
period (float) – the period in second.
maximum_period=None (float) – if set to a float >=
period
func
may take up tomaximum_period
time without raising an exception. A warning is printed out to stdout instead.name=None (str) – a name for the background task.
raise_exceptions=True – if set to
True
exceptions are put into the exception queue to let a test fail if a background thread fails.args=None (tuple) – a tuple of positional arguments that are passed to the background task when it is called.
kwargs=None (dict) – a dictionary of keyword arguments that are passed to the background task when it is called.
- Returns
- a callable object is returned that stops
execution when called.
- Return type
stop-trigger (callable)
-
setUp
()¶ This method is called from test runner before a test is executed to create the test-fixtures.
-
classmethod
setUpClass
()¶ This method is called for each class instance.
-
set_metadata
(name, value)¶ Set metadata for a running test.
- Parameters
- Raises
AssertionError – if
name
was already set to non accidently overwrite existing metdata
-
set_output_capture_enable
(enabled=True)¶ Enables or disables capturing of stdout and stderr for later use in test reports.
- Parameters
enabled=True (bool) – the new enabled state
-
skip_step
(message)¶ Skip the current test step.
-
skip_test
(message)¶ Skip the current test.
-
sleep
(duration)¶ Delay execution for a given duration and print progress.
- Parameters
duration (float) – sleep duration
-
step
(title, *args, **kwargs)¶ Run a test step that is documented in the test report.
-
stop_thread
(name_or_function)¶ This method stops a specific thread with the name or function described by
name_or_function
by callingjoin
with join timeout supplied byrun_periodic
orrun_background
or on it. It does not set a stop condition so the thread must end and must not include an endless loop. If the thread is still alive after the first join messages are printed to inform the user.
-
tearDown
()¶ This method is called from test runner after a test was executed regardless of it’s result.
This method shall destroy all used fixtures.
-
classmethod
tearDownClass
()¶ This method is called for each class instance.