Fixtures

Fixtures provide data, objects or services for tests. They are automatically initialized and uninitialized by the test runner.

Fixtures have a scope:

  • 'test': Fixture is initialized and uninitialized for every single test. Smallest scope.

  • 'class': Fixture is initialized when a class holding tests is run for the first time, and uninitialized after all tests of the are run.

  • 'module': Fixture is initialized once for all tests in a module.

  • 'session': Fixture is initialized for the whole session. Greatest scope.

Fixtures are implemented as python functions returning an object or python generators yielding a single object. Python generators offer set up and tear down code while python function fixture only offer set up code.

To use a fixture in a test, simply add a parameter with the fixture’s name to the test. The yielded object is passed to the test when it is executed by the test runner.

Fixtures are collected automatically by the test runner by searching the test specifiers.

To create a fixture decorate it with htf.fixture.

@htf.fixture('test')
def server() -> Generator["Server", None, None]:
    server = Server()
    yield server
    server.stop()

# pass a parameter with the fixture's name to use the fixture
def test_server(server):
    server.handle_request()

Fixtures can also depend on other fixtures. To create a dependency simply add a parameter to the fixture like you would for a test.

@htf.fixture('test')
def client(server):
    return Client()

def test_client(client, server):
    server.run()
    client.connect()

Circular dependencies are detected and raise an exception.

Requested fixtures must have the same or a greater scope than the requesting fixture. I.e. a fixture with scope ‘test’ can request fixtures with all other scopes, but a fixture with scope ‘session’ can only request other fixtures with ‘session’ scope.

You can change the fixture name by setting the name parameter.

Generally fixtures are only called when specifically requested by tests or other fixtures. For auxillary fixtures which require no direct interaction, you can set the auto parameter to True. In this case they are set up and torn down when their scope is entered or left.

Fixtures can be async, too.

@htf.fixture('test')
async def async_fixture() -> AsyncGenerator["Server", None]:
    server = await Server.create()
    yield server
    await server.stop()

Creating Fixtures

To create fixtures develop them using the htf.fixture decorator.

Using Fixtures

To use fixtures, add the sources containing the fixtures as test specifier to the htf command line utility or htf.main. The fixtures are collected automatically.

Fixture Decorator

The htf.fixture decorator is used to create fixtures.

htf.fixture(scope: str, name: str | None = None, auto: bool = False) → Callable[[...], Any]

Decorator to mark a function as a fixture.

Parameters:

  • scope – the scope determining the life-time of the fixture. Choose out of ‘session’, ‘module’, ‘class’ and ‘test’.

  • name=None – the name to be used for the fixture

  • auto=False – automatically call the fixture when its scope is entered, instead of when requested

Built-in Fixtures

htf is shipped with many built-in fixtures for different use cases.

Assertions (assertions)

assertions lets you easily do assertions. The fixture is htf.assertions. The scope is 'session'.

def test_with_assertions(assertions):
    """
    Args:
        assertions (htf.assertions): the assertions fixture
    """
    assertions.assert_true(True)

File and URL Attachments (attachments)

htf.fixtures.attachments lets you attach files and URLs to a test report. The scope is 'test'.

def test_attach_file(attachments):
    """
    Args:
        attachments (htf.fixtures.attachments): the attachments fixture
    """
    attachments.attach_file("example.jpg", "an example image attached to a report")

Attaching files as bytes

def test_attach_file_as_bytes(attachments):
    """
    Args:
        attachments (htf.fixtures.attachments): the attachments fixture
    """
    filename = "example.png"
    mime_type = "image/png"
    with open(filename, "rb") as f:
        data_bytes = f.read()
        attachments.attach_bytes("Example", data_bytes, filename, mime_type)

Attaching a URL

def test_attach_url(attachments):
    """
    Args:
        attachments (htf.fixtures.attachments): the attachments fixture
    """
    attachments.attach_url("https://hilster.io", "an example URL attached to a report")
class htf.fixtures.attachments(result: TestResult)

Attachments to the test report. Scope is 'test'.

attach_bytes(title: str, data_bytes: bytes, filename: str, mime_type: str) → None

Accepts bytes and mime type and converts them to a data url and attaches it to the result.

Parameters:

  • title – the title

  • data_bytes – the bytes to attach

  • filename – the file name

  • mime_type – MIME type of the data bytes

attach_file(filename: str, title: str) → None

Attach a file to the current test that is put into the test report.

Parameters:

  • filename – the filename to be attached

  • title – the title

attach_url(url: str, title: str) → None

Attach an url to the current test that is put into the test report. This results in an external link and should be used for huge files.

Parameters:

  • url – the url to be attached

  • title – the title

set_maximum_attachment_size(size: int) → None

Set the maximum attachment size to size bytes.

Parameters:

size – the maximum attachment size in bytes. Default ist 10 MB.

To increase the maximum attachment size simply create a new fixture with the maximum size set to the desired value:

@htf.fixture('test')
def big_attachments(attachments):
    attachments.set_maximum_attachment_size(100*1024*1024)  # 100 MB
    yield attachments

Context (context)

The htf.fixtures.context is used to share data between steps within a behavior driven test. The scope is 'test'.

class htf.fixtures.context

A context fixture used to share context data in behavior driven steps.

@htf.given("the DUT is up and running")
def up_and_running(context):
    context.dut = DUT()

@htf.when("button is pressed")
def button_is_pressed(context):
    context.dut.press_button()

@htf.then(r"the button state shall be '([a-z]+)'")
def button_state(pressed, context):
    htf.assert_equal(context.dut.get_button_state(), pressed)

Delay Functions (delay)

htf.fixtures.delay lets you delay tests. The remaining delay is printed to the terminal. htf.fixtures.delay is callable. The scope is 'session'.

def test_with_delay(delay):
    """
    Args:
        delay (htf.fixtures.delay): the delay fixture
    """
    delay(5)
class htf.fixtures.delay

Delay execution for a given duration and print progress. Scope is 'session'.

Parameters:

duration – delay duration in seconds

Sleep Functions (async) (sleep)

htf.fixtures.sleep lets you delay tests. The remaining delay is printed to the terminal. htf.fixtures.sleep is callable. The scope is 'session'.

async def test_with_sleep(desleeplay):
    """
    Args:
        sleep (htf.fixtures.sleep): the sleep fixture
    """
    await sleep(5)
class htf.fixtures.sleep

Delay execution for a given duration and print progress. Used in async context only. Scope is 'session'.

Parameters:

duration – delay duration in seconds

Interactive Testing (interaction)

The htf.fixtures.interaction fixture is documented in the Interactive Testing section.

Issue Tracking (jira)

The usage of the htf.fixtures.jira fixture is documented in the JIRA (Atlassian) section.

Append Metadata to Test Reports (metadata)

htf.fixtures.metadata lets you add metadata to tests. htf.fixtures.metadata works like a dict but also support setters and getters. The scope is 'test'.

def test_with_metadata(metadata):
    """
    Args:
        metadata (htf.fixtures.metadata): the metadata fixture
    """
    metadata.set("key", "value")
    metadata["Date"] = datetime.now()
class htf.fixtures.metadata(result: TestResult)

Set metadata in test report. Scope is 'test'.

get(name: str) → Any

Get metadata for name.

Parameters:

name – metadata name

Returns:

the value

Return type:

object

set(name: str, value: Any) → None

Set a metadata name to a value.

Parameters:

  • name – metadata name

  • value – metadata value

Access Settings (settings)

Settings lets you easily access settings in a test. The returned object is an instance of htf.Settings. The scope is 'session'.

def test_with_settings(settings):
    """
    Args:
        settings (htf.fixtures.settings): the settings fixture
    """
    print(settings.example_setting)

Access Testrun Parameters (parameters)

Parameters lets you easily access test run parameters in a test. The returned object is an instance of a read-only dict like object. The scope is 'session'.

def test_parameters(parameters):
    """
    Args:
        settings (htf.fixtures.parameters): the parameters fixture
    """
    print(parameters['name'])

   # htf --parameter name=foobar
   # htf.main(parameters=dict(name="foobar"))

Parameters can also be used to control the behavior of other fixtures.

@htf.fixture(scope="test")
def io(parameters):
    if parameters.get('io_enable', default=False):
        yield IO(parameters['io_settings'])
    else:
        yield SimulatedIO(parameters['io_settings'])
class htf.fixtures.Parameters(data: Dict[Any, Any])

Access parameters in test report. Scope is 'session'. Is instantiated automatically.

Structure Tests (step)

htf.fixtures.step lets you separate your test into steps. The scope is 'test'.

def test_with_step(step):
    """
    Args:
        step (htf.fixtures.step): the step fixture
    """
    with step("First step"):
        htf.assert_true(True)

    # steps can be nested, too
    with step("Outer step"):
        with step("Inner step"):
            htf.assert_true(True)

You can also add additional data to steps that will be put into the test report.

def test_with_additional_step_data(step):
    """
    Args:
        step (htf.fixtures.step): the step fixture
    """
    with step("First step", 1, 2, 3, step_id=100):
        htf.assert_true(True)

To skip a step run

def test_with_skipped_step(step):
    """
    Args:
        step (htf.fixtures.step): the step fixture
    """
    with step("Skipped step"):
        step.skip("This step is skipped.")
class htf.fixtures.step(result: TestResult)

Run a test step that is documented in the test report. Scope is 'test'.

Parameters:

  • title – the step’s title.

  • callable_obj=None – a callable object that is called within the step or None.

  • *args – a tuple of positional arguments for the step’s context.

  • **kwargs – a dictionary of keyword arguments for the step’s context.

skip(reason: str | None = None) → None

Skip a step.

Parameters:

reason=None – the reason for the skipped step

Raises:

htf.SkipStep –

Test Information (test)

htf.fixtures.test gives you access to information about the current test. The scope is 'test'.

def test_info(test):
    """
    Args:
        test (htf.fixtures.test): the test fixture
    """
    print("Test function name:", test.name)
    print("Test class name:", test.class_name)
    print("Test module:", test.module)
    print("Docstring:", test.doc)
    print("Tags:", test.tags)
    print("Current stdout:", test.stdout)
    print("Current stderr:", test.stderr)
    print("Current result:", test.result)  # None

The properties test.result, test.was_successful, test.was_not_successful, test.was_failure, test.was_error, test.exception_type, test.exception_value and test.exception_traceback are only available after the test was run in the context of another fixture using the test fixture.

Thus, it is possible to add actions in fixtures depending on the test’s result.

@htf.fixture(scope="test", auto=True)
def after_failed_test(test):
    yield

    if test.was_not_successful:
        print(f"The test {test.name} was not successful! It failed with {test.exception_type.__class__.__name__}: '{test.exception_value}'")


def test_failure(test):
    assert False, "This test fails"
class htf.fixtures.test(result: TestResult)

Information about the running test. Scope is 'test'.

Feature Information (feature)

htf.fixtures.feature gives you access to information about the current feature running in a behavior driven test. The scope is 'test'.

@htf.given("when I would like to see feature information")
def given_feature(feature):
    """
    Args:
        test (htf.fixtures.feature): the feature fixture
    """
    print("Feature name:", feature.name)
    print("Feature doc:", feature.doc)
    print("Feature filename:", feature.filename)
class htf.fixtures.feature(result: TestResult)

Information about the running feature in behavior driven testing. Scope is 'test'.

Scenario Information (scenario)

htf.fixtures.scenario gives you access to information about the current scenario running in a behavior driven test. The scope is 'test'.

@htf.given("when I would like to see scenario information")
def given_feature(scenario):
    """
    Args:
        test (htf.fixtures.scenario): the scenario fixture
    """
    print("Scenario name:", scenario.name)
class htf.fixtures.scenario(result: TestResult)

Information about the running scenario in behavior driven testing. Scope is 'test'.

Statistics (statistics)

htf.fixtures.statistics gives you statistics about the current test session. The scope is 'session'.

def test_statistics(statistics):
    """
    Args:
        statistics (htf.fixtures.statistics): the statistics fixture
    """
    print("Total number of tests:", statistics.number_of_tests)
    print("Tests run so far:", statistics.tests_run)
    print("Percent of tests finished:", statistics.done_percent)
    print("Successes:", statistics.successes)
    print("Failures:", statistics.failures)
    print("Errors:", statistics.errors)
    print("Skipped:", statistics.skipped)
    print("Percent of successes:", statistics.success_percent)
    print("Percent of failures:", statistics.failure_percent)
    print("Percent of errors:", statistics.error_percent)
    print("Percent of skipped:", statistics.skipped_percent)
class htf.fixtures.statistics(statistics_provider: statistics_provider)

Statistics about the test session. Scope is 'session'.

Run Threads (threads)

htf.fixtures.threads lets you start threads that are stopped automatically. You can start continuously running threads and periodically executed threads. The scope is 'test'. All started threads are automatically stopped after the starting test ends. You do not need to worry about stopping threads yourself.

def test_with_background_thread(threads):
    """
    Args:
        threads (htf.fixtures.threads): the threads fixture
    """
    def background_thread():
        while True:
            print("running in background")

    threads.run_background(background_thread)

def test_with_periodic_thread(threads):
    """
    Args:
        threads (htf.fixtures.threads): the threads fixture
    """
    def periodic_thread():
        print("running periodically in background")

    threads.run_periodic(periodic_thread, period=1.0)
    time.sleep(5)

Automatic stoppable threads are instrumented to be stoppable under all circumstances. Thus the performance may be impacted but while True loops are allowed in these threads, the development is very easy and you do not need any overhead to stop the thread. If you need more performance for background threads you have to make your threads stoppable from the outside and not use while True loops.

def test_stoppable_thread(threads):
    # create an event to stop the thread
    stop_condition = threading.Event()

    def background_thread():
        while not stop_condition.is_set():
            print("running ..")
            time.sleep(.5)

    # normally started threads that is stopped automatically with reduced performance
    threads.run_background(background_thread)

    # high performance background thread that
    stop = threads.run_background(background_thread, stop=stop_condition.set, force_stop=False)

    # wait
    time.sleep(2.0)

    # stop the thread explicitly
    stop()
class htf.fixtures.threads

Fixture for threads with scope ‘test’ that are stopped automatically and can run in background and periodically.

run_background(func: Callable[[...], Any], name: str | None = None, stop: Callable[[...], bool] | None = None, force_stop: bool = True, join_timeout: float = 1.0, args: Tuple[Any] | None = None, kwargs: Dict[str, Any] | None = None) → Callable[[], None]

With this method you can run another method in background.

Parameters:

  • func – the method to be run in background.

  • name=None – a name for the background task. Default: None. If name is None func.__name__ is used.

  • stop=None – callable to stop the thread

  • force_stop=True – force the thread to stop when leaving its scope (can impact performance)

  • join_timeout=1 – the timeout to be used when calling join on the thread.

  • args=None – a tuple of positional arguments that are passed to the background task when it is called.

  • kwargs=None – a dictionary of keyword arguments that are passed to the background task when it is called.

Returns:

a callable object is returned that stops

execution when called.

Return type:

stop-trigger

run_periodic(func: Callable[[...], Any], period: float, maximum_period: float | None = None, name: str | None = None, raise_exception: bool = True, args: Tuple[Any] | None = None, kwargs: Dict[str, Any] | None = None) → Callable[[], None]

With this method you can run another method periodically in background with a given period.

Parameters:

  • func – the method to be called periodically in background.

  • period – the period in second.

  • maximum_period=None – if set to a float >= period func may take up to maximum_period time without raising an exception. A warning is printed out to stdout instead.

  • name=None – a name for the background task.

  • raise_exceptions=True – if set to True exceptions are put into the exception queue to let a test fail if a background thread fails.

  • args=None – a tuple of positional arguments that are passed to the background task when it is called.

  • kwargs=None – a dictionary of keyword arguments that are passed to the background task when it is called.

Returns:

a callable object is returned that stops

execution when called.

Return type:

stop-trigger

stop() → None

This method stops all background threads by calling thread.join() until it is not alive anymore.

stop_thread(name_or_function: str | Callable) → int

This method stops a specific thread with the name or function described by name_or_function by calling join with join timeout supplied by run_periodic or run_background or on it. It does not set a stop condition so the thread must end and must not include an endless loop. If the thread is still alive after the first join messages are printed to inform the user.

Parameters:

name_or_function – the name or the function (reference) of the thread to be stopped.

Returns:

the number of stopped threads.

Return type:

int

Raises:

Exception – if the thread was not found

Run Session Threads (session_threads)

Like htf.fixtures.threads but with scope 'session'.

Run Processes (processes)

htf.fixtures.processes lets you start processes that are stopped automatically. You can start single background processes and process pools. It is based on multiprocessing. The scope is 'test'. All started processes are automatically stopped after the starting test ends. You do not need to worry about stopping processes yourself.

Create a single background process:

def test_process_stopping(processes):
    """
    Test a stopping background process.

    Args:
        processes (htf.fixtures.processes): the processes fixture
    """
    def process(*what):
        print("process is running ..")
        for i in range(2):
            print(" ".join(what))
            time.sleep(1.0)

    processes.run(target=process, args=("a", "b"))
    time.sleep(3.0)

Create a pool and run tasks in parallel:

def square(x):
    return x**2


def test_pool(processes):
    """
    Test processes fixture's pool.

    Args:
        processes (htf.fixtures.processes): the processes fixture
    """
    pool = processes.pool(processes=2)
    print(pool.map(square, range(10)))
    time.sleep(1.0)
class htf.fixtures.processes

Fixture for background processes with scope 'test' that are stopped automatically. Uses multiprocessing.

pool(processes: int | None = None, initializer: Callable[[...], Any] | None = None, initargs: Iterable[Any] = (), maxtasksperchild: int | None = None, timeout: float = 1.0) → multiprocessing.pool.Pool

Create a pool to run tasks in parallel.

Parameters:

  • processes=None – the number of child processes to be started.

  • initializer=None – f initializer is not None then each worker process will call initializer(*initargs) when it starts.

  • initargs=() – arguments for initializer.

  • maxtasksperchild=None – the maximum number of tasks per child process.

  • timeout=1.0 – the timeout for joining the pool process.

Returns:

the pool

run(group: None = None, target: Callable[[...], Any] | None = None, name: str | None = None, args: Iterable[Any] = (), kwargs: Dict[str, Any] | None = None, daemon: bool | None = None, timeout: float = 1.0) → Process

Run a background process.

Parameters:

  • group=None – the group

  • target=None – A callable to be run in background. Must be pickleable.

  • name=None – the name of the background process.

  • args=() – the positional arguments passed to the background process callable.

  • kwargs=None – the keyword arguments passed to the background process callable.

  • daemon=None – if set to True the child process is terminated if the parent process ends.

  • timeout=1.0 – the timeout for joining, terminating and killing the process.

Returns:

the process

Run Session Processes (session_processes)

Like htf.fixtures.processes but with scope 'session'.

Type Hints for fixtures

To be able to use code completion in you IDE add type hints or docstrings to your tests so that the IDE is able to find the right fixture.

# type hint:
def test_step(step : htf.fixtures.step):
    with step("A step"):
        pass

# docstring:
def test_step(step):
    """
    This test ...

    Args:
        step (htf.fixtures.step): the step fixture
    """
    with step("A step"):
        pass

Example: Cache Fixture

Assume you want to cache data over multiple test runs to reduce the runtime. You could create a cache fixture.

import json
# cache fixture
@htf.fixture('session')
def cache():
    # set up
    filename = "cache.json"
    try:
        with open(filename, "r") as fp:
            data = json.load(fp)
    except FileNotFoundError:
        with open(filename, "w") as fp:
            json.dump({}, fp)
            print("Created cache file '{}'".format(filename))
        data = {}

    yield data

    # tear down
    with open(filename, "w") as fp:
        json.dump(data, fp)
        print("Wrote cache file '{}'".format(filename))

# test using cache fixture
def test_cache(cache):
    try:
        # count up test run number
        cache["#"] += 1
    except KeyError:
        cache["#"] = 1

    print("This is test run {}".format(cache["#"]))

Tagging Fixtures

Fixtures can also be tagged. Thus it is easy to select a set of fixture with a tag expression.

For example you can create fixtures for simulated and hardware in the loop devices with different tags and select a group to run the same tests with and without hardware.

htf support -F <expression> and --fixture-tags=<expression>. Unlike selecting tests by tags all untagged fixtures are included in the result. So if you use fixture tags they will only be used to filter actually tagged fixtures.

Example:

class Device:
    pass


@htf.tags("simulation")
@htf.fixture('test', name="device")
def device_simulation():
    print("simulated device")
    yield Device()


@htf.tags("hardware")
@htf.fixture('test', name="device")
def device_hardware():
    print("hardware device")
    yield Device()


def test_device(device):
    pass


if __name__ == "__main__":
    htf.main(fixture_tags="hardware")
    # $ htf --fixture-tags=hardware