Core Functionalities

This chapter demonstrates the core of the HILSTER Testing Framework.

Foreword

All test reports are best viewed with a modern browser e.g. Google Chrome or Mozilla Firefox. You should expect some restrictions using Microsoft Internet Explorer.

Test Results

The Test Results demo is located in core/test_results.py.

# -*- coding: utf-8 -*-
#
# Copyright (c) 2020, HILSTER Testing Solutions GmbH, Germany - https://hilster.io
# All rights reserved.
#

import htf


def test_assert():
    """
    This test shows the easiest way to do assertions.
    """
    assert True


def test_success(assertions):
    """
    This test succeeds.

    Args:
        assertions (htf.fixtures.assertions): the assertions fixture
    """
    assertions.assert_true(True)


def test_failure(assertions):
    """
    This test fails.

    Args:
        assertions (htf.fixtures.assertions): the assertions fixture
    """
    assertions.assert_true(False)


def test_error():
    """
    This test raises an exception

    Args:
        assertions (htf.fixtures.assertions): the assertions fixture
    """
    raise Exception("Error!")


@htf.skip_if(True, "this test is skipped")
def test_skip_if():
    """
    This test is skipped
    """
    pass


if __name__ == '__main__':
    htf.main()

To execute the test, run

htf -o core/test_results.py

from the command line or run it from PyCharm. If you want to change the number of individual tests, you can do so by modifying the variable values at the top of the script.

This will run all tests in the core/test_results.py script and create an HTML test report named testreport.html, which should be opened by default (because of the -o parameter).

Documentation for the HTML test report can be found here.

Docstrings

Tests can be documented using Python’s docstrings. htf supports docstrings in reStructured text format. This way the user can add some markup.

More information about reStructured text markup can be found here.

Docstrings are converted into html for the HTML test report, too.

The demo is located in core/test_docstrings.py.

# -*- coding: utf-8 -*-
#
# Copyright (c) 2020, HILSTER Testing Solutions GmbH, Germany - https://hilster.io
# All rights reserved.
#

import htf


def test_docstrings():
    """
    Section
    =======

    Docstrings support sections.

    Bullet lists
    ============

    - This is item 1
    - This is item 2

    Tables
    ======

    Grid table:

    +------------+------------+-----------+
    | Header 1   | Header 2   | Header 3  |
    +============+============+===========+
    | body row 1 | column 2   | column 3  |
    +------------+------------+-----------+
    | body row 2 | Cells may span columns.|
    +------------+------------+-----------+
    | body row 3 | Cells may  | - Cells   |
    +------------+ span rows. | - contain |
    | body row 4 |            | - blocks. |
    +------------+------------+-----------+

    Simple table:

    =====  =====  ======
       Inputs     Output
    ------------  ------
      A      B    A or B
    =====  =====  ======
    False  False  False
    True   False  True
    False  True   True
    True   True   True
    =====  =====  ======

    A transition marker is a horizontal line
    of 4 or more repeated punctuation
    characters.

    ------------

    A transition should not begin or end a
    section or document, nor should two
    transitions be immediately adjacent.
"""


if __name__ == '__main__':
    htf.main()

To execute the docstrings demo from the command line, run

htf -o core/test_docstrings.py

Steps

Tests can be split up into smaller chunks with a mechanism called steps. The test steps show up in the test reports and help to give tests a better structure and make debugging easier.

The steps demo is located in core/test_steps.py.

# -*- coding: utf-8 -*-
#
# Copyright (c) 2020, HILSTER Testing Solutions GmbH, Germany - https://hilster.io
# All rights reserved.
#

import htf


def test_steps(step):
    """
    This test shows how to use steps to structure your tests.

    Args:
         step (htf.fixtures.step): the step fixture
    """
    with step("Initialization"):
        print("init.. ")

    with step("Body"):
        print("body.. ")

        with step("Nested step"):
            print("This is printed in a nested step")

    with step("Cleanup"):
        print("cleanup.. ")


if __name__ == '__main__':
    htf.main()

To execute the steps demo from the command line, run

htf -o core/test_steps.py

Attachments

Tests support file and url attachments. This is a great way to attach test artifact to test reports.

The demo is located in core/test_attachments.py.

# -*- coding: utf-8 -*-
#
# Copyright (c) 2020, HILSTER Testing Solutions GmbH, Germany - https://hilster.io
# All rights reserved.
#

import htf
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D


def test_file_attachment(step, attachments):
    """
    This test plots a graph and attaches it to the test report.
    It also shows the usage of tags.

    Args:
        step (htf.fixtures.step): the step fixture
        attachments (htf.fixtures.attachments): the attachments fixture
    """
    with step("Create plot"):
        def lorenz(x, y, z, s=10, r=28, b=2.667):
            x_dot = s * (y - x)
            y_dot = r * x - y - x * z
            z_dot = x * y - b * z
            return x_dot, y_dot, z_dot

        dt = 0.01
        step_count = 10000

        # Need one more for the initial values
        xs = np.empty((step_count + 1,))
        ys = np.empty((step_count + 1,))
        zs = np.empty((step_count + 1,))

        # Setting initial values
        xs[0], ys[0], zs[0] = (0., 1., 1.05)

        # Stepping through "time".
        for i in range(step_count):
            # Derivatives of the X, Y, Z state
            x_dot, y_dot, z_dot = lorenz(xs[i], ys[i], zs[i])
            xs[i + 1] = xs[i] + (x_dot * dt)
            ys[i + 1] = ys[i] + (y_dot * dt)
            zs[i + 1] = zs[i] + (z_dot * dt)

        fig = plt.figure()
        # ax = fig.gca(projection='3d')
        ax = Axes3D(fig)

        ax.plot(xs, ys, zs, lw=0.5)
        ax.set_xlabel("X Axis")
        ax.set_ylabel("Y Axis")
        ax.set_zlabel("Z Axis")
        ax.set_title("Lorenz Attractor")

        # plt.show()
        plt.savefig("figure.png")

        with step("Attach plot"):
            attachments.attach_file("figure.png", "Lorenz Attractor")


def test_url_attachment(attachments):
    """
    This test attaches a URL to a test results.

    Args:
        attachments (htf.fixtures.attachments): the attachments fixture
    """
    attachments.attach_url("https://hilster.io", "HILSTER")


if __name__ == '__main__':
    htf.main()

To execute the attachments demo from the command line, run

htf -o core/test_attachments.py

Tags

htf supports tags to easily tag tests and test cases.

The tags demo is located in core/test_tags.py.

# -*- coding: utf-8 -*-
#
# Copyright (c) 2020, HILSTER Testing Solutions GmbH, Germany - https://hilster.io
# All rights reserved.
#

import htf


@htf.tags("a")
def test_method_a():
    pass


@htf.tags("b")
def test_method_b():
    pass


@htf.tags("c")
def test_method_c():
    pass


if __name__ == '__main__':
    htf.main(tags="a|b")

The tags selector is a logical expression.

To execute the tags demo from the command line, run

htf -o core/test_tags.py --tags="a|b"

for example to select all tests tagged with a or b. You can also try different tag selectors.

More information on tags can be found in Tagging, Tagging for htf and Tagging for htf.main().

Metadata

Tests can be expanded using metadata. htf supports metadata for test cases and for the test run.

More information about metadata can be found in Metadata for htf and Metadata for htf.main() and in the Keywords section in the docs.

The metadata demo is located in core/test_metadata.py.

# -*- coding: utf-8 -*-
#
# Copyright (c) 2020, HILSTER Testing Solutions GmbH, Germany - https://hilster.io
# All rights reserved.
#

import htf

Author = htf.Author("$Author: Chuck Norris $")
Source = htf.Source("$Source: /path/to/test_script.py $")
ReviewDate = htf.MetaData("01.01.1970")


def test_with_metadata(metadata):
    """
    This test shows how to use metadata.

    Args:
        metadata (htf.fixtures.metadata): the metadata fixture
    """
    metadata.set('key', 'value')
    metadata['key2'] = 'value2'


if __name__ == '__main__':
    # you can also add test run metadata
    metadata = dict(date="today", time="now", foo="bar")
    htf.main(title="Metadata", metadata=metadata)

To execute the test with metadata from the command line, run

htf -o core/test_metadata.py -mdate=today,time=now,foo=bar

All metadata is included in the test reports.

Fixtures

Fixtures are resources needed by the tests. They can supplement for example communication features, data sources or settings.

Scope:

Fixtures do not need to be attached to individual tests. Their lifetime can range from an individual test to an entire test session.

Usage:

Fixtures have a name and tests with a parameter of that name will be started with the fixtures. If a fixture depends on another fixture, you can pass the needed fixture as a parameter.

Implementation:

Fixtures implemented as Python Generators yielding relevant objects. The generator is decorated with @htf.fixture(scope).

You can find more information in the Fixtures documentation.

The fixture demo is located in core/test_fixtures.py.

import htf


@htf.fixture('session')
def data_storage():
    data = []
    yield data


@htf.fixture('test')
def logger():
    print("Test started.")
    yield None
    print("Test stopped.")


@htf.test
@htf.data(range(10))
def new_style_test(i, data_storage, logger):
    data_storage.append(i)
    print(data_storage)
    assert True


if __name__ == '__main__':
    htf.main()

To execute the fixture demo from the command line, run

htf -o core/test_fixtures.py

Threads

A common way to provide tests with the means to communicate to a device via a serial port or a network socket is by running the networking code concurrently to the test in a thread. htf offers a wrapper around threads that ties the life time of the thread to the life time of a test.

The threads demo is located in core/test_threads.py.

# -*- coding: utf-8 -*-
#
# Copyright (c) 2020, HILSTER Testing Solutions GmbH, Germany - https://hilster.io
# All rights reserved.
#

import htf
import time


def test_background(threads):
    """
    This test shows how to use threads.

    Args:
         threads (htf.fixtures.threads): the threads fixture
    """
    def bg():
        print("Running in background!")

    threads.run_background(bg)
    time.sleep(1.0)


def test_periodic(threads):
    """
    This test shows how to use threads to run a function periodically.

    Args:
         threads (htf.fixtures.threads): the threads fixture
    """
    def p():
        print("Periodically called!")

    threads.run_periodic(p, period=1.0)
    time.sleep(3.0)


if __name__ == '__main__':
    htf.main()

To execute the test from command line, run

htf -o core/test_threads.py

Run All Tests

To execute all tests from core, run

htf -o core

This will also create a test report named testreport.html which contains the results of all tests in the core demo.