htf — Command line utility

The HILSTER Testing Framework comes with a command line utility called htf.

htf can be used to run tests, open docs and view the included open-source licenses.

Running tests

To run all tests within the current folder simply run

htf

By default htf creates an HTML-testreport called testreport.html in the current folder.

Specifying tests

htf may be followed by run and <tests>. You can also omit run and only supply test-specifiers, eg.

htf run <tests>

respectively

htf <tests>

<tests> is a list of test-specifiers. They are separated by whitespaces.

A test-specifier may be a python module or a feature file, a globbing-expression for python modules (e.g. test_*.py), a globbing expression for feature files (e.g. features/*.feature), a folder or an import string. Test-specifiers can be mixed.

If a test-specifier is a python module the file is imported and all testable objects are collected into a test suite that is used for the test run.

htf test_1.py

If a test-specifier ist a globbing-expression the expression is evaluated and all matching files are imported.

htf test_\*.py

Note: the \* lets not Bash (or any other shell) evaluate it and passes it to htf.

If a test-specifier is a folder the folder is searched recursively and all python modules found within (see File pattern) are imported and all testable objects are collected into a test suite. Subsequently the test suite is executed.

Consider the folder tests, which contains python modules with unit tests. To execute all test cases in the folder tests, simply run

htf tests

In case tests is an importable package or module, located in the python search path, the names might collide. In this case, simply use

htf tests/

If a test-specifier is an import string, this may be a package, a module, a class (inheriting from htf.TestCase) or a method.

If a test-specifier is a package or a module, it is imported and containing tests are executed.

htf tests  # Example for a python package containing tests
htf tests.test_example  # Example for a python module containing tests

If a class is used it should be a testcase inheriting from htf.TestCase. This testcase is used in a test suite and all tests are run.

htf tests.test_example.ExampleTestCase # Example for a class containing tests

If a method is used it should be a method within a test case.

htf tests.test_example.ExampleTestCase.test_one # Example for a test method

Running multiple tests

To run multiple tests within a test run simply add more test-specifiers at the end.

htf test1 test2 test3

Different test-specifiers can also be mixed

htf tests/ test1.Test1TestCase test2.py

Test-title

To specify the test’s title add the -T or --title option followed by a string that will be used as the title. If the title contains whitespaces, make sure to put quotes around it.

htf --title="This is my first test" tests/

HTML-report

By default htf generates an HTML-report called testreport.html. To specify an HTML-report add -H or --html-report.

htf --html-report="tests.html" tests/

This option may be repeated to generate multiple HTML-reports.

htf --html-report="tests1.html" --html-report="tests2.html" tests/

To open all HTML-reports use -o or --open-report.

htf --html-report="tests.html" -o tests/

JUnit-XML-report

htf is also able to generate JUnit-XML-compatible XML-reports. The reports can be evaluated by different tools, eg. Jenkins.

To specify an XML-report add -x or --junit-xml-report.

htf --junit-xml-report=results.xml tests/

This option may also be repeated.

JSON-report

htf is also able to generate JSON-reports.

To specify a JSON-report add -j or --json-report.

htf --json-report=results.json tests/

This option may also be repeated.

YAML-report

htf is also able to generate YAML-reports.

To specify a YAML-report add -Y or --yaml-report.

htf --yaml-report=results.yaml tests/

This option may also be repeated.

Report Server

To send test results to a test report server (hre) use -r or --report-server.

htf -r https://reports.host/project/token
htf --report-server=https://reports.host/project/token

To disable SSL-verfification you can set the environment variable HTF_INSECURE to any value.

HTF_INSECURE=1 htf -r https://reports.host/project/token  # Linux
set HTF_INSECURE=1  # Windows
htf -r https://reports.host/project/token  # Windows

Draft Reports

To create draft reports after each test use -1 or --create-drafts.

htf -1

or

htf --create-drafts

Minimized Report Size

To minimize the report size use the -M or --minimize-report-size switch. When used for example captured stdout and stderr is only kept for failed tests. By default the report size is not minimized.

htf -M tests/

Shuffle Tests

To shuffle tests before running add -S` or ``--shuffle.

htf --shuffle tests/

To set the shuffle seed use --shuffle-seed=<int>|last. If 'last' is used the seed is loaded from 'shuffle_seed.txt'. If the file cannot be found one seed is created.

After a seed is created it is stored to be used next time. Using the seed the shuffle result can be reproduced.

htf --shuffle --shuffle-seed=12345 tests/
htf --shuffle --shuffle-seed=last tests/

Filename-templates

Filenames for reports can include prepopulated templates.

Available templates:

{{title}} - the title as a filename

{{date}} - the current date "%Y-%m-%d_%H-%M-%S"

{{hostname}}, {{host}} or {{node}} - the hostname

{{htf_version}} or {{version}} - the htf version, eg. htf_1.0.0

{{python_version}} - the python version, eg. Python_3.6.1

To generate an HTML-report containing the current node and the current date use

htf -H test_{{host}}_{{date}}.html tests/

Fail fast

If -f or --fail-fast is used the test run ends after the first failing test.

File pattern

If a test-specifier is a folder the default pattern used for file discovery is test*.py. The pattern can be changed using the -p or --pattern options.

htf -p \*.py tests/

Wildcards (*) are supported. Depending on your shell, escaping must be applied.

Tagging

To select tests using a logical expression use the -t or --tags option followed by an expression.

htf --tags='foo|bar'

To list all available tags use the tags commands.

htf tags /path/to/test

To use tags for fixtures use -F``or ``--fixture-tags in addition.

htf -F hardware
htf --fixture-tags=hardware

Metadata

To add metadata about the test run use the -m or --metadata option followed by a comma-separated list of key-value pairs.

These options can also be repeated.

This feature can for example be used to add information about a tested firmware version, etc.

htf -m key1=value1,key2=value2 --metadata key3=value3  # etc.

The metadata will occur in the test reports.

Overwriting Settings

You can overwrite settings and even hierarchical settings from command line using -s <name>=<value> or --setting <name>=<value>.

Hierarchies in settings can be realized with a . e.g. -s top.center.bottom=1.

For example the following call

htf -s a=1,b=2 -s top.sub=3

leads to settings looking like

{
    'a': '1',
    'b': '2',
    'top': {
        'sub': '3'
    }
}

Please note that all supplied settings are strings and you have to care about the casting when these will be used.

This option support to do manual experiments, etc.

Parameters

You can add parameters to test runs from command line to control the behavior of tests and fixtures. Add -P <name>=<value> or --parameter <name>=<value> to the call.

The parameters are available via the htf.fixtures.Parameters fixture.

htf -P name1=value1,name2=value2 --parameter=name3=value3

The parameters will also appear in the test result metadata.

Bash-completion

htf supports bash-completion which provides automatic command completion for the bash shell.

Bash-completion completes the first positional commands (run, docs and licenses, etc.) but also expands test-specifiers (eg. file names, packages, modules, classes and methods) and other parameters like --html-report, --metadata, etc.

Using the bash-completion makes it easy to specify tests (without typing-errors).

Let’s assume you a have an importable python package called tests. Two test-modules called test_1.py and test_2.py are located within tests.

Both contain a test-case called TestCase1 and TestCase2, respectively.

To easily run tests.test_1.TestCase1, type

htf te<TAB>
    tests    tests.
htf tests.<TAB>
htf tests.test_<TAB>
    tests.test_1    tests.test_1.    tests.test_2    tests.test_2.
htf tests.test_1.<TAB>
htf tests.test_1.TestCa<TAB>
htf tests.test_1.TestCase<TAB>
    tests.test_1.TestCase    tests.test_1.TestCase.
htf tests.test_1.TestCase<ENTER>
# ...

Enable Bash-completion

To enable bash-completion type

eval $(htf complete)

To persistently enable bash completion, add the command to your ~.bashrc. alternatively put it into a new file called /etc/bash_completion.d/htf, eg.

echo "eval $(htf complete)" >> ~/.bashrc
. ~/.bashrc

or

echo "eval $(htf complete)" > /etc/bash_completion.d/htf
. /etc/bash_completion.d/htf

To check if bash-completion is enabled type

complete -p | grep htf

Open docs

To open the current docs in your webbrowser, type

htf docs

View licenses

To view the included and used open-source licenses, type

htf licenses

or

htf licenses --compact

to print the compact form.

To output the contents in Markdown format, type

htf licenses --markdown

or

htf licenses --markdown --compact

to print the compact form.

Show version

To show the htf version, type

htf version