Test Reports

HILSTER Testing Framework includes test reports for HTML, XML, YAML and JSON.

HTML Test Report

The HTML test report is a monolithic Javascript application that does not need to load any external data.

When loading the test report, a progress bar appears, and the percentage of loaded tests is shown. The HTML test report usually loads very fast.

When the test report has finished loading, a responsive interface is shown, allowing you to interact with the test report.

Navigation

The navigation is located in the sidebar to the left. The first two entries, Overview and Metadata, are helpful for getting an initial impression of the test run. When the test report is loaded, Overview is shown by default, visualizing and summarizing the test results in an easy to understand way. Metadata shows test run specific information.

The remaining entries are used for navigating through the test results, and filtering which test results to display. Clicking All shows all test results. When selecting result types (Success, Error, Failure and Skipped), the test results get filtered accordingly. It is possible to select multiple result types at once. The number of tests within the respective result types are shown next to the filters.

Note that clicking All/pressing esc resets any applied filters.

Information about the host, the platform the tests were run on and licensing information are shown at the bottom of the sidebar on the left.

The sidebar can be hidden by clicking on the hamburger icon on the top, next to the logo.

Test title and test date are shown in the top right corner.

Overview

The visualisation is hierarchical: The inner ring represents the the four results types success, error, failure and skipped, as shown in the sidebar. The next ring shows more detailed information, such as exception types for failures and errors, and skip reasons for skipped tests. These information get even more detailed in the outer ring, making it possible to easily identify concrete exception strings within respective exception types.

Hovering over the diagram reveals additional information, such as hierarchy paths and statistics.

Clicking a section of the diagram searches the test report for the respective result type/exception type/exception string.

Test results

The main content is located to the right of the sidebar, containing all test results. Clicking on a result expands the respective test and shows further information. By default all test results are collapsed.

Every test result contains different test properties, such as result, traceback, metadata, documentation, tags, dates, attachments, captured stdout and captured stderr. To improve performance, long output (stdout and stderr) is not loaded by default. Clicking on show more loads the respective output.

File attachments

The HTML test report supports file attachments that you can attach to a test using htf.TestCase.attach_file (or htf.TestCase.attachFile).

File attachments are not supported by all browsers. For more information see Browser Limitations.

URL attachments

You can also attach URLs by using htf.TestCase.attach_url (or htf.TestCase.attachUrl) in your test.

Searching

Clicking on the magnifier icon on the top opens a search interface. The HTML test report offers three different search methods: Default search, global search and special pattern search.

Note that only selected categories get searched. In order to search all testresults, you would need to select All in the sidebar.

The search interface can be closed by clicking on the small x to the right of the search bar.

Default search

By default, only test titles get searched. Typing in the search bar instantly triggers the search.

Toggling the regex slider, located to the right side of the search bar, allows the user to perform a search using regular expressions.

The test report also offers two additional advanced searching methods, global search and special pattern search.

Global search

Global search requires the search term to be prefixed with global:, and allows the user to search not only titles, but also in any test properties within the test results, such as Dates, Traceback or Tags.

For example to search for tests that include foo, the user would use global:foo.

To search for tests that include foo or bar, you would use global:foo|bar, combining global search with regular expressions.

Special pattern search

Special pattern search allows to search in specific test properties, such as Tags, Dates or Traceback, by using the name of the respective property as prefix in lowercase. For example, to search for tests tagged with tag_1, one would search for tags:tag_1. Accordingly, searching for tests which were started on 2020-01-09, one would search for dates:2020-01-09.

This can also be used to filter for all tests containing a specific property. For example, to only show tests containing Tags, one would search for tags:.

Goto

When pressing g, an input field appears next to the search bar, allowing the user to jump to a specific test. Entering the test run number (which can be shown by pressing n ) of the respective test and pressing Enter expands and focuses the test.

Bookmarking

Right-clicking a test bookmarks the test for easier access. A bookmark-section is shown in the sidebar when there are active bookmarks.

Draft

Even when not all tests are run yet, the test report can be interacted with. This is indicated by a red draft sign in the sidebar.

Printing

While in test run overview, a print icon is shown in the upper right corner. Clicking it will render a print optimized version of the test report, including an overview diagram, the test run metadata and a signature page.

In print view, every test will get expanded and printed on a dedicated page for easier readability.

Dark Mode

For easier readability, the testreport color scheme can be changed to a darker variant by pressing D. Pressing the hotkey again switches back to the light theme.

Hotkeys

The HTML test report supports the following hotkeys:

Key	Description
`h`	Show help (any click or key but `h` hides it again)
`esc`	Reset view: hide help, hide search box, show all results
`a`	Collapse all expanded test results
`o`	Show test run overview diagram
`m`	Show test run metadata
`a`	Show all results
`s`	Toggle successes
`f`	Toggle failures
`e`	Toggle errors
`S`	Toggle skipped
`F`	Show search interface
`R`	Show search interface and toggle regex parameter
`g`	Show goto interface to jump to a specific test
`n`	Toggle test run numbers
`b`	Toggle sidebar
`l`	Show open source licenses
`p`	Show printable version of test report
`D`	Activate Dark Mode

The only hotkey you have to remember is h though.

Browser Limitations

Most modern browsers, such as Firefox, Chrome and Microsoft Edge (Version 80+), support the whole functionality of the test report.

File attachments are realized using data-urls.

Some Microsoft-browsers do not support them at all. Chrome and Chromium have security features in place, but support an acceptable workaround.

The following browser limitations are known:

Browser	Limitations
`IE 11 and older`	Not officially supported.
`Edge (Version 78 and` `earlier)`	Only image attachments can be downloaded by right clicking and choosing `save image as`. Downloading other types of attachments is not possible, since older versions of Edge do not fully support downloading data-urls. Everything else works well.
`Chrome/Chromium/` `Edge (Version 80+)`	Attachments cannot be viewed or downloaded with a single click on it. As a workaround you can use the context menu and use `save as` or `open in another tab`.

Your browser must have Javascript enabled to view the test report. There are some plugins like NoScript that will likely cause problems. Also old versions of Internet Explorer do not load Javascript by default.

JUnit XML Test Report

HILSTER Testing Framework creates JUnit XML test reports that can be used in other tools like Continuous Integration systems, etc.

The JUnit XML format does not include all available data htf offers.

JSON Test Report

The JSON test report includes all test results with all data to be used in your tools.

YAML Test Report

The YAML test report includes all test results with all data to be used in your tools.

IBM® Engineering Requirements Management DOORS Test Report

The IBM® Engineering Requirements Management DOORS Test Report is a special test report that creates a DXL script to be imported into DOORS. Thus your test results can easily be imported to DOORS.

The DOORS Test Report works in conjunction with the @meets_doors_requirements decorator that attaches the tested requirements to individual tests.

@meets_doors_requirements("doors://host:port/path-without-id-",
                   "REQ_1", "REQ_2")
def test_with_useful_name():
    pass

The IBM Rational DOORS Test Report can only be used within Python code because it needs configuration parameters.

import htf

@htf.meets_doors_requirements("REQ_1", "REQ_2", "REQ_3", "TEST_1")
def test_with_doors_links():
    pass

htf.main(title="DOORS Test Report",
    extra_report=[
             DOORSTestReport(filename="doors.dxl",
                             report_module="/path/to/report/module",
                             requirements_modules={
                                 "REQ_.*" : "/path/to/requirements_module",
                                 "TEST_.*" : "/path/to/test_specification",
                             },
                             links_modules={
                                 "REQ_.*" : "/path/to/links/to/requirements_module",
                                 "TEST_.*" : "/path/to/links/to/test_specification",
                                 }
                             )
             ],
    )

class htf.DOORSTestReport(filename: str, report_module: str, requirements_modules: Dict[str, str], links_modules: Dict[str, str] | None = None, render_meta_data: bool = True, render_steps: bool = True)

DOORSTestReport creates a DOORS test report. Links are automatically created from the created DOORS module to the fulfilled requirements.

To state the fulfilled requirements the meets_doors_requirements decorator should be used.

Initialize a DOORSTestReport instance.

Parameters:

filename – the filename for the DXL test report.
report_module – the DOORS path to the report module.
requirements_modules – maps links to DOORS modules. The keys are regular expressions and the values are the desired target modules.
links_modules={".*" – “DOORS Links”}: maps requirements to desired link modules. The keys are regular expressions and the values are the desired link modules.
render_meta_data=True – if set to True metadata is rendered in DOORS
render_steps=True – if set to True steps are rendered in DOORS

render(data: Dict[str, Any]) → None

Generates and stores a DXL encoded TestReport.

Parameters:: data – test result data