Test Reports

HILSTER Testing Framework includes test reports for HTML, XML, YAML and JSON.

HTML Test Report

The HTML test report is a monolithic Javascript application that does not need to load any external data.

When loading the test report, a progress bar appears, and the percentage of loaded tests is shown. The HTML test report usually loads very fast.

When the test report has finished loading, a responsive interface is shown, allowing you to interact with the test report.

Overview

The visualisation is hierarchical: The inner ring represents the the four results types success, error, failure and skipped, as shown in the sidebar. The next ring shows more detailed information, such as exception types for failures and errors, and skip reasons for skipped tests. These information get even more detailed in the outer ring, making it possible to easily identify concrete exception strings within respective exception types.

Hovering over the diagram reveals additional information, such as hierarchy paths and statistics.

Clicking a section of the diagram searches the test report for the respective result type/exception type/exception string.

Test results

The main content is located to the right of the sidebar, containing all test results. Clicking on a result expands the respective test and shows further information. By default all test results are collapsed.

Every test result contains different test properties, such as result, traceback, metadata, documentation, tags, dates, attachments, captured stdout and captured stderr. To improve performance, long output (stdout and stderr) is not loaded by default. Clicking on show more loads the respective output.

File attachments

The HTML test report supports file attachments that you can attach to a test using htf.TestCase.attach_file (or htf.TestCase.attachFile).

File attachments are not supported by all browsers. For more information see Browser Limitations.

URL attachments

You can also attach URLs by using htf.TestCase.attach_url (or htf.TestCase.attachUrl) in your test.

Searching

Clicking on the magnifier icon on the top opens a search interface. The HTML test report offers three different search methods: Default search, global search and special pattern search.

Note that only selected categories get searched. In order to search all testresults, you would need to select All in the sidebar.

The search interface can be closed by clicking on the small x to the right of the search bar.

Goto

When pressing g, an input field appears next to the search bar, allowing the user to jump to a specific test. Entering the test run number (which can be shown by pressing n ) of the respective test and pressing Enter expands and focuses the test.

Bookmarking

Right-clicking a test bookmarks the test for easier access. A bookmark-section is shown in the sidebar when there are active bookmarks.

Draft

Even when not all tests are run yet, the test report can be interacted with. This is indicated by a red draft sign in the sidebar.

Printing

While in test run overview, a print icon is shown in the upper right corner. Clicking it will render a print optimized version of the test report, including an overview diagram, the test run metadata and a signature page.

In print view, every test will get expanded and printed on a dedicated page for easier readability.

Dark Mode

For easier readability, the testreport color scheme can be changed to a darker variant by pressing D. Pressing the hotkey again switches back to the light theme.

Hotkeys

The HTML test report supports the following hotkeys:

Key

Description

h

Show help (any click or key but h hides it again)

esc

Reset view: hide help, hide search box, show all results

a

Collapse all expanded test results

o

Show test run overview diagram

m

Show test run metadata

a

Show all results

s

Toggle successes

f

Toggle failures

e

Toggle errors

S

Toggle skipped

F

Show search interface

R

Show search interface and toggle regex parameter

g

Show goto interface to jump to a specific test

n

Toggle test run numbers

b

Toggle sidebar

l

Show open source licenses

p

Show printable version of test report

D

Activate Dark Mode

The only hotkey you have to remember is h though.

Browser Limitations

Most modern browsers, such as Firefox, Chrome and Microsoft Edge (Version 80+), support the whole functionality of the test report.

File attachments are realized using data-urls.

Some Microsoft-browsers do not support them at all. Chrome and Chromium have security features in place, but support an acceptable workaround.

The following browser limitations are known:

Browser

Limitations

IE 11 and older

  • Not officially supported.

Edge (Version 78 and earlier)

  • Only image attachments can be downloaded by right clicking and choosing save image as. Downloading other types of attachments is not possible, since older versions of Edge do not fully support downloading data-urls. Everything else works well.

Chrome/Chromium/ Edge (Version 80+)

  • Attachments cannot be viewed or downloaded with a single click on it. As a workaround you can use the context menu and use save as or open in another tab.

Your browser must have Javascript enabled to view the test report. There are some plugins like NoScript that will likely cause problems. Also old versions of Internet Explorer do not load Javascript by default.

JUnit XML Test Report

HILSTER Testing Framework creates JUnit XML test reports that can be used in other tools like Continuous Integration systems, etc.

The JUnit XML format does not include all available data htf offers.

JSON Test Report

The JSON test report includes all test results with all data to be used in your tools.

YAML Test Report

The YAML test report includes all test results with all data to be used in your tools.

IBM Rational DOORS Test Report

The IBM Rational DOORS Test Report is a special test report that creates a DXL script to be imported into IBM Rational DOORS. Thus your test results can easily be imported to IBM Rational DOORS.

The IBM Rational DOORS Test Report works in conjunction with the @meets_doors_requirements decorator that attaches the tested requirements to individual tests.

@meets_doors_requirements("doors://host:port/path-without-id-",
                   "REQ_1", "REQ_2")
def test_with_useful_name():
    pass

The IBM Rational DOORS Test Report can only be used within Python code because it needs configuration parameters.

from __future__ import absolute_import, division, print_function, unicode_literals
import htf

@htf.meets_doors_requirements("REQ_1", "REQ_2", "REQ_3", "TEST_1")
def test_with_doors_links():
    pass

htf.main(title="DOORS Test Report",
    extra_report=[
             DOORSTestReport(filename="doors.dxl",
                             report_module="/path/to/report/module",
                             requirements_modules={
                                 "REQ_.*" : "/path/to/requirements_module",
                                 "TEST_.*" : "/path/to/test_specification",
                             },
                             links_modules={
                                 "REQ_.*" : "/path/to/links/to/requirements_module",
                                 "TEST_.*" : "/path/to/links/to/test_specification",
                                 }
                             )
             ],
    )
class htf.DOORSTestReport(filename, report_module, requirements_modules, links_modules=None, render_meta_data=True, render_steps=True)

DOORSTestReport creates a DOORS test report. Links are automatically created from the created DOORS module to the fulfilled requirements.

To state the fulfilled requirements the meets_doors_requirements decorator should be used.

Initialize a DOORSTestReport instance.

Parameters
  • filename (str) – the filename for the DXL test report.

  • report_module (str) – the DOORS path to the report module.

  • requirements_modules (dict) – maps links to DOORS modules. The keys are regular expressions and the values are the desired target modules.

  • links_modules={".*" – “DOORS Links”} (dict): maps requirements to desired link modules. The keys are regular expressions and the values are the desired link modules.

  • render_meta_data=True (bool) – if set to True metadata is rendered in DOORS

  • render_steps=True (bool) – if set to True steps are rendered in DOORS

render(data)

Generates and stores a DXL encoded TestReport.

Parameters

data (dict) – test result data