<!-- HILSTER htf 4.3.1 -->
**htf 4.3.1** · [HTML](../testreports.html) · [Complete PDF manual](../_downloads/htf-4.3.1.pdf)

<a id="test-reports"></a>

# Test Reports

HILSTER Testing Framework includes test reports for HTML, XML, YAML and JSON.

<a id="html-test-report"></a>

## HTML Test Report

The HTML test report is a monolithic Javascript application that does not need to load any external data.

### Historical HTML test-report walkthrough

[Download recording](_media/e3d8adbb97c6757c.mp4) · [Download transcript](_media/692ffb266307d36f.txt)

```default
Historical HTML test-report walkthrough
======================================

Visual walkthrough of the silent original screen recording, based on reviewed
overview/detail/filter/search/attachment/print frames. This records the historical
interface, not a newly executed test run or current product validation.

Original recording: showcase_testreport.mp4

1. Open the report overview. A doughnut chart and result bar summarize six tests:
   three successful tests, one failure, one error, and one skipped test. Hovering
   the chart highlights the successful half and displays its count/percentage.

2. Open an error result. The expanded red result card displays an exception,
   traceback, timestamps/duration, documentation, metadata and captured stdout.

3. Browse the All and Success result lists from the sidebar. The All list contains
   docstring, error, failure, plot/attachment, skip and CAPTCHA examples; Success
   narrows the list to the three successful tests.

4. Use the search box to filter for attach. Matching text is highlighted in the
   plot/attachments result. Its expanded green card shows the documentation, tags,
   successful Create plot/Attach plot steps, metadata, stdout and a Lorenz Attractor
   image attachment. Clear the search to return to the result list.

5. Open the printable report and scroll through it. The print view contains the
   HILSTER title page, overview and individual test details, including the docstring
   example with table/text content and the error example with its traceback.
```

When loading the test report, a progress bar appears, and the percentage of loaded tests is shown. The HTML test report
usually loads very fast.

When the test report has finished loading, a responsive interface is shown, allowing you to interact with the test report.

### Navigation

The navigation is located in the sidebar to the left. The first two entries, `Overview` and `Metadata`,
are helpful for getting an initial impression of the test run. When the test report is loaded, `Overview`
is shown by default, visualizing and summarizing the test results in an easy to understand way. `Metadata`
shows test run specific information.

The remaining entries are used for navigating through the test results, and filtering which test results to
display.
Clicking `All` shows all test results. When selecting result types (`Success`, `Error`, `Failure` and
`Skipped`), the test results get filtered accordingly. It is possible to select multiple result types at once.
The number of tests within the respective result types are shown next to the filters.

Note that clicking `All`/pressing `esc` resets any applied filters.

Information about the host, the platform the tests were run on and licensing information are shown at the bottom
of the sidebar on the left.

The sidebar can be hidden by clicking on the hamburger icon on the top, next to the logo.

Test title and test date are shown in the top right corner.

### Overview

The visualisation is hierarchical: The inner ring represents the the four results types `success`, `error`,
`failure` and `skipped`, as shown in the sidebar. The next ring shows more detailed information, such as
exception types for failures and errors, and skip reasons for skipped tests. These information get even more
detailed in the outer ring, making it possible to easily identify concrete exception strings within respective
exception types.

Hovering over the diagram reveals additional information, such as hierarchy paths and statistics.

Clicking a section of the diagram searches the test report for the respective
result type/exception type/exception string.

### Test results

The main content is located to the right of the sidebar, containing all test results.
Clicking on a result expands the respective test and shows further information.
By default all test results are collapsed.

Every test result contains different test properties, such as `result`, `traceback`, `metadata`,
`documentation`, `tags`, `dates`, `attachments`, captured `stdout` and captured `stderr`.
To improve performance, long output (`stdout` and `stderr`) is not loaded by default. Clicking on `show more`
loads the respective output.

<a id="html-test-report-file-attachments"></a>

#### File attachments

The HTML test report supports file attachments that you can attach to a test using
[`htf.TestCase.attach_file`](writing_tests.md#htf.TestCase.attach_file) (or `htf.TestCase.attachFile`).

File attachments are not supported by all browsers. For more information see [Browser Limitations](#browser-limitations).

<a id="html-test-report-url-attachments"></a>

#### URL attachments

You can also attach URLs by using [`htf.TestCase.attach_url`](writing_tests.md#htf.TestCase.attach_url) (or `htf.TestCase.attachUrl`) in your test.

### Searching

Clicking on the magnifier icon on the top opens a search interface. The HTML test report offers three
different search methods: Default search, global search and special pattern search.

Note that only selected categories get searched. In order to search all testresults, you would need to select `All`
in the sidebar.

The search interface can be closed by clicking on the small `x` to the right of the search bar.

#### Default search

By default, only test titles get searched. Typing in the search bar instantly triggers the search.

Toggling the regex slider, located to the right side of the search bar, allows the user to perform a
search using regular expressions.

The test report also offers two additional advanced searching methods, global search and special pattern
search.

#### Global search

Global search requires the search term to be prefixed with `global:`, and allows the user to search not
only titles, but also in any test properties within the test results, such as `Dates`, `Traceback`
or `Tags`.

For example to search for tests that include `foo`, the user would use `global:foo`.

To search for tests that include `foo` or `bar`, you would use `global:foo|bar`, combining global search
with regular expressions.

#### Special pattern search

Special pattern search allows to search in specific test properties, such as `Tags`, `Dates` or
`Traceback`, by using the name of the respective property as prefix in lowercase. For example,
to search for tests tagged with `tag_1`, one would search for `tags:tag_1`.
Accordingly, searching for tests which were started on `2020-01-09`, one would search for
`dates:2020-01-09`.

This can also be used to filter for all tests containing a specific property. For example, to only show
tests containing `Tags`, one would search for `tags:`.

#### Goto

When pressing `g`, an input field appears next to the search bar, allowing the user to jump to a specific test.
Entering the test run number (which can be shown by pressing `n` ) of the respective test and pressing `Enter`
expands and focuses the test.

### Bookmarking

Right-clicking a test bookmarks the test for easier access. A bookmark-section is shown in the sidebar when there
are active bookmarks.

### Draft

Even when not all tests are run yet, the test report can be interacted with. This is indicated by a red `draft`
sign in the sidebar.

<a id="print-view"></a>

### Printing

While in test run overview, a print icon is shown in the upper right corner. Clicking it will render a print
optimized version of the test report, including an overview diagram, the test run metadata and a signature page.

In print view, every test will get expanded and printed on a dedicated page for easier readability.

### Dark Mode

For easier readability, the testreport color scheme can be changed to a darker variant by pressing `D`.
Pressing the hotkey again switches back to the light theme.

### Hotkeys

The HTML test report supports the following hotkeys:

| Key   | Description                                              |
|-------|----------------------------------------------------------|
| `h`   | Show help (any click or key but `h` hides it again)      |
| `esc` | Reset view: hide help, hide search box, show all results |
| `a`   | Collapse all expanded test results                       |
| `o`   | Show test run overview diagram                           |
| `m`   | Show test run metadata                                   |
| `a`   | Show all results                                         |
| `s`   | Toggle successes                                         |
| `f`   | Toggle failures                                          |
| `e`   | Toggle errors                                            |
| `S`   | Toggle skipped                                           |
| `F`   | Show search interface                                    |
| `R`   | Show search interface and toggle regex parameter         |
| `g`   | Show goto interface to jump to a specific test           |
| `n`   | Toggle test run numbers                                  |
| `b`   | Toggle sidebar                                           |
| `l`   | Show open source licenses                                |
| `p`   | Show printable version of test report                    |
| `D`   | Activate Dark Mode                                       |

The only hotkey you have to remember is `h` though.

<a id="browser-limitations"></a>

### Browser Limitations

Most modern browsers, such as Firefox, Chrome and Microsoft Edge (Version 80+), support
the whole functionality of the test report.

File attachments are realized using data-urls.

Some Microsoft-browsers do not support them at all.
Chrome and Chromium have security features in place, but support an acceptable workaround.

The following browser limitations are known:

| Browser                                     | Limitations                                                                                                                                                                                                                                                                     |
|---------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `IE 11 and older`                           | - Not officially supported.                                                                                                                                                                                                                                                     |
| `Edge (Version 78 and`<br/>`earlier)`       | - Only image attachments can be downloaded by right<br/>  clicking and choosing `save image as`. Downloading<br/>  other types of attachments is not possible, since older<br/>  versions of Edge do not fully support downloading<br/>  data-urls. Everything else works well. |
| `Chrome/Chromium/`<br/>`Edge (Version 80+)` | - Attachments cannot be viewed or downloaded with a<br/>  single click on it. As a workaround you can use the<br/>  context menu and use `save as` or<br/>  `open in another tab`.                                                                                              |

Your browser must have Javascript enabled to view the test report. There are some plugins like `NoScript`
that will likely cause problems. Also old versions of Internet Explorer do not load Javascript by default.

## JUnit XML Test Report

HILSTER Testing Framework creates JUnit XML test reports that can be used in other tools
like Continuous Integration systems, etc.

The JUnit XML format does not include all available data htf offers.

## JSON Test Report

The JSON test report includes all test results with all data to be used in your tools.

## YAML Test Report

The YAML test report includes all test results with all data to be used in your tools.

<a id="doors-test-report"></a>

## IBM® Engineering Requirements Management DOORS Test Report

The IBM® Engineering Requirements Management DOORS Test Report is a special test report that creates a DXL script
to be imported into DOORS. Thus your test results can easily be imported
to DOORS.

The DOORS Test Report works in conjunction with the `@meets_doors_requirements`
decorator that attaches the tested requirements to individual tests.

```python
@meets_doors_requirements("doors://host:port/path-without-id-",
                   "REQ_1", "REQ_2")
def test_with_useful_name():
    pass
```

The IBM Rational DOORS Test Report can only be used within Python code because it needs
configuration parameters.

```python
import htf

@htf.meets_doors_requirements("REQ_1", "REQ_2", "REQ_3", "TEST_1")
def test_with_doors_links():
    pass

htf.main(title="DOORS Test Report",
    extra_report=[
             DOORSTestReport(filename="doors.dxl",
                             report_module="/path/to/report/module",
                             requirements_modules={
                                 "REQ_.*" : "/path/to/requirements_module",
                                 "TEST_.*" : "/path/to/test_specification",
                             },
                             links_modules={
                                 "REQ_.*" : "/path/to/links/to/requirements_module",
                                 "TEST_.*" : "/path/to/links/to/test_specification",
                                 }
                             )
             ],
    )
```

### *class* htf.DOORSTestReport(filename: [str](https://docs.python.org/3/builtins/stdtypes.html#str), report_module: [str](https://docs.python.org/3/builtins/stdtypes.html#str), requirements_modules: [Dict](https://docs.python.org/3/library/typing.html#typing.Dict)[[str](https://docs.python.org/3/builtins/stdtypes.html#str), [str](https://docs.python.org/3/builtins/stdtypes.html#str)], links_modules: [Dict](https://docs.python.org/3/library/typing.html#typing.Dict)[[str](https://docs.python.org/3/builtins/stdtypes.html#str), [str](https://docs.python.org/3/builtins/stdtypes.html#str)] | [None](https://docs.python.org/3/builtins/constants.html#None) = None, render_meta_data: [bool](https://docs.python.org/3/builtins/functions.html#bool) = True, render_steps: [bool](https://docs.python.org/3/builtins/functions.html#bool) = True)

`DOORSTestReport` creates a DOORS test report.
Links are automatically created from the created DOORS module to the fulfilled requirements.

To state the fulfilled requirements the [`meets_doors_requirements`](decorators.md#htf.meets_doors_requirements) decorator should be used.

Initialize a `DOORSTestReport` instance.

* **Parameters:**
  * **filename** – the filename for the DXL test report.
  * **report_module** – the DOORS path to the report module.
  * **requirements_modules** – maps links to DOORS modules.
    The keys are regular expressions and the values are
    the desired target modules.
  * **links_modules={".\*"** – “DOORS Links”}: maps requirements to desired link modules.
    The keys are regular expressions and the values are
    the desired link modules.
  * **render_meta_data=True** – if set to True metadata is rendered in DOORS
  * **render_steps=True** – if set to True steps are rendered in DOORS

#### render(data: [Dict](https://docs.python.org/3/library/typing.html#typing.Dict)[[str](https://docs.python.org/3/builtins/stdtypes.html#str), [Any](https://docs.python.org/3/library/typing.html#typing.Any)]) → [None](https://docs.python.org/3/builtins/constants.html#None)

Generates and stores a DXL encoded TestReport.

* **Parameters:**
  **data** – test result data
