Test Reports¶
HILSTER Testing Framework includes test reports for HTML, XML, YAML and JSON.
HTML Test Report¶
The HTML test report is a monolithic Javascript application that does not need to load any external data.
Historical HTML test-report walkthrough
Download recording · Download transcript
Historical HTML test-report walkthrough
======================================
Visual walkthrough of the silent original screen recording, based on reviewed
overview/detail/filter/search/attachment/print frames. This records the historical
interface, not a newly executed test run or current product validation.
Original recording: showcase_testreport.mp4
1. Open the report overview. A doughnut chart and result bar summarize six tests:
three successful tests, one failure, one error, and one skipped test. Hovering
the chart highlights the successful half and displays its count/percentage.
2. Open an error result. The expanded red result card displays an exception,
traceback, timestamps/duration, documentation, metadata and captured stdout.
3. Browse the All and Success result lists from the sidebar. The All list contains
docstring, error, failure, plot/attachment, skip and CAPTCHA examples; Success
narrows the list to the three successful tests.
4. Use the search box to filter for attach. Matching text is highlighted in the
plot/attachments result. Its expanded green card shows the documentation, tags,
successful Create plot/Attach plot steps, metadata, stdout and a Lorenz Attractor
image attachment. Clear the search to return to the result list.
5. Open the printable report and scroll through it. The print view contains the
HILSTER title page, overview and individual test details, including the docstring
example with table/text content and the error example with its traceback.
When loading the test report, a progress bar appears, and the percentage of loaded tests is shown. The HTML test report usually loads very fast.
When the test report has finished loading, a responsive interface is shown, allowing you to interact with the test report.
Overview¶
The visualisation is hierarchical: The inner ring represents the the four results types success, error,
failure and skipped, as shown in the sidebar. The next ring shows more detailed information, such as
exception types for failures and errors, and skip reasons for skipped tests. These information get even more
detailed in the outer ring, making it possible to easily identify concrete exception strings within respective
exception types.
Hovering over the diagram reveals additional information, such as hierarchy paths and statistics.
Clicking a section of the diagram searches the test report for the respective result type/exception type/exception string.
Test results¶
The main content is located to the right of the sidebar, containing all test results. Clicking on a result expands the respective test and shows further information. By default all test results are collapsed.
Every test result contains different test properties, such as result, traceback, metadata,
documentation, tags, dates, attachments, captured stdout and captured stderr.
To improve performance, long output (stdout and stderr) is not loaded by default. Clicking on show more
loads the respective output.
File attachments¶
The HTML test report supports file attachments that you can attach to a test using
htf.TestCase.attach_file (or htf.TestCase.attachFile).
File attachments are not supported by all browsers. For more information see Browser Limitations.
URL attachments¶
You can also attach URLs by using htf.TestCase.attach_url (or htf.TestCase.attachUrl) in your test.
Searching¶
Clicking on the magnifier icon on the top opens a search interface. The HTML test report offers three different search methods: Default search, global search and special pattern search.
Note that only selected categories get searched. In order to search all testresults, you would need to select All
in the sidebar.
The search interface can be closed by clicking on the small x to the right of the search bar.
Default search¶
By default, only test titles get searched. Typing in the search bar instantly triggers the search.
Toggling the regex slider, located to the right side of the search bar, allows the user to perform a search using regular expressions.
The test report also offers two additional advanced searching methods, global search and special pattern search.
Global search¶
Global search requires the search term to be prefixed with global:, and allows the user to search not
only titles, but also in any test properties within the test results, such as Dates, Traceback
or Tags.
For example to search for tests that include foo, the user would use global:foo.
To search for tests that include foo or bar, you would use global:foo|bar, combining global search
with regular expressions.
Special pattern search¶
Special pattern search allows to search in specific test properties, such as Tags, Dates or
Traceback, by using the name of the respective property as prefix in lowercase. For example,
to search for tests tagged with tag_1, one would search for tags:tag_1.
Accordingly, searching for tests which were started on 2020-01-09, one would search for
dates:2020-01-09.
This can also be used to filter for all tests containing a specific property. For example, to only show
tests containing Tags, one would search for tags:.
Goto¶
When pressing g, an input field appears next to the search bar, allowing the user to jump to a specific test.
Entering the test run number (which can be shown by pressing n ) of the respective test and pressing Enter
expands and focuses the test.
Bookmarking¶
Right-clicking a test bookmarks the test for easier access. A bookmark-section is shown in the sidebar when there are active bookmarks.
Draft¶
Even when not all tests are run yet, the test report can be interacted with. This is indicated by a red draft
sign in the sidebar.
Printing¶
While in test run overview, a print icon is shown in the upper right corner. Clicking it will render a print optimized version of the test report, including an overview diagram, the test run metadata and a signature page.
In print view, every test will get expanded and printed on a dedicated page for easier readability.
Dark Mode¶
For easier readability, the testreport color scheme can be changed to a darker variant by pressing D.
Pressing the hotkey again switches back to the light theme.
Hotkeys¶
The HTML test report supports the following hotkeys:
Key |
Description |
|---|---|
|
Show help (any click or key but |
|
Reset view: hide help, hide search box, show all results |
|
Collapse all expanded test results |
|
Show test run overview diagram |
|
Show test run metadata |
|
Show all results |
|
Toggle successes |
|
Toggle failures |
|
Toggle errors |
|
Toggle skipped |
|
Show search interface |
|
Show search interface and toggle regex parameter |
|
Show goto interface to jump to a specific test |
|
Toggle test run numbers |
|
Toggle sidebar |
|
Show open source licenses |
|
Show printable version of test report |
|
Activate Dark Mode |
The only hotkey you have to remember is h though.
Browser Limitations¶
Most modern browsers, such as Firefox, Chrome and Microsoft Edge (Version 80+), support the whole functionality of the test report.
File attachments are realized using data-urls.
Some Microsoft-browsers do not support them at all. Chrome and Chromium have security features in place, but support an acceptable workaround.
The following browser limitations are known:
Browser |
Limitations |
|---|---|
|
|
|
|
|
|
Your browser must have Javascript enabled to view the test report. There are some plugins like NoScript
that will likely cause problems. Also old versions of Internet Explorer do not load Javascript by default.
JUnit XML Test Report¶
HILSTER Testing Framework creates JUnit XML test reports that can be used in other tools like Continuous Integration systems, etc.
The JUnit XML format does not include all available data htf offers.
JSON Test Report¶
The JSON test report includes all test results with all data to be used in your tools.
YAML Test Report¶
The YAML test report includes all test results with all data to be used in your tools.
IBM® Engineering Requirements Management DOORS Test Report¶
The IBM® Engineering Requirements Management DOORS Test Report is a special test report that creates a DXL script to be imported into DOORS. Thus your test results can easily be imported to DOORS.
The DOORS Test Report works in conjunction with the @meets_doors_requirements
decorator that attaches the tested requirements to individual tests.
@meets_doors_requirements("doors://host:port/path-without-id-",
"REQ_1", "REQ_2")
def test_with_useful_name():
pass
The IBM Rational DOORS Test Report can only be used within Python code because it needs configuration parameters.
import htf
@htf.meets_doors_requirements("REQ_1", "REQ_2", "REQ_3", "TEST_1")
def test_with_doors_links():
pass
htf.main(title="DOORS Test Report",
extra_report=[
DOORSTestReport(filename="doors.dxl",
report_module="/path/to/report/module",
requirements_modules={
"REQ_.*" : "/path/to/requirements_module",
"TEST_.*" : "/path/to/test_specification",
},
links_modules={
"REQ_.*" : "/path/to/links/to/requirements_module",
"TEST_.*" : "/path/to/links/to/test_specification",
}
)
],
)
- class htf.DOORSTestReport(filename: str, report_module: str, requirements_modules: Dict[str, str], links_modules: Dict[str, str] | None = None, render_meta_data: bool = True, render_steps: bool = True)¶
DOORSTestReportcreates a DOORS test report. Links are automatically created from the created DOORS module to the fulfilled requirements.To state the fulfilled requirements the
meets_doors_requirementsdecorator should be used.Initialize a
DOORSTestReportinstance.- Parameters:
filename – the filename for the DXL test report.
report_module – the DOORS path to the report module.
requirements_modules – maps links to DOORS modules. The keys are regular expressions and the values are the desired target modules.
links_modules={".*" – “DOORS Links”}: maps requirements to desired link modules. The keys are regular expressions and the values are the desired link modules.
render_meta_data=True – if set to True metadata is rendered in DOORS
render_steps=True – if set to True steps are rendered in DOORS