Matrix Requirements
Matrix Requirements is an Application Lifecycle Management System and Quality Management System made to design medical products.
QABench offers an integration (called htf-matrixreq
) to enable specification of tests within
Matrix Requirements, as well as sending test results back.
Installation
htf-matrixreq
is included when htf[full]
is installed on your system. It can also be installed
explicitly with htf[matrixreq]
.
See installation instructions for more details.
General Usage
In general, Matrix Requirements allows you to create and manage requirements. In addition, it allows you to specify tests that cover these requirements, and therefore link to them.
For tests, multiple test executions can be created and managed in folders.
htf-matrixreq
is run with at least one of these folder ids, recursively collects
the test execution items, finds the corresponding test specifications, runs the test and sends back
the results for the implemented tests to the corresponding test execution items automatically.
htf-matrixreq
needs at least one test execution folder in Matrix Requirements.
To create that please follow the instructions in the
Matrix Requirements Documentation.
Tests that will be run need to implement at least on test specification. See Linking Implemented Tests for more details.
To get the id of the test execution folder, simply click on it in the browser and get the id from the url or the title.
![_images/matrixreq-test-execution-folder-id.png](_images/matrixreq-test-execution-folder-id.png)
In the example, the test execution folder id is F-XTC-7
.
Linking Implemented Tests
You can link the implemented Matrix Requirements test specifications in to tests in htf
using
the decorator :func:htf.implements
for script based tests or using the Implements:
keyword for
BDD tests.
The following example shows a test with linked test specifications in Python.
@htf.implements("TC-1", "https://<user>.matrixreq.com/<project>/TC-2")
def test_testcase_1_and_2(step):
with step("First test step", expected_result="This step succeeds"):
assert True
with step("Second test step", expected_result="This step succeeds, too"):
assert True
The following example shows a corresponding test in BDD.
Feature: Feature with implemented tests
Implements: TC-1, https://<user>.matrixreq.com/<project>/TC-2
Scenario: Scenario with implemented tests
Implements: TC-3
# ...
API Token
htf-matrixreq
uses the Rest API of Matrix Requirements and thus needs an API token to function.
To create an API token, login into Matrix Requirements and navigate to
https://<user>.matrixreq.com/adminConfig/Token
.
Create an API token and store it.
![_images/matrixreq-token.png](_images/matrixreq-token.png)
Usage from Command Line
htf-matrixreq
supports the same command line arguments as htf
does and adds a few other.
For example to run all tests in the folder tests
for the test execution folder ids
F-XTC-1
and F-XTC-7
, run:
htf-matrixreq --items F-XTC-1,F-XTC-7 tests/
A full example in which all options are set via command line would be:
htf-matrixreq \
--host https://<user>.matrixreq.com \
--project <project> --items F-XTC-1,F-XTC-7 \
--token <token> \
--set Tester=tester \
--set Version=1.2.3 \
tests/
This example also sets the extra fields Version
and Tester` within the Matrix Requirements test execution item,
and gets all other options from command line.
Usage in Python
htf.matrixreq.main
supports the same parameters as htf.main
does and adds a few other.
from htf.matrixreq import main
if __name__ == "__main__":
main(
host="https://<user>.matrixreq.com",
project="<project>",
token="<token>",
items=["F-XTC-1", "XTC-7"],
extra_fields={
"Tester": "tester",
"Version": "1.2.3",
},
tests=["tests/"],
)
htf.matrixreq.run
is an alias for htf.matrixreq.main
.
Python Functions
- htf.matrixreq.main(host: str | None = None, project: str | None = None, items: List[str] | None = None, token: str | None = None, extra_fields: Dict[str, str] | None = None, insecure: bool = False, date_format: str = 'yyyy/MM/dd', date_time_format: str = 'yyyy/MM/dd HH:mm:ss', *htf_args: Any, **htf_kwargs: Any) int
The htf.matrix.main() and
htf-matrixreq
command line utility is used to run tests and send the results back to Matrix ALM/QMS.- Parameters:
project – the name of the project
items – the items to search for test execution item
token – the REST API token
extra_fields – extra fields to set for updated test execution items
insecure – if set to
True
SSL/TLS certificates are not checked(default (date_time_format) – “yyyy/MM/dd”): the date format for the project
(default – “yyyy/MM/dd HH:mm:ss”): the date time format for the project
*htf_args – htf positional arguments
**htf_kwargs – htf keyword arguments
- Returns:
the return code of
htf.main()
plus the number of errors and warnings produced by this tool.- Return type:
- htf.matrixreq.run(host: str | None = None, project: str | None = None, items: List[str] | None = None, token: str | None = None, extra_fields: Dict[str, str] | None = None, insecure: bool = False, date_format: str = 'yyyy/MM/dd', date_time_format: str = 'yyyy/MM/dd HH:mm:ss', *htf_args: Any, **htf_kwargs: Any) int
The htf.matrix.main() and
htf-matrixreq
command line utility is used to run tests and send the results back to Matrix ALM/QMS.- Parameters:
project – the name of the project
items – the items to search for test execution item
token – the REST API token
extra_fields – extra fields to set for updated test execution items
insecure – if set to
True
SSL/TLS certificates are not checked(default (date_time_format) – “yyyy/MM/dd”): the date format for the project
(default – “yyyy/MM/dd HH:mm:ss”): the date time format for the project
*htf_args – htf positional arguments
**htf_kwargs – htf keyword arguments
- Returns:
the return code of
htf.main()
plus the number of errors and warnings produced by this tool.- Return type:
Options
To use htf-matrixreq
, you need to specify the hostname, the project name, the API token and, if needed,
the insecure flag.
These options can either be set via command line options, by storing them in htf.settings
, and
by setting them via environment variables.
Environment variables are overwritten by htf.settings
,
whereas command line options overwrite everything else.
Command Line Options
To specify the hostname via command line, use --host <host>
.
To specify the project name via command line, use --project <project>
.
To specify the API token via command line, use --token <API Token>
.
The full list of command line options is:
usage: htf-matrixreq [-h] [--host HOST] [--project PROJECT] --items ITEMS [--token TOKEN] [--insecure]
[--set field1=value1,field2=value2,...,fieldN=valueN] [--date-format DATE_FORMAT]
[--date-time-format DATE_TIME_FORMAT]
htf-matrixreq - HILSTER Testing Framework integration for Matrix Requiements ALM/QMS System (HILSTER, https://hilster.io)
options:
-h, --help show this help message and exit
--host HOST The host of the Matrix Requirements project (required)
--project PROJECT The name of the Matrix Requirements project (required)
--items ITEMS The items to recursively search for tests in. Can be a comma-separated list of items, e.g.
--item 'F-XTC-1,F-XTC-2'. (required, can be repeated)
--token TOKEN The REST API token to use (required)
--insecure Disable SSL/TLS certificate check (can be useful behind some proxy servers)
--set field1=value1,field2=value2,...,fieldN=valueN
Set extra fields on every item, e.g. Tester, Version, etc. Can be repeated.
--date-format DATE_FORMAT
The date format for the project.
--date-time-format DATE_TIME_FORMAT
The date time format for the project.
Settings
You can specify hostname, project name, API token and the insecure flag in htf.settings
, too.
To do that add a section in the settings called matrixreq
.
YAML Example:
matrixreq:
host: https://<user>.matrixreq.com
project: <project>
token: <token>
date_format: "yyyy/MM/dd"
date_time_format: "yyyy/MM/dd HH:mm:ss"
extra_fields:
Tester: <tester>
Version: <version>
insecure: false # true|false
JSON Example:
"matrixreq": {
"host": "https://<user>.matrixreq.com",
"project": "<project>",
"token": "<token>",
"date_format": "yyyy/MM/dd",
"date_time_format": "yyyy/MM/dd HH:mm:ss",
"extra_fields" {
"Tester": "<tester>",
"Version": "<version>"
},
"insecure": false
}
You can leave out any value if it is not needed, or if it is specified elsewhere.
Environment Variables
The following tables shows the supported environment variables and their meaning.
Environment Variable |
Meaning |
---|---|
MATRIXREQ_HOST |
Hostname, e.g. https://<user>.matrixreq.com |
MATRIXREQ_INSECURE |
Disable SSL/TLS certificate verification. Any value is supported. |
MATRIXREQ_PROJECT |
Project Name |
MATRIXREQ_TOKEN |
API Token, e.g. <token> |
MATRIXREQ_DATE_FORMAT |
The date format. Default: “yyyy/MM/dd” |
MATRIXREQ_DATE_TIME_FORMAT |
The date time format. Default: “yyyy/MM/dd HH:mm:ss” |
Setting Extra Fields
You can set extra fields within Matrix Requirements test execution items such as Tester
or Version
when sending test results back to test execution items.
To do this in the command line, add --set name=value
pairs.
For example in order to set the Version
to 1.2.3
, use --item Version=1.2.3
.
You can also set these extra values using htf.settings
.
Date and Time Formats
htf-matrixreq
needs to set the test execution timestamps in the specified format set
within the Matrix Requirements project.
To view and change the formats, login into Matrix Requirements and navigate to
https://<user>.matrixreq.com/adminConfig/settingsdatetime
.
![_images/matrixreq-date-and-time-settings.png](_images/matrixreq-date-and-time-settings.png)
If the formats differ the test execution timestamps will not be set.
Proxy
You can specify proxies to use by setting the environment variable https_proxy
to https://<username>:<password>@<host>:<port>
.
On Linux enter:
export https_proxy=https://<username>:<password>@<host>:<port>
and on Windows enter:
set https_proxy=https://<username>:<password>@<host>:<port>
SSL/TLS Verification
In some circumstances it might be necessary to disable SSL/TLS verification.
To do this simply add the option --insecure
to htf-matrixreq
or
set the environment variable MATRIXREQ_INSECURE
to any value.
On Linux enter:
export MATRIXREQ_INSECURE=true
and on Windows enter:
set MATRIXREQ_INSECURE=true
Debugging
To enable debugging of HTTPS communication set the environment variable
MATRIXREQ_DEBUG
to any value and htf-matrixreq
.
On Linux enter:
export MATRIXREQ_DEBUG=true
and on Windows enter:
set MATRIXREQ_DEBUG=true
Changelog
htf-matrixreq-1.0.1
fix setting of actual results
htf-matrixreq-1.0.0
initial integration