Data Driven Testing

Data Driven Testing allows dynamic generation of tests using a data source. The data source can be Python objects, JSON, CSV or YAML files.

To use Data Driven Testing simply decorate tests using the data or combinatoric decorators as mentioned below.

Code-based Examples

Full Example

In the following example the test function is called three times with a value from a list of scalars. To use data driven testing in htf the test methods are decorated with the decorator supplying the data values.

import htf

@htf.data([1.0, 2.0, 3.0])
def test_scalar(value):
    print("value:", value)

if __name__ == "__main__":
    htf.main()

List of Scalars

The following examples demonstrate the usage of Python objects as data input. The decorator to be used when the data is supplied as a Python object is @htf.data(<object>).

The following example uses a list of integers as data.

@htf.data([23, 24, 25])
def test_scalar(value):
    print("value:", value)

# generates calls
# test_scalar(value=23)
# test_scalar(value=24)
# ...

Iterator

An iterator can also be supplied.

@htf.data(iter([23, 24, 25]))
def test_iterator(value):
    print("value:", value)

Generator

It is also possible to supply a generator.

def generator() -> Generator[int, None, None]:
    yield 10
    yield 20
    yield 30

@htf.data(generator())
def test_generator_instance(value):
    print("value:", value)

# generates calls
# test_generator_instance(value=10)
# test_generator_instance(value=20)
# ...

@htf.data(generator)
def test_generator(value):
    print("value:", value)

# generates calls
# test_generator(value=10)
# test_generator(value=20)
# ...

Async generators are supported, too.

async def generator() -> AsyncGenerator[int, None]:
    yield 10
    yield 20
    yield 30

@htf.data(generator())
async def test_generator_instance(value):
    print("value:", value)

List of Dicts

The following example supplies a list of dictionaries. The number of parameters must match the number of dictionary entries.

@htf.data([dict(a=23, b=1), dict(a=24, b=2), dict(a=25, b=3)])
def test_dict(a, b):
   print("a:", a, "b:", b)

# generates calls
# test_dict(a=23, b=1)
# test_dict(a=24, b=2)
# ...

List of Lists

Lastly it is also possible to supply lists of lists. When called, value contains a list.

@htf.data([[1, 2, 3], [4, 5, 6]])
def test_list_of_int(value):
    print("value:", value)

# generates calls
# test_list_of_int(value=[1, 2, 3])
# ...

@htf.data([[dict(value=23)], [dict(value=24)], [dict(value=25)]])
def test_list_of_lists_of_dict(values):
   print("value:", values)

# generates calls
# test_list_of_lists_of_dict(value=23)
# test_list_of_lists_of_dict(value=24)
# ...

Unpack the content of value by setting unpack=True to distribute the list entries to three parameters.

@htf.data([[1, 2, 3], [4, 5, 6]], unpack=True)
def test_list_of_int_unpack(a, b, c):
    print("a:", a, "b:", b, "c:", c)

# generates
# test_list_of_int_unpack(a=1, b=2, c=3)
# ...

File-based Examples

The following examples demonstrate the usage of various file formats as data input. The supported file formats and their corresponding decorators are:

File type	Decorator
JSON-file	`@htf.json_data("filename.json")`
YAML-file	`@htf.yaml_data("filename.yml")`
CSV-file	`@htf.csv_data("filename.csv")`

In each case there is also the option of first creating an iterator for the file content which is handed to the @htf.data() decorator. For this purpose the following syntax is used:

File type	Decorator and Iterator
JSON-file	`@htf.data(htf.JSONFileIterator("filename.json"))`
YAML-file	`@htf.data(htf.YAMLFileIterator("filename.yml"))`
CSV-file	`@htf.data(htf.CSVFileIterator("filename.csv"))`

Furthermore, decorators for JSON and YAML input have the option unpack set to False by default. If changed to True lists will be unpacked as demonstrated in the below examples. There is no such option for the CSV decorator.

JSON Test Data

To supply simple scalar values as data from a JSON file, use the following code:

@htf.json_data("ddt_scalar.json")
def test_json_scalar(value):
    print("value: ", value)

# generates calls
# test_json(value=1337)
# test_json(value=1338)
# ...

@htf.data(htf.JSONFileIterator("ddt_scalar.json"))
def test_json_iterator(value):
    print("value:", value)

# generates calls
# test_json_iterator(value=1337)
# test_json_iterator(value=1338)
# ...

Contents of ddt_scalar.json:

[1337, 1338, 1339]

If the supplied data comes in form of nested lists, unpack can be set to True to assign each inner list element to a variable.

@htf.json_data("ddt_list.json", unpack=True)
def test_json_list_unpack(a, b, c):
    print("value: ", a, b, c)

# generates calls
# test_json_list(a=111, b=222, c=333)
# test_json_list(a=11, b=22, c=33)
# ...

@htf.json_data("ddt_list.json", unpack=False)
def test_json_list(value):
    print("value: ", value)

# generates calls
# test_json_list(value=[111, 222, 333])
# test_json_list(value=[11, 22, 33])
# ...

Contents of ddt_list.json:

[[1,2,3], [11,22,33], [111,222,333]]

YAML Test Data

Supplying scalar values as data from a YAML file works analogously to JSON files.

@htf.yaml_data("ddt_scalar.yml")
def test_yaml_scalar(value):
    print("value:", value)

# generates calls
# test_yaml(value=1337)
# test_yaml(value=1338)
# ...

@htf.data(htf.YAMLFileIterator("ddt_scalar.yml"))
def test_yaml_iterator(value):
    print("value:", value)

# generates calls
# test_yaml_iterator(value=1337)
# test_yaml_iterator(value=1338)
# ...

Contents of ddt_scalar.yml:

[1337, 1338, 1339]

The following code showcases the use of the unpack option in the case of a YAML file.

@htf.yaml_data("ddt_list.yml", unpack=True)
def test_yaml_list_unpack(a, b, c):
    print("value: ", a, b, c)

# generates calls
# test_yaml_list(a=111, b=222, c=333)
# test_yaml_list(a=11, b=22, c=33)
# ...

@htf.yaml_data("ddt_list.yml", unpack=False)
def test_yaml_list(value):
    print("value: ", value)

# generates
# test_yaml_list(value=[111, 222, 333])
# test_yaml_list(value=[11, 22, 33])
# ...

Contents of ddt_list.yml:

[[1,2,3], [11,22,33], [111,222,333]]

CSV Test Data

Tests can also be generated using CSV data sources. The first line of the CSV data is a header. The parameter names of the decorated method must match the entries of the header, i.e. the fist line of each column must contain the name of the parameter to which the data is to be assigned. Each line is supplied as a dict. Note that the entries of the dict are strings by default.

@htf.csv_data("ddt.csv")
def test_csv(a, b):
    print("a: ", a, "b: ", b)

# generates calls
# test_csv(a=1337, b=2222)
# a: 1337 b: 2222
# test_csv(a=1338, b=3333)
# a: 1338 b: 3333
# ...

@htf.data(htf.CSVFileIterator("ddt.csv"))
def test_csv_iterator(a, b):
    print("a: ", a, "b: ", b)

# generates calls
# test_csv(a=1337, b=2222)
# a: 1337 b: 2222
# test_csv(a=1337, b=3333)
# a: 1338 b: 3333
# ...

Contents of ddt.csv:

CSV data source values are not converted automatically. The desired conversion can be specified with the optional convert_to parameter. convert_to can be a callable or list thereof. If conversion fails with a ValueError the value is passed as a string.

@htf.csv_data("ddt.csv", convert_to=[int, float])
def test_csv(a, b):
    print("a: ", a, "b: ", b)

# generates calls
# test_csv(a=1337, b=2222.0)
# a: 1337 b: 2222.0
# test_csv(a=1338, b=3333.0)
# a: 1338 b: 3333.0
# ...

Combinatoric Data Generation

Combinatoric decorators can be used to manipulate the supplied data.

Cartesian Product

The product function produces every possible combination of items, taking one item of each given iterable.

@htf.product([1, 2, 3], repeat=2)
def test_product(a, b):
    print("a:", a, "b:", b)

# generates calls
# test_product(a=1, b=1)
# test_product(a=1, b=2)
# ...

@htf.product([1, 2], [3, 4], repeat=1)
def test_products(a, b):
    print("a:", a, "b:", b)

# generates calls
# test_products(a=1, b=3)
# test_products(a=2, b=3)
# ...

In combination with the aforementioned file iterator functions, this can be used to produce combinations of elements of given data files.

@htf.product(htf.JSONFileIterator("ddt_scalar.json"), repeat=2)
def test_json_products(a, b):
    print("a:", a, "b:", b)

# generates calls
# test_products(a=1337, b=1337)
# test_products(a=1337, b=1338)
# ...

@htf.product(list(range(10)), repeat=3)
def test_long_products(*args):
    print("args:", args)

# generates calls
# test_long_products(*(0, 0, 1))
# test_long_products(*(0, 0, 2))
# ...

Permutations

The permutations function creates all permutations of length r of a given iterable.

@htf.permutations([1, 2, 3], r=2)
def test_permutations(a, b):
    print("a:", a, "b:", b)

# generates calls
# test_permutations(a=1, b=2)
# test_permutations(a=1, b=3)
# ...

Combinations

The combinations function produces all possible combinations of elements of an iterable without changing their order. Elements are not combined with themselves.

@htf.combinations([1, 2, 3], r=2)
def test_combinations(a, b):
    print("a:", a, "b:", b)

# generates calls
# test_combinations(a=1, b=2)
# test_combinations(a=1, b=3)
# ...

The combinations_with_replacement function does the same as the combinations function, except elements are also combined with themselves.

@htf.combinations_with_replacement([1, 2, 3], r=2)
def test_combinations_with_replacement(a, b):
    print("a:", a, "b:", b)

# generates calls
# test_combinations_with_replacement(a=1, b=1)
# test_combinations_with_replacement(a=1, b=2)
# ...

Data Decorators

htf.data(iterable: Iterable, unpack: bool = False) → Callable[[Callable[[...], Any]], Any]

Create a data driven test using data.

Parameters:

iterable – an iterable used as parameter data source. Can also be a callable.
unpack=False – if set to True iterables are unpacked.

htf.csv_data(filename: str, convert_to: Iterable[Any] | Generator[Any, None, None] | List[Any] | Any | None = None) → Callable[[Callable[[...], Any]], Any]

Create a data driven test using data from a CSV file. The first line of the CSV data is a header. Each column from the first is used as the parameter name of the decorated method. Each line is supplied as a dict.

Parameters:

filename – the filename to be read.
convert_to – data types to convert read values to (e.g. float, int)

htf.yaml_data(filename: str, unpack: bool = False) → Callable[[Callable[[...], Any]], Any]

Create a data driven test using data from a YAML file.

Parameters:

filename – the filename to be read.
unpack=False – if set to True iterables are unpacked.

htf.json_data(filename: str, unpack: bool = False) → Callable[[Callable[[...], Any]], Any]

Create a data driven test using data from a json file.

Parameters:

filename – the filename to be read.
unpack=False – if set to True iterables are unpacked.

Combinatoric Decorators

htf.product(*iterables: Iterable, **repeat: int) → Callable[[Callable[[...], Any]], Any]

Create a data driven test using itertools.product with a cartesian product of input *iterables.

Parameters:

*iterables – iterables to be passed to itertools.product
repeat=1 – number of repetitions.

htf.permutations(iterable: Iterable, r: int | None = None) → Callable[[Callable[[...], Any]], Any]

Create a data driven test using itertools.permutations with successive r length permutations of elements in the iterable.

Parameters:

iterable – iterable to be passed to itertools.permutations
r=None – length of permutations

htf.combinations(iterable: Iterable, r: int) → Callable[[Callable[[...], Any]], Any]

Create a data driven test using itertools.combinations with successive r length subsequences of elements from iterable.

Parameters:

iterable – iterable to be passed to itertools.combinations
r – length of combinations

htf.combinations_with_replacement(iterable: Iterable, r: int) → Callable[[Callable[[...], Any]], Any]

Create a data driven test using itertools.combinations with successive r length subsequences of elements from iterable allowing individual elements to be repeated more than once.

Parameters:

iterable – iterable to be passed to itertools.combinations_with_replacement
r – length of combinations

File Iterators

class htf.CSVFileIterator(filename: str, convert_to: Iterable[Any] | Generator[Any, None, None] | List[Any] | Any | None = None)

CSV file iterator.

Parameters:: filename – the filename to be read.
Returns:: iterator for the file contents as dict. convert_to: data types to convert read values to (e.g. float, int)
Return type:: iterator

class htf.YAMLFileIterator(filename: str)

YAML file iterator.

Parameters:: filename – the filename to be read.
Returns:: iterator for the file contents.
Return type:: iterator

class htf.JSONFileIterator(filename: str)

JSON file iterator.

Parameters:: filename – the filename to be read.
Returns:: iterator for the file contents.
Return type:: iterator