Data Driven Testing

Data Driven Testing allows dynamic generation of tests using a data source. The data source can be Python objects, JSON, CSV or YAML files.

To use Data Driven Testing simply decorate tests using the data or combinatoric decorators as mentioned below.

Code-based Examples

Full Example

In the following example the test function is called three times with a value from a list of scalars. To use data driven testing in htf the test methods are decorated with the decorator supplying the data values.

import htf

@htf.data([1.0, 2.0, 3.0])
def test_scalar(value):
    print("value:", value)

if __name__ == "__main__":
    htf.main()

List of Scalars

The following examples demonstrate the usage of Python objects as data input. The decorator to be used when the data is supplied as a Python object is @htf.data(<object>).

The following example uses a list of integers as data.

@htf.data([23, 24, 25])
def test_scalar(value):
    print("value:", value)

# generates calls
# test_scalar(value=23)
# test_scalar(value=24)
# ...

Iterator

An iterator can also be supplied.

@htf.data(iter([23, 24, 25]))
def test_iterator(value):
    print("value:", value)

Generator

It is also possible to supply a generator.

def generator():
    yield 10
    yield 20
    yield 30

@htf.data(generator())
def test_generator_instance(value):
    print("value:", value)

# generates calls
# test_generator_instance(value=10)
# test_generator_instance(value=20)
# ...

@htf.data(generator)
def test_generator(value):
    print("value:", value)

# generates calls
# test_generator(value=10)
# test_generator(value=20)
# ...

List of Dicts

The following example supplies a list of dictionaries. The number of parameters must match the number of dictionary entries.

@htf.data([dict(a=23, b=1), dict(a=24, b=2), dict(a=25, b=3)])
def test_dict(a, b):
   print("a:", a, "b:", b)

# generates calls
# test_dict(a=23, b=1)
# test_dict(a=24, b=2)
# ...

List of Lists

Lastly it is also possible to supply lists of lists. When called, value contains a list.

@htf.data([[1, 2, 3], [4, 5, 6]])
def test_list_of_int(value):
    print("value:", value)

# generates calls
# test_list_of_int(value=[1, 2, 3])
# ...

@htf.data([[dict(value=23)], [dict(value=24)], [dict(value=25)]])
def test_list_of_lists_of_dict(values):
   print("value:", values)

# generates calls
# test_list_of_lists_of_dict(value=23)
# test_list_of_lists_of_dict(value=24)
# ...

Unpack the content of value by setting unpack=True to distribute the list entries to three parameters.

@htf.data([[1, 2, 3], [4, 5, 6]], unpack=True)
def test_list_of_int_unpack(a, b, c):
    print("a:", a, "b:", b, "c:", c)

# generates
# test_list_of_int_unpack(a=1, b=2, c=3)
# ...

File-based Examples

The following examples demonstrate the usage of various file formats as data input. The supported file formats and their corresponding decorators are:

File type

Decorator

JSON-file

@htf.json_data("filename.json")

YAML-file

@htf.yaml_data("filename.yml")

CSV-file

@htf.csv_data("filename.csv")

In each case there is also the option of first creating an iterator for the file content which is handed to the @htf.data() decorator. For this purpose the following syntax is used:

File type

Decorator and Iterator

JSON-file

@htf.data(htf.JSONFileIterator("filename.json"))

YAML-file

@htf.data(htf.YAMLFileIterator("filename.yml"))

CSV-file

@htf.data(htf.CSVFileIterator("filename.csv"))

Furthermore, decorators for JSON and YAML input have the option unpack set to False by default. If changed to True lists will be unpacked as demonstrated in the below examples. There is no such option for the CSV decorator.

JSON Test Data

To supply simple scalar values as data from a JSON file, use the following code:

@htf.json_data("ddt_scalar.json")
def test_json_scalar(value):
    print("value: ", value)

# generates calls
# test_json(value=1337)
# test_json(value=1338)
# ...

@htf.data(htf.JSONFileIterator("ddt_scalar.json"))
def test_json_iterator(value):
    print("value:", value)

# generates calls
# test_json_iterator(value=1337)
# test_json_iterator(value=1338)
# ...

Contents of ddt_scalar.json:

[1337, 1338, 1339]

If the supplied data comes in form of nested lists, unpack can be set to True to assign each inner list element to a variable.

@htf.json_data("ddt_list.json", unpack=True)
def test_json_list_unpack(a, b, c):
    print("value: ", a, b, c)

# generates calls
# test_json_list(a=111, b=222, c=333)
# test_json_list(a=11, b=22, c=33)
# ...

@htf.json_data("ddt_list.json", unpack=False)
def test_json_list(value):
    print("value: ", value)

# generates calls
# test_json_list(value=[111, 222, 333])
# test_json_list(value=[11, 22, 33])
# ...

Contents of ddt_list.json:

[[1,2,3], [11,22,33], [111,222,333]]

YAML Test Data

Supplying scalar values as data from a YAML file works analogously to JSON files.

@htf.yaml_data("ddt_scalar.yml")
def test_yaml_scalar(value):
    print("value:", value)

# generates calls
# test_yaml(value=1337)
# test_yaml(value=1338)
# ...

@htf.data(htf.YAMLFileIterator("ddt_scalar.yml"))
def test_yaml_iterator(value):
    print("value:", value)

# generates calls
# test_yaml_iterator(value=1337)
# test_yaml_iterator(value=1338)
# ...

Contents of ddt_scalar.yml:

[1337, 1338, 1339]

The following code showcases the use of the unpack option in the case of a YAML file.

@htf.yaml_data("ddt_list.yml", unpack=True)
def test_yaml_list_unpack(a, b, c):
    print("value: ", a, b, c)

# generates calls
# test_yaml_list(a=111, b=222, c=333)
# test_yaml_list(a=11, b=22, c=33)
# ...

@htf.yaml_data("ddt_list.yml", unpack=False)
def test_yaml_list(value):
    print("value: ", value)

# generates
# test_yaml_list(value=[111, 222, 333])
# test_yaml_list(value=[11, 22, 33])
# ...

Contents of ddt_list.yml:

[[1,2,3], [11,22,33], [111,222,333]]

CSV Test Data

Tests can also be generated using CSV data sources. The first line of the CSV data is a header. The parameter names of the decorated method must match the entries of the header, i.e. the fist line of each column must contain the name of the parameter to which the data is to be assigned. Each line is supplied as a dict. Note that the entries of the dict are strings by default.

@htf.csv_data("ddt.csv")
def test_csv(a, b):
    print("a: ", a, "b: ", b)

# generates calls
# test_csv(a=1337, b=2222)
# a: 1337 b: 2222
# test_csv(a=1338, b=3333)
# a: 1338 b: 3333
# ...

@htf.data(htf.CSVFileIterator("ddt.csv"))
def test_csv_iterator(a, b):
    print("a: ", a, "b: ", b)

# generates calls
# test_csv(a=1337, b=2222)
# a: 1337 b: 2222
# test_csv(a=1337, b=3333)
# a: 1338 b: 3333
# ...

Contents of ddt.csv:

a,b
1337,2222
1338,3333
1339,4444

CSV data source values are not converted automatically. The desired conversion can be specified with the optional convert_to parameter. convert_to can be a callable or list thereof. If conversion fails with a ValueError the value is passed as a string.

@htf.csv_data("ddt.csv", convert_to=[int, float])
def test_csv(a, b):
    print("a: ", a, "b: ", b)

# generates calls
# test_csv(a=1337, b=2222.0)
# a: 1337 b: 2222.0
# test_csv(a=1338, b=3333.0)
# a: 1338 b: 3333.0
# ...

Combinatoric Data Generation

Combinatoric decorators can be used to manipulate the supplied data.

Cartesian Product

The product function produces every possible combination of items, taking one item of each given iterable.

@htf.product([1, 2, 3], repeat=2)
def test_product(a, b):
    print("a:", a, "b:", b)

# generates calls
# test_product(a=1, b=1)
# test_product(a=1, b=2)
# ...

@htf.product([1, 2], [3, 4], repeat=1)
def test_products(a, b):
    print("a:", a, "b:", b)

# generates calls
# test_products(a=1, b=3)
# test_products(a=2, b=3)
# ...

In combination with the aforementioned file iterator functions, this can be used to produce combinations of elements of given data files.

@htf.product(htf.JSONFileIterator("ddt_scalar.json"), repeat=2)
def test_json_products(a, b):
    print("a:", a, "b:", b)

# generates calls
# test_products(a=1337, b=1337)
# test_products(a=1337, b=1338)
# ...

@htf.product(list(range(10)), repeat=3)
def test_long_products(*args):
    print("args:", args)

# generates calls
# test_long_products(*(0, 0, 1))
# test_long_products(*(0, 0, 2))
# ...

Permutations

The permutations function creates all permutations of length r of a given iterable.

@htf.permutations([1, 2, 3], r=2)
def test_permutations(a, b):
    print("a:", a, "b:", b)

# generates calls
# test_permutations(a=1, b=2)
# test_permutations(a=1, b=3)
# ...

Combinations

The combinations function produces all possible combinations of elements of an iterable without changing their order. Elements are not combined with themselves.

@htf.combinations([1, 2, 3], r=2)
def test_combinations(a, b):
    print("a:", a, "b:", b)

# generates calls
# test_combinations(a=1, b=2)
# test_combinations(a=1, b=3)
# ...

The combinations_with_replacement function does the same as the combinations function, except elements are also combined with themselves.

@htf.combinations_with_replacement([1, 2, 3], r=2)
def test_combinations_with_replacement(a, b):
    print("a:", a, "b:", b)

# generates calls
# test_combinations_with_replacement(a=1, b=1)
# test_combinations_with_replacement(a=1, b=2)
# ...

Data Decorators

htf.data(iterable, unpack=False)

Create a data driven test using data.

Parameters
  • iterable – an iterable used as parameter data source. Can also be a callable.

  • unpack=False (bool) – if set to True iterables are unpacked.

htf.csv_data(filename, convert_to=None)

Create a data driven test using data from a CSV file. The first line of the CSV data is a header. Each column from the first is used as the parameter name of the decorated method. Each line is supplied as a dict.

Parameters
  • filename (str) – the filename to be read.

  • convert_to (list of types) – data types to convert read values to (e.g. float, int)

htf.yaml_data(filename, unpack=False)

Create a data driven test using data from a YAML file.

Parameters
  • filename (str) – the filename to be read.

  • unpack=False (bool) – if set to True iterables are unpacked.

htf.json_data(filename, unpack=False)

Create a data driven test using data from a json file.

Parameters
  • filename (str) – the filename to be read.

  • unpack=False (bool) – if set to True iterables are unpacked.

Combinatoric Decorators

htf.product(*iterables, **repeat)

Create a data driven test using itertools.product with a cartesian product of input *iterables.

Parameters
  • *iterables – iterables to be passed to itertools.product

  • repeat=1 – number of repetitions.

htf.permutations(iterable, r=None)

Create a data driven test using itertools.permutations with successive r length permutations of elements in the iterable.

Parameters
htf.combinations(iterable, r)

Create a data driven test using itertools.combinations with successive r length subsequences of elements from iterable.

Parameters
htf.combinations_with_replacement(iterable, r)

Create a data driven test using itertools.combinations with successive r length subsequences of elements from iterable allowing individual elements to be repeated more than once.

Parameters

File Iterators

class htf.CSVFileIterator(filename, convert_to=None)

CSV file iterator.

Parameters

filename (str) – the filename to be read.

Returns

iterator for the file contents as dict. convert_to (list of types): data types to convert read values to (e.g. float, int)

Return type

iterator

class htf.YAMLFileIterator(filename)

YAML file iterator.

Parameters

filename (str) – the filename to be read.

Returns

iterator for the file contents.

Return type

iterator

class htf.JSONFileIterator(filename)

JSON file iterator.

Parameters

filename (str) – the filename to be read.

Returns

iterator for the file contents.

Return type

iterator