diff --git a/tests/README-TESTS.md b/tests/README-TESTS.md index 408ae3ec..41655834 100644 --- a/tests/README-TESTS.md +++ b/tests/README-TESTS.md @@ -1,3 +1,5 @@ +
+ - [README-TESTS](#README-TESTS) - [Running the test suite](#running-the-test-suite) - [Writing new tests](#writing-new-tests) @@ -16,13 +18,13 @@ - [Creating file copies](#creating-file-copies) - [Customizing the output check](#customizing-the-output-check) - [Running all commands under valgrind](#running-all-commands-under-valgrind) - - [Manually expanding variables in strings](#manually-expanding-variables-in-strings) + - [Manually expanding variables in strings](#manually-expanding-variables) - [Hooks](#hooks) - [Possible pitfalls](#possible-pitfalls) - [Bash test cases](#bash-test-cases) - +
# README-TESTS This test suite is intended for system tests, i.e. for running a binary with @@ -31,6 +33,7 @@ especially useful for a regression test suite, but can be also used for testing of new features where unit testing is not feasible, e.g. to test new command line parameters. +
## Running the test suite The test suite is written for Python 3 and is not compatible with Python 2, thus @@ -38,7 +41,8 @@ it must be run with `python3` and not with `python` (which is usually an alias for Python 2). Then navigate to the `tests/` subdirectory and run: -``` shell + +```bash python3 -m pip install -r requirements.txt python3 runner.py ``` @@ -58,6 +62,9 @@ Optionally one can provide the `--debug` flag which will instruct test suite to print all command invocations and all expected and obtained outputs to the standard output. +[TOC](#TOC) + +
## Writing new tests The test suite is intended to run a binary and compare its standard output, @@ -65,7 +72,8 @@ standard error and return value against provided values. This is implemented using Python's [unittest](https://docs.python.org/3/library/unittest.html) module and thus all test files are Python files. The simplest test has the following structure: -``` python + +```python # -*- coding: utf-8 -*- import system_tests @@ -101,6 +109,9 @@ When creating new tests, follow roughly these steps: actually run! Either run the suite with the `-v` option which will output all test cases that were run or simply add an error and check if errors occur. +[TOC](#TOC) + +
## Test suite The test suite itself uses the builtin `unittest` module of Python to discover @@ -119,12 +130,14 @@ group test cases. ### Configuration +
#### INI style The test suite is configured via `INI` style files using Python's builtin [ConfigParser](https://docs.python.org/3/library/configparser.html) module. Such a configuration file looks roughly like this: -``` ini + +```ini [DEFAULT] some_var: some_val @@ -157,19 +170,22 @@ documentation. The `ConfigParser` module is used with the following defaults: Please keep in mind that leading and trailing whitespaces are **stripped** from strings when extracting variable values. So this: -``` ini +```ini some_var: some value with whitespaces before and after ``` is equivalent to this: -``` ini +```ini some_var:some value with whitespaces before and after ``` +[TOC](#TOC) + +
#### Parameters The test suite's configuration file should have the following form: -``` ini +```ini [General] timeout: 0.1 @@ -208,7 +224,7 @@ extensions). For this, the test suite supports the `ENV` and `ENV fallback` sections. In conjunction with the extended interpolation of the `ConfigParser` module, these can be quite useful. Consider the following example: -``` ini +```ini [General] timeout: 0.1 @@ -257,6 +273,9 @@ Please note that while the `INI` file allows for variables with whitespaces or variable names in Python. +[TOC](#TOC) + +
### Test cases The test cases are defined in Python source files utilizing the unittest module, @@ -274,7 +293,7 @@ user. An example test case file would look like this: -``` python +```python # -*- coding: utf-8 -*- import system_tests @@ -324,7 +343,7 @@ possible to access `abort_exit_value` via `system_tests.abort_exit_value` The substitution of values is performed using the template module from Python's string library via `safe_substitute`. In the above example the command would thus expand to: -``` shell +```bash /path/to/the/dir/build/bin/binary -c /path/to/the/dir/conf/main.cfg -i invalid_input_file ``` and similarly for `stdout` and `stderr`. @@ -341,6 +360,9 @@ should be run and all return 0, one can write `retval = 10 * [0]` instead of writing 0 ten times. The same is of course possible for strings. +[TOC](#TOC) + +
### Multiline strings It is generally recommended to use Python's multiline strings (strings starting @@ -354,7 +376,7 @@ strings start and end with a single `"` but multiline strings start with three `"`. Also, while the variable names must be indented, new lines in multiline strings must not or additional whitespaces will be added. E.g.: -``` python +```python stderr = [ """something else""" @@ -376,6 +398,9 @@ as the indentation might have suggested. Also note that in this example the string will not be terminated with a newline character. To achieve that put the `"""` on the following line. +[TOC](#TOC) + +
### Paths Some test cases require the specification of paths (e.g. to the location of test @@ -388,7 +413,7 @@ quite verbose. A slightly simpler alternative is the function `path` from `system_tests` which converts all `/` inside your string into the platform's default path separator: -``` python +```python # -*- coding: utf-8 -*- from system_tests import CaseMeta, path @@ -402,12 +427,16 @@ class AnInformativeName(metaclass=CaseMeta): ``` +[TOC](#TOC) + +
## Advanced test cases This section describes more advanced features that are probably not necessary the "standard" usage of the test suite. +
### Providing standard input to commands The test suite supports providing a standard input to commands in a similar @@ -415,7 +444,8 @@ fashion as the standard output and error are specified: it expects a list (with the length equal to the number of commands) of standard inputs (either strings or `bytes`). For commands that expect no standard input, simply set the respective entry to `None`: -``` python + +```python # -*- coding: utf-8 -*- import system_tests @@ -450,6 +480,9 @@ If all commands don't expect any standard input, omit the attribute `stdin`, the test suite will implicitly assume `None` for every command. +[TOC](#TOC) + +
### Using a different output encoding The test suite will try to interpret the program's output as utf-8 encoded @@ -458,7 +491,8 @@ strings and if that fails it will try the `iso-8859-1` encoding (also know as If the tested program outputs characters in another encoding then it can be supplied as the `encodings` parameter in each test case: -``` python + +```python # -*- coding: utf-8 -*- import system_tests @@ -489,6 +523,9 @@ encodings can be found [here](https://docs.python.org/3/library/codecs.html#standard-encodings). +[TOC](#TOC) + +
### Working with binary output Some programs output binary data directly to stdout or stderr. Such programs can @@ -497,7 +534,8 @@ be also tested by specifying the type `bytes` as the only member in the string. An example test case would look like this: -``` python + +```python # -*- coding: utf-8 -*- import system_tests @@ -524,7 +562,8 @@ Using the bytes encoding has the following limitations: The test suite supports setting or modifying environment variables for individual test cases. This can be accomplished by adding a member dictionary named `env` with the appropriate variable names and keys: -``` python + +```python # -*- coding: utf-8 -*- from system_tests import CaseMeta, path @@ -553,6 +592,9 @@ overridden). If no variables should be inherited set `inherit_env` to `False` and your test case will get only the specified environment variables. +[TOC](#TOC) + +
### Creating file copies For tests that modify their input file it is useful to run these with a @@ -562,7 +604,7 @@ and deletes the copies after the test ran. Example: -``` python +```python # -*- coding: utf-8 -*- import system_tests @@ -591,6 +633,9 @@ course expanded too) named `invalid_input_file_copy` and deleted. Please note that variable expansion in the filenames is possible. +[TOC](#TOC) + +
### Customizing the output check Some tests do not require a "brute-force" comparison of the whole output of a @@ -600,7 +645,7 @@ cases, one can customize how stdout and stderr checked for errors. The `system_tests.Case` class has two public functions for the check of stdout & stderr: `compare_stdout` & `compare_stderr`. They have the following interface: -``` python +```python compare_stdout(self, i, command, got_stdout, expected_stdout) compare_stderr(self, i, command, got_stderr, expected_stderr) ``` @@ -626,7 +671,7 @@ the obtained output to standard error **and nothing else**. This is useful for test cases where stderr is filled with warnings that are not worth being tracked by the test suite. It can be used in the following way: -``` python +```python # -*- coding: utf-8 -*- import system_tests @@ -646,6 +691,9 @@ class AnInformativeName(metaclass=system_tests.CaseMeta): ``` +[TOC](#TOC) + +
### Running all commands under valgrind The test suite can run all commands under a memory checker like @@ -656,11 +704,13 @@ checking tool. The test suite will then prefix **all** commands with the specified command. For example this configuration file: -``` ini + +```ini [General] timeout: 0.1 memcheck: valgrind --quiet ``` + will result in every command specified in the test cases being run as `valgrind --quiet $command`. @@ -683,6 +733,9 @@ into account: collect test coverage). +[TOC](#TOC) + +
### Manually expanding variables in strings In case completely custom checks have to be run but one still wants to access @@ -693,7 +746,7 @@ variable substitution using the test suite's configuration file. Unfortunately, it has to run in a class member function. The `setUp()` function can be used for this, as it is run before each test. For example like this: -``` python +```python class SomeName(metaclass=system_tests.CaseMeta): def setUp(self): @@ -708,7 +761,7 @@ This example will work, as the test runner reads the data for `commands`, work is creating a new member in `setUp()` and trying to use it as a variable for expansion, like this: -``` python +```python class SomeName(metaclass=system_tests.CaseMeta): def setUp(self): @@ -721,7 +774,8 @@ static class members (which `new_var` is not). Also, if you modify a static class member in `setUp()` the changed version will **not** be used for variable expansion, as the variables are saved in a new dictionary **before** `setUp()` runs. Thus this: -``` python + +```python class SomeName(metaclass=system_tests.CaseMeta): new_var = "foo" @@ -734,13 +788,16 @@ class SomeName(metaclass=system_tests.CaseMeta): will result in `another_string` being "foo" and not "bar". +[TOC](#TOC) + +
### Hooks The `Case` class provides two hooks that are run after each command and after all commands, respectively. The hook which is run after each successful command has the following signature: -``` Python +```python post_command_hook(self, i, command) ``` with the following parameters: @@ -749,7 +806,7 @@ with the following parameters: The hook which is run after all test takes no parameters except `self`: -``` Python +```python post_tests_hook(self) ``` @@ -757,7 +814,7 @@ By default, these hooks do nothing. They can be used to implement custom checks after certain commands, e.g. to check if a file was created. Such a test can be implemented as follows: -``` Python +```python # -*- coding: utf-8 -*- import system_tests @@ -788,7 +845,12 @@ class AnInformativeName(metaclass=system_tests.CaseMeta): for expansion. +[TOC](#TOC) + +
## Bash test cases - Previously, Exiv2 had some bash test scripts, which were saved as the file `EXIV2_DIR/test/*.sh`. We're going to rewrite them as Python test scripts and save them to the directory `EXIV2_DIR/tests/bash_tests`. - These Python test scripts are based on [unittest](https://docs.python.org/3/library/unittest.html) and written in a common format, which is different from the format described in [Writing new tests](#writing-new-tests), but can be executed compatibly by `python3 runner.py`. + +[TOC](#TOC)