Enhanced documentation formatting.

main
Robin Mills 5 years ago
parent dbf2c1c52d
commit e60710d925

@ -1,3 +1,5 @@
<div id="TOC"/>
- [README-TESTS](#README-TESTS)
- [Running the test suite](#running-the-test-suite)
- [Writing new tests](#writing-new-tests)
@ -16,13 +18,13 @@
- [Creating file copies](#creating-file-copies)
- [Customizing the output check](#customizing-the-output-check)
- [Running all commands under valgrind](#running-all-commands-under-valgrind)
- [Manually expanding variables in strings](#manually-expanding-variables-in-strings)
- [Manually expanding variables in strings](#manually-expanding-variables)
- [Hooks](#hooks)
- [Possible pitfalls](#possible-pitfalls)
- [Bash test cases](#bash-test-cases)
<!-- The TOC is generated by Markdown Preview Enhanced -->
<div id="README-TESTS"/>
# README-TESTS
This test suite is intended for system tests, i.e. for running a binary with
@ -31,6 +33,7 @@ especially useful for a regression test suite, but can be also used for testing
of new features where unit testing is not feasible, e.g. to test new command
line parameters.
<div id="running-the-test-suite"/>
## Running the test suite
The test suite is written for Python 3 and is not compatible with Python 2, thus
@ -38,7 +41,8 @@ it must be run with `python3` and not with `python` (which is usually an alias
for Python 2).
Then navigate to the `tests/` subdirectory and run:
``` shell
```bash
python3 -m pip install -r requirements.txt
python3 runner.py
```
@ -58,6 +62,9 @@ Optionally one can provide the `--debug` flag which will instruct test suite to
print all command invocations and all expected and obtained outputs to the
standard output.
[TOC](#TOC)
<div id="writing-new-tests"/>
## Writing new tests
The test suite is intended to run a binary and compare its standard output,
@ -65,6 +72,7 @@ standard error and return value against provided values. This is implemented
using Python's [unittest](https://docs.python.org/3/library/unittest.html) module and thus all test files are Python files.
The simplest test has the following structure:
```python
# -*- coding: utf-8 -*-
@ -101,6 +109,9 @@ When creating new tests, follow roughly these steps:
actually run! Either run the suite with the `-v` option which will output all
test cases that were run or simply add an error and check if errors occur.
[TOC](#TOC)
<div id="test-suite"/>
## Test suite
The test suite itself uses the builtin `unittest` module of Python to discover
@ -119,11 +130,13 @@ group test cases.
### Configuration
<div id="ini-style"/>
#### INI style
The test suite is configured via `INI` style files using Python's builtin
[ConfigParser](https://docs.python.org/3/library/configparser.html)
module. Such a configuration file looks roughly like this:
```ini
[DEFAULT]
some_var: some_val
@ -165,6 +178,9 @@ is equivalent to this:
some_var:some value with whitespaces before and after
```
[TOC](#TOC)
<div id="parameters"/>
#### Parameters
The test suite's configuration file should have the following form:
@ -257,6 +273,9 @@ Please note that while the `INI` file allows for variables with whitespaces or
variable names in Python.
[TOC](#TOC)
<div id="test-cases"/>
### Test cases
The test cases are defined in Python source files utilizing the unittest module,
@ -324,7 +343,7 @@ possible to access `abort_exit_value` via `system_tests.abort_exit_value`
The substitution of values is performed using the template module from Python's
string library via `safe_substitute`. In the above example the command would
thus expand to:
``` shell
```bash
/path/to/the/dir/build/bin/binary -c /path/to/the/dir/conf/main.cfg -i invalid_input_file
```
and similarly for `stdout` and `stderr`.
@ -341,6 +360,9 @@ should be run and all return 0, one can write `retval = 10 * [0]` instead of
writing 0 ten times. The same is of course possible for strings.
[TOC](#TOC)
<div id="multiline-strings"/>
### Multiline strings
It is generally recommended to use Python's multiline strings (strings starting
@ -376,6 +398,9 @@ as the indentation might have suggested.
Also note that in this example the string will not be terminated with a newline
character. To achieve that put the `"""` on the following line.
[TOC](#TOC)
<div id="paths"/>
### Paths
Some test cases require the specification of paths (e.g. to the location of test
@ -402,12 +427,16 @@ class AnInformativeName(metaclass=CaseMeta):
```
[TOC](#TOC)
<div id="advanced-test-cases"/>
## Advanced test cases
This section describes more advanced features that are probably not necessary
the "standard" usage of the test suite.
<div id="providing-standard-input-to-commands"/>
### Providing standard input to commands
The test suite supports providing a standard input to commands in a similar
@ -415,6 +444,7 @@ fashion as the standard output and error are specified: it expects a list (with
the length equal to the number of commands) of standard inputs (either strings
or `bytes`). For commands that expect no standard input, simply set the
respective entry to `None`:
```python
# -*- coding: utf-8 -*-
@ -450,6 +480,9 @@ If all commands don't expect any standard input, omit the attribute `stdin`, the
test suite will implicitly assume `None` for every command.
[TOC](#TOC)
<div id="using-a-different-output-encoding"/>
### Using a different output encoding
The test suite will try to interpret the program's output as utf-8 encoded
@ -458,6 +491,7 @@ strings and if that fails it will try the `iso-8859-1` encoding (also know as
If the tested program outputs characters in another encoding then it can be
supplied as the `encodings` parameter in each test case:
```python
# -*- coding: utf-8 -*-
@ -489,6 +523,9 @@ encodings can be found
[here](https://docs.python.org/3/library/codecs.html#standard-encodings).
[TOC](#TOC)
<div id="working-with-binary-output"/>
### Working with binary output
Some programs output binary data directly to stdout or stderr. Such programs can
@ -497,6 +534,7 @@ be also tested by specifying the type `bytes` as the only member in the
string.
An example test case would look like this:
```python
# -*- coding: utf-8 -*-
@ -524,6 +562,7 @@ Using the bytes encoding has the following limitations:
The test suite supports setting or modifying environment variables for
individual test cases. This can be accomplished by adding a member dictionary
named `env` with the appropriate variable names and keys:
```python
# -*- coding: utf-8 -*-
@ -553,6 +592,9 @@ overridden). If no variables should be inherited set `inherit_env` to `False`
and your test case will get only the specified environment variables.
[TOC](#TOC)
<div id="creating-file-copies"/>
### Creating file copies
For tests that modify their input file it is useful to run these with a
@ -591,6 +633,9 @@ course expanded too) named `invalid_input_file_copy` and
deleted. Please note that variable expansion in the filenames is possible.
[TOC](#TOC)
<div id="customizing-the-output-check"/>
### Customizing the output check
Some tests do not require a "brute-force" comparison of the whole output of a
@ -646,6 +691,9 @@ class AnInformativeName(metaclass=system_tests.CaseMeta):
```
[TOC](#TOC)
<div id="running-all-commands-under-valgrind"/>
### Running all commands under valgrind
The test suite can run all commands under a memory checker like
@ -656,11 +704,13 @@ checking tool. The test suite will then prefix **all** commands with the
specified command.
For example this configuration file:
```ini
[General]
timeout: 0.1
memcheck: valgrind --quiet
```
will result in every command specified in the test cases being run as `valgrind
--quiet $command`.
@ -683,6 +733,9 @@ into account:
collect test coverage).
[TOC](#TOC)
<div id="manually-expanding-variables"/>
### Manually expanding variables in strings
In case completely custom checks have to be run but one still wants to access
@ -721,6 +774,7 @@ static class members (which `new_var` is not). Also, if you modify a static
class member in `setUp()` the changed version will **not** be used for variable
expansion, as the variables are saved in a new dictionary **before** `setUp()`
runs. Thus this:
```python
class SomeName(metaclass=system_tests.CaseMeta):
@ -734,13 +788,16 @@ class SomeName(metaclass=system_tests.CaseMeta):
will result in `another_string` being "foo" and not "bar".
[TOC](#TOC)
<div id="hooks"/>
### Hooks
The `Case` class provides two hooks that are run after each command and after
all commands, respectively. The hook which is run after each successful command
has the following signature:
``` Python
```python
post_command_hook(self, i, command)
```
with the following parameters:
@ -749,7 +806,7 @@ with the following parameters:
The hook which is run after all test takes no parameters except `self`:
``` Python
```python
post_tests_hook(self)
```
@ -757,7 +814,7 @@ By default, these hooks do nothing. They can be used to implement custom checks
after certain commands, e.g. to check if a file was created. Such a test can be
implemented as follows:
``` Python
```python
# -*- coding: utf-8 -*-
import system_tests
@ -788,7 +845,12 @@ class AnInformativeName(metaclass=system_tests.CaseMeta):
for expansion.
[TOC](#TOC)
<div id="bash-test-cases"/>
## Bash test cases
- Previously, Exiv2 had some bash test scripts, which were saved as the file `EXIV2_DIR/test/*.sh`. We're going to rewrite them as Python test scripts and save them to the directory `EXIV2_DIR/tests/bash_tests`.
- These Python test scripts are based on [unittest](https://docs.python.org/3/library/unittest.html) and written in a common format, which is different from the format described in [Writing new tests](#writing-new-tests), but can be executed compatibly by `python3 runner.py`.
[TOC](#TOC)

Loading…
Cancel
Save