Some programs output binary data directly to stdout or stderr. Such programs can
@ -497,7 +534,8 @@ be also tested by specifying the type `bytes` as the only member in the
string.
An example test case would look like this:
``` python
```python
# -*- coding: utf-8 -*-
import system_tests
@ -524,7 +562,8 @@ Using the bytes encoding has the following limitations:
The test suite supports setting or modifying environment variables for
individual test cases. This can be accomplished by adding a member dictionary
named `env` with the appropriate variable names and keys:
``` python
```python
# -*- coding: utf-8 -*-
from system_tests import CaseMeta, path
@ -553,6 +592,9 @@ overridden). If no variables should be inherited set `inherit_env` to `False`
and your test case will get only the specified environment variables.
[TOC](#TOC)
<divid="creating-file-copies"/>
### Creating file copies
For tests that modify their input file it is useful to run these with a
@ -562,7 +604,7 @@ and deletes the copies after the test ran.
Example:
```python
```python
# -*- coding: utf-8 -*-
import system_tests
@ -591,6 +633,9 @@ course expanded too) named `invalid_input_file_copy` and
deleted. Please note that variable expansion in the filenames is possible.
[TOC](#TOC)
<divid="customizing-the-output-check"/>
### Customizing the output check
Some tests do not require a "brute-force" comparison of the whole output of a
@ -600,7 +645,7 @@ cases, one can customize how stdout and stderr checked for errors.
The `system_tests.Case` class has two public functions for the check of stdout &
stderr: `compare_stdout`&`compare_stderr`. They have the following interface:
```python
```python
compare_stdout(self, i, command, got_stdout, expected_stdout)
compare_stderr(self, i, command, got_stderr, expected_stderr)
```
@ -626,7 +671,7 @@ the obtained output to standard error **and nothing else**. This is useful for
test cases where stderr is filled with warnings that are not worth being tracked
by the test suite. It can be used in the following way:
```python
```python
# -*- coding: utf-8 -*-
import system_tests
@ -646,6 +691,9 @@ class AnInformativeName(metaclass=system_tests.CaseMeta):
```
[TOC](#TOC)
<divid="running-all-commands-under-valgrind"/>
### Running all commands under valgrind
The test suite can run all commands under a memory checker like
@ -656,11 +704,13 @@ checking tool. The test suite will then prefix **all** commands with the
specified command.
For example this configuration file:
``` ini
```ini
[General]
timeout: 0.1
memcheck: valgrind --quiet
```
will result in every command specified in the test cases being run as `valgrind
--quiet $command`.
@ -683,6 +733,9 @@ into account:
collect test coverage).
[TOC](#TOC)
<divid="manually-expanding-variables"/>
### Manually expanding variables in strings
In case completely custom checks have to be run but one still wants to access
@ -693,7 +746,7 @@ variable substitution using the test suite's configuration file.
Unfortunately, it has to run in a class member function. The `setUp()` function
can be used for this, as it is run before each test. For example like this:
```python
```python
class SomeName(metaclass=system_tests.CaseMeta):
def setUp(self):
@ -708,7 +761,7 @@ This example will work, as the test runner reads the data for `commands`,
work is creating a new member in `setUp()` and trying to use it as a variable
for expansion, like this:
```python
```python
class SomeName(metaclass=system_tests.CaseMeta):
def setUp(self):
@ -721,7 +774,8 @@ static class members (which `new_var` is not). Also, if you modify a static
class member in `setUp()` the changed version will **not** be used for variable
expansion, as the variables are saved in a new dictionary **before**`setUp()`
runs. Thus this:
``` python
```python
class SomeName(metaclass=system_tests.CaseMeta):
new_var = "foo"
@ -734,13 +788,16 @@ class SomeName(metaclass=system_tests.CaseMeta):
will result in `another_string` being "foo" and not "bar".
[TOC](#TOC)
<divid="hooks"/>
### Hooks
The `Case` class provides two hooks that are run after each command and after
all commands, respectively. The hook which is run after each successful command
has the following signature:
``` Python
```python
post_command_hook(self, i, command)
```
with the following parameters:
@ -749,7 +806,7 @@ with the following parameters:
The hook which is run after all test takes no parameters except `self`:
``` Python
```python
post_tests_hook(self)
```
@ -757,7 +814,7 @@ By default, these hooks do nothing. They can be used to implement custom checks
after certain commands, e.g. to check if a file was created. Such a test can be
implemented as follows:
``` Python
```python
# -*- coding: utf-8 -*-
import system_tests
@ -788,7 +845,12 @@ class AnInformativeName(metaclass=system_tests.CaseMeta):
for expansion.
[TOC](#TOC)
<divid="bash-test-cases"/>
## Bash test cases
- Previously, Exiv2 had some bash test scripts, which were saved as the file `EXIV2_DIR/test/*.sh`. We're going to rewrite them as Python test scripts and save them to the directory `EXIV2_DIR/tests/bash_tests`.
- These Python test scripts are based on [unittest](https://docs.python.org/3/library/unittest.html) and written in a common format, which is different from the format described in [Writing new tests](#writing-new-tests), but can be executed compatibly by `python3 runner.py`.
Robin Mills
5 years ago