Some programs output binary data directly to stdout or stderr. Such programs can
Some programs output binary data directly to stdout or stderr. Such programs can
@ -497,7 +534,8 @@ be also tested by specifying the type `bytes` as the only member in the
string.
string.
An example test case would look like this:
An example test case would look like this:
``` python
```python
# -*- coding: utf-8 -*-
# -*- coding: utf-8 -*-
import system_tests
import system_tests
@ -524,7 +562,8 @@ Using the bytes encoding has the following limitations:
The test suite supports setting or modifying environment variables for
The test suite supports setting or modifying environment variables for
individual test cases. This can be accomplished by adding a member dictionary
individual test cases. This can be accomplished by adding a member dictionary
named `env` with the appropriate variable names and keys:
named `env` with the appropriate variable names and keys:
``` python
```python
# -*- coding: utf-8 -*-
# -*- coding: utf-8 -*-
from system_tests import CaseMeta, path
from system_tests import CaseMeta, path
@ -553,6 +592,9 @@ overridden). If no variables should be inherited set `inherit_env` to `False`
and your test case will get only the specified environment variables.
and your test case will get only the specified environment variables.
[TOC](#TOC)
<divid="creating-file-copies"/>
### Creating file copies
### Creating file copies
For tests that modify their input file it is useful to run these with a
For tests that modify their input file it is useful to run these with a
@ -562,7 +604,7 @@ and deletes the copies after the test ran.
Example:
Example:
```python
```python
# -*- coding: utf-8 -*-
# -*- coding: utf-8 -*-
import system_tests
import system_tests
@ -591,6 +633,9 @@ course expanded too) named `invalid_input_file_copy` and
deleted. Please note that variable expansion in the filenames is possible.
deleted. Please note that variable expansion in the filenames is possible.
[TOC](#TOC)
<divid="customizing-the-output-check"/>
### Customizing the output check
### Customizing the output check
Some tests do not require a "brute-force" comparison of the whole output of a
Some tests do not require a "brute-force" comparison of the whole output of a
@ -600,7 +645,7 @@ cases, one can customize how stdout and stderr checked for errors.
The `system_tests.Case` class has two public functions for the check of stdout &
The `system_tests.Case` class has two public functions for the check of stdout &
stderr: `compare_stdout`&`compare_stderr`. They have the following interface:
stderr: `compare_stdout`&`compare_stderr`. They have the following interface:
```python
```python
compare_stdout(self, i, command, got_stdout, expected_stdout)
compare_stdout(self, i, command, got_stdout, expected_stdout)
compare_stderr(self, i, command, got_stderr, expected_stderr)
compare_stderr(self, i, command, got_stderr, expected_stderr)
```
```
@ -626,7 +671,7 @@ the obtained output to standard error **and nothing else**. This is useful for
test cases where stderr is filled with warnings that are not worth being tracked
test cases where stderr is filled with warnings that are not worth being tracked
by the test suite. It can be used in the following way:
by the test suite. It can be used in the following way:
```python
```python
# -*- coding: utf-8 -*-
# -*- coding: utf-8 -*-
import system_tests
import system_tests
@ -646,6 +691,9 @@ class AnInformativeName(metaclass=system_tests.CaseMeta):
```
```
[TOC](#TOC)
<divid="running-all-commands-under-valgrind"/>
### Running all commands under valgrind
### Running all commands under valgrind
The test suite can run all commands under a memory checker like
The test suite can run all commands under a memory checker like
@ -656,11 +704,13 @@ checking tool. The test suite will then prefix **all** commands with the
specified command.
specified command.
For example this configuration file:
For example this configuration file:
``` ini
```ini
[General]
[General]
timeout: 0.1
timeout: 0.1
memcheck: valgrind --quiet
memcheck: valgrind --quiet
```
```
will result in every command specified in the test cases being run as `valgrind
will result in every command specified in the test cases being run as `valgrind
--quiet $command`.
--quiet $command`.
@ -683,6 +733,9 @@ into account:
collect test coverage).
collect test coverage).
[TOC](#TOC)
<divid="manually-expanding-variables"/>
### Manually expanding variables in strings
### Manually expanding variables in strings
In case completely custom checks have to be run but one still wants to access
In case completely custom checks have to be run but one still wants to access
@ -693,7 +746,7 @@ variable substitution using the test suite's configuration file.
Unfortunately, it has to run in a class member function. The `setUp()` function
Unfortunately, it has to run in a class member function. The `setUp()` function
can be used for this, as it is run before each test. For example like this:
can be used for this, as it is run before each test. For example like this:
```python
```python
class SomeName(metaclass=system_tests.CaseMeta):
class SomeName(metaclass=system_tests.CaseMeta):
def setUp(self):
def setUp(self):
@ -708,7 +761,7 @@ This example will work, as the test runner reads the data for `commands`,
work is creating a new member in `setUp()` and trying to use it as a variable
work is creating a new member in `setUp()` and trying to use it as a variable
for expansion, like this:
for expansion, like this:
```python
```python
class SomeName(metaclass=system_tests.CaseMeta):
class SomeName(metaclass=system_tests.CaseMeta):
def setUp(self):
def setUp(self):
@ -721,7 +774,8 @@ static class members (which `new_var` is not). Also, if you modify a static
class member in `setUp()` the changed version will **not** be used for variable
class member in `setUp()` the changed version will **not** be used for variable
expansion, as the variables are saved in a new dictionary **before**`setUp()`
expansion, as the variables are saved in a new dictionary **before**`setUp()`
runs. Thus this:
runs. Thus this:
``` python
```python
class SomeName(metaclass=system_tests.CaseMeta):
class SomeName(metaclass=system_tests.CaseMeta):
new_var = "foo"
new_var = "foo"
@ -734,13 +788,16 @@ class SomeName(metaclass=system_tests.CaseMeta):
will result in `another_string` being "foo" and not "bar".
will result in `another_string` being "foo" and not "bar".
[TOC](#TOC)
<divid="hooks"/>
### Hooks
### Hooks
The `Case` class provides two hooks that are run after each command and after
The `Case` class provides two hooks that are run after each command and after
all commands, respectively. The hook which is run after each successful command
all commands, respectively. The hook which is run after each successful command
has the following signature:
has the following signature:
``` Python
```python
post_command_hook(self, i, command)
post_command_hook(self, i, command)
```
```
with the following parameters:
with the following parameters:
@ -749,7 +806,7 @@ with the following parameters:
The hook which is run after all test takes no parameters except `self`:
The hook which is run after all test takes no parameters except `self`:
``` Python
```python
post_tests_hook(self)
post_tests_hook(self)
```
```
@ -757,7 +814,7 @@ By default, these hooks do nothing. They can be used to implement custom checks
after certain commands, e.g. to check if a file was created. Such a test can be
after certain commands, e.g. to check if a file was created. Such a test can be
implemented as follows:
implemented as follows:
``` Python
```python
# -*- coding: utf-8 -*-
# -*- coding: utf-8 -*-
import system_tests
import system_tests
@ -788,7 +845,12 @@ class AnInformativeName(metaclass=system_tests.CaseMeta):
for expansion.
for expansion.
[TOC](#TOC)
<divid="bash-test-cases"/>
## Bash test cases
## Bash test cases
- Previously, Exiv2 had some bash test scripts, which were saved as the file `EXIV2_DIR/test/*.sh`. We're going to rewrite them as Python test scripts and save them to the directory `EXIV2_DIR/tests/bash_tests`.
- Previously, Exiv2 had some bash test scripts, which were saved as the file `EXIV2_DIR/test/*.sh`. We're going to rewrite them as Python test scripts and save them to the directory `EXIV2_DIR/tests/bash_tests`.
- These Python test scripts are based on [unittest](https://docs.python.org/3/library/unittest.html) and written in a common format, which is different from the format described in [Writing new tests](#writing-new-tests), but can be executed compatibly by `python3 runner.py`.
- These Python test scripts are based on [unittest](https://docs.python.org/3/library/unittest.html) and written in a common format, which is different from the format described in [Writing new tests](#writing-new-tests), but can be executed compatibly by `python3 runner.py`.
Robin Mills
5 years ago