xymp
/
exiv2
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Enhanced documentation formatting.

Browse Source
main
Robin Mills 5 years ago
parent dbf2c1c52d
commit e60710d925
1 changed files with 89 additions and 27 deletions
Show Stats Download Patch File Download Diff File

76
tests/README-TESTS.md
Unescape Escape View File

@ -1,3 +1,5 @@
<div id="TOC"/>
- [README-TESTS](#README-TESTS) - [README-TESTS](#README-TESTS)
- [Running the test suite](#running-the-test-suite) - [Running the test suite](#running-the-test-suite)
- [Writing new tests](#writing-new-tests) - [Writing new tests](#writing-new-tests)
@ -16,13 +18,13 @@
- [Creating file copies](#creating-file-copies) - [Creating file copies](#creating-file-copies)
- [Customizing the output check](#customizing-the-output-check) - [Customizing the output check](#customizing-the-output-check)
- [Running all commands under valgrind](#running-all-commands-under-valgrind) - [Running all commands under valgrind](#running-all-commands-under-valgrind)
- [Manually expanding variables in strings](#manually-expanding-variables-in-strings) - [Manually expanding variables in strings](#manually-expanding-variables)
- [Hooks](#hooks) - [Hooks](#hooks)
- [Possible pitfalls](#possible-pitfalls) - [Possible pitfalls](#possible-pitfalls)
- [Bash test cases](#bash-test-cases) - [Bash test cases](#bash-test-cases)
<!-- The TOC is generated by Markdown Preview Enhanced -->
<div id="README-TESTS"/>
# README-TESTS # README-TESTS
This test suite is intended for system tests, i.e. for running a binary with This test suite is intended for system tests, i.e. for running a binary with
@ -31,6 +33,7 @@ especially useful for a regression test suite, but can be also used for testing
of new features where unit testing is not feasible, e.g. to test new command of new features where unit testing is not feasible, e.g. to test new command
line parameters. line parameters.
<div id="running-the-test-suite"/>
## Running the test suite ## Running the test suite
The test suite is written for Python 3 and is not compatible with Python 2, thus The test suite is written for Python 3 and is not compatible with Python 2, thus
@ -38,7 +41,8 @@ it must be run with `python3` and not with `python` (which is usually an alias
for Python 2). for Python 2).
Then navigate to the `tests/` subdirectory and run: Then navigate to the `tests/` subdirectory and run:
``` shell
```bash
python3 -m pip install -r requirements.txt python3 -m pip install -r requirements.txt
python3 runner.py python3 runner.py
``` ```
@ -58,6 +62,9 @@ Optionally one can provide the `--debug` flag which will instruct test suite to
print all command invocations and all expected and obtained outputs to the print all command invocations and all expected and obtained outputs to the
standard output. standard output.
[TOC](#TOC)
<div id="writing-new-tests"/>
## Writing new tests ## Writing new tests
The test suite is intended to run a binary and compare its standard output, The test suite is intended to run a binary and compare its standard output,
@ -65,6 +72,7 @@ standard error and return value against provided values. This is implemented
using Python's [unittest](https://docs.python.org/3/library/unittest.html) module and thus all test files are Python files. using Python's [unittest](https://docs.python.org/3/library/unittest.html) module and thus all test files are Python files.
The simplest test has the following structure: The simplest test has the following structure:
```python ```python
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
@ -101,6 +109,9 @@ When creating new tests, follow roughly these steps:
actually run! Either run the suite with the `-v` option which will output all actually run! Either run the suite with the `-v` option which will output all
test cases that were run or simply add an error and check if errors occur. test cases that were run or simply add an error and check if errors occur.
[TOC](#TOC)
<div id="test-suite"/>
## Test suite ## Test suite
The test suite itself uses the builtin `unittest` module of Python to discover The test suite itself uses the builtin `unittest` module of Python to discover
@ -119,11 +130,13 @@ group test cases.
### Configuration ### Configuration
<div id="ini-style"/>
#### INI style #### INI style
The test suite is configured via `INI` style files using Python's builtin The test suite is configured via `INI` style files using Python's builtin
[ConfigParser](https://docs.python.org/3/library/configparser.html) [ConfigParser](https://docs.python.org/3/library/configparser.html)
module. Such a configuration file looks roughly like this: module. Such a configuration file looks roughly like this:
```ini ```ini
[DEFAULT] [DEFAULT]
some_var: some_val some_var: some_val
@ -165,6 +178,9 @@ is equivalent to this:
some_var:some value with whitespaces before and after some_var:some value with whitespaces before and after
``` ```
[TOC](#TOC)
<div id="parameters"/>
#### Parameters #### Parameters
The test suite's configuration file should have the following form: The test suite's configuration file should have the following form:
@ -257,6 +273,9 @@ Please note that while the `INI` file allows for variables with whitespaces or
variable names in Python. variable names in Python.
[TOC](#TOC)
<div id="test-cases"/>
### Test cases ### Test cases
The test cases are defined in Python source files utilizing the unittest module, The test cases are defined in Python source files utilizing the unittest module,
@ -324,7 +343,7 @@ possible to access `abort_exit_value` via `system_tests.abort_exit_value`
The substitution of values is performed using the template module from Python's The substitution of values is performed using the template module from Python's
string library via `safe_substitute`. In the above example the command would string library via `safe_substitute`. In the above example the command would
thus expand to: thus expand to:
``` shell ```bash
/path/to/the/dir/build/bin/binary -c /path/to/the/dir/conf/main.cfg -i invalid_input_file /path/to/the/dir/build/bin/binary -c /path/to/the/dir/conf/main.cfg -i invalid_input_file
``` ```
and similarly for `stdout` and `stderr`. and similarly for `stdout` and `stderr`.
@ -341,6 +360,9 @@ should be run and all return 0, one can write `retval = 10 * [0]` instead of
writing 0 ten times. The same is of course possible for strings. writing 0 ten times. The same is of course possible for strings.
[TOC](#TOC)
<div id="multiline-strings"/>
### Multiline strings ### Multiline strings
It is generally recommended to use Python's multiline strings (strings starting It is generally recommended to use Python's multiline strings (strings starting
@ -376,6 +398,9 @@ as the indentation might have suggested.
Also note that in this example the string will not be terminated with a newline Also note that in this example the string will not be terminated with a newline
character. To achieve that put the `"""` on the following line. character. To achieve that put the `"""` on the following line.
[TOC](#TOC)
<div id="paths"/>
### Paths ### Paths
Some test cases require the specification of paths (e.g. to the location of test Some test cases require the specification of paths (e.g. to the location of test
@ -402,12 +427,16 @@ class AnInformativeName(metaclass=CaseMeta):
``` ```
[TOC](#TOC)
<div id="advanced-test-cases"/>
## Advanced test cases ## Advanced test cases
This section describes more advanced features that are probably not necessary This section describes more advanced features that are probably not necessary
the "standard" usage of the test suite. the "standard" usage of the test suite.
<div id="providing-standard-input-to-commands"/>
### Providing standard input to commands ### Providing standard input to commands
The test suite supports providing a standard input to commands in a similar The test suite supports providing a standard input to commands in a similar
@ -415,6 +444,7 @@ fashion as the standard output and error are specified: it expects a list (with
the length equal to the number of commands) of standard inputs (either strings the length equal to the number of commands) of standard inputs (either strings
or `bytes`). For commands that expect no standard input, simply set the or `bytes`). For commands that expect no standard input, simply set the
respective entry to `None`: respective entry to `None`:
```python ```python
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
@ -450,6 +480,9 @@ If all commands don't expect any standard input, omit the attribute `stdin`, the
test suite will implicitly assume `None` for every command. test suite will implicitly assume `None` for every command.
[TOC](#TOC)
<div id="using-a-different-output-encoding"/>
### Using a different output encoding ### Using a different output encoding
The test suite will try to interpret the program's output as utf-8 encoded The test suite will try to interpret the program's output as utf-8 encoded
@ -458,6 +491,7 @@ strings and if that fails it will try the `iso-8859-1` encoding (also know as
If the tested program outputs characters in another encoding then it can be If the tested program outputs characters in another encoding then it can be
supplied as the `encodings` parameter in each test case: supplied as the `encodings` parameter in each test case:
```python ```python
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
@ -489,6 +523,9 @@ encodings can be found
[here](https://docs.python.org/3/library/codecs.html#standard-encodings). [here](https://docs.python.org/3/library/codecs.html#standard-encodings).
[TOC](#TOC)
<div id="working-with-binary-output"/>
### Working with binary output ### Working with binary output
Some programs output binary data directly to stdout or stderr. Such programs can Some programs output binary data directly to stdout or stderr. Such programs can
@ -497,6 +534,7 @@ be also tested by specifying the type `bytes` as the only member in the
string. string.
An example test case would look like this: An example test case would look like this:
```python ```python
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
@ -524,6 +562,7 @@ Using the bytes encoding has the following limitations:
The test suite supports setting or modifying environment variables for The test suite supports setting or modifying environment variables for
individual test cases. This can be accomplished by adding a member dictionary individual test cases. This can be accomplished by adding a member dictionary
named `env` with the appropriate variable names and keys: named `env` with the appropriate variable names and keys:
```python ```python
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
@ -553,6 +592,9 @@ overridden). If no variables should be inherited set `inherit_env` to `False`
and your test case will get only the specified environment variables. and your test case will get only the specified environment variables.
[TOC](#TOC)
<div id="creating-file-copies"/>
### Creating file copies ### Creating file copies
For tests that modify their input file it is useful to run these with a For tests that modify their input file it is useful to run these with a
@ -591,6 +633,9 @@ course expanded too) named `invalid_input_file_copy` and
deleted. Please note that variable expansion in the filenames is possible. deleted. Please note that variable expansion in the filenames is possible.
[TOC](#TOC)
<div id="customizing-the-output-check"/>
### Customizing the output check ### Customizing the output check
Some tests do not require a "brute-force" comparison of the whole output of a Some tests do not require a "brute-force" comparison of the whole output of a
@ -646,6 +691,9 @@ class AnInformativeName(metaclass=system_tests.CaseMeta):
``` ```
[TOC](#TOC)
<div id="running-all-commands-under-valgrind"/>
### Running all commands under valgrind ### Running all commands under valgrind
The test suite can run all commands under a memory checker like The test suite can run all commands under a memory checker like
@ -656,11 +704,13 @@ checking tool. The test suite will then prefix **all** commands with the
specified command. specified command.
For example this configuration file: For example this configuration file:
```ini ```ini
[General] [General]
timeout: 0.1 timeout: 0.1
memcheck: valgrind --quiet memcheck: valgrind --quiet
``` ```
will result in every command specified in the test cases being run as `valgrind will result in every command specified in the test cases being run as `valgrind
--quiet $command`. --quiet $command`.
@ -683,6 +733,9 @@ into account:
collect test coverage). collect test coverage).
[TOC](#TOC)
<div id="manually-expanding-variables"/>
### Manually expanding variables in strings ### Manually expanding variables in strings
In case completely custom checks have to be run but one still wants to access In case completely custom checks have to be run but one still wants to access
@ -721,6 +774,7 @@ static class members (which `new_var` is not). Also, if you modify a static
class member in `setUp()` the changed version will **not** be used for variable class member in `setUp()` the changed version will **not** be used for variable
expansion, as the variables are saved in a new dictionary **before** `setUp()` expansion, as the variables are saved in a new dictionary **before** `setUp()`
runs. Thus this: runs. Thus this:
```python ```python
class SomeName(metaclass=system_tests.CaseMeta): class SomeName(metaclass=system_tests.CaseMeta):
@ -734,13 +788,16 @@ class SomeName(metaclass=system_tests.CaseMeta):
will result in `another_string` being "foo" and not "bar". will result in `another_string` being "foo" and not "bar".
[TOC](#TOC)
<div id="hooks"/>
### Hooks ### Hooks
The `Case` class provides two hooks that are run after each command and after The `Case` class provides two hooks that are run after each command and after
all commands, respectively. The hook which is run after each successful command all commands, respectively. The hook which is run after each successful command
has the following signature: has the following signature:
``` Python ```python
post_command_hook(self, i, command) post_command_hook(self, i, command)
``` ```
with the following parameters: with the following parameters:
@ -749,7 +806,7 @@ with the following parameters:
The hook which is run after all test takes no parameters except `self`: The hook which is run after all test takes no parameters except `self`:
``` Python ```python
post_tests_hook(self) post_tests_hook(self)
``` ```
@ -757,7 +814,7 @@ By default, these hooks do nothing. They can be used to implement custom checks
after certain commands, e.g. to check if a file was created. Such a test can be after certain commands, e.g. to check if a file was created. Such a test can be
implemented as follows: implemented as follows:
``` Python ```python
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
import system_tests import system_tests
@ -788,7 +845,12 @@ class AnInformativeName(metaclass=system_tests.CaseMeta):
for expansion. for expansion.
[TOC](#TOC)
<div id="bash-test-cases"/>
## Bash test cases ## Bash test cases
- Previously, Exiv2 had some bash test scripts, which were saved as the file `EXIV2_DIR/test/*.sh`. We're going to rewrite them as Python test scripts and save them to the directory `EXIV2_DIR/tests/bash_tests`. - Previously, Exiv2 had some bash test scripts, which were saved as the file `EXIV2_DIR/test/*.sh`. We're going to rewrite them as Python test scripts and save them to the directory `EXIV2_DIR/tests/bash_tests`.
- These Python test scripts are based on [unittest](https://docs.python.org/3/library/unittest.html) and written in a common format, which is different from the format described in [Writing new tests](#writing-new-tests), but can be executed compatibly by `python3 runner.py`. - These Python test scripts are based on [unittest](https://docs.python.org/3/library/unittest.html) and written in a common format, which is different from the format described in [Writing new tests](#writing-new-tests), but can be executed compatibly by `python3 runner.py`.
[TOC](#TOC)

Write Preview
Loading…
Cancel
Save