diff --git a/tests/doc.md b/tests/README.md similarity index 88% rename from tests/doc.md rename to tests/README.md index 67da03f2..089331ce 100644 --- a/tests/doc.md +++ b/tests/README.md @@ -1,7 +1,27 @@ -# TL;DR +- [Introduction](#introduction) + - [Running the test suite](#running-the-test-suite) + - [Writing new tests](#writing-new-tests) + - [Test suite](#test-suite) + - [Configuration](#configuration) + - [INI style](#ini-style) + - [Parameters](#parameters) + - [Test cases](#test-cases) + - [Multiline strings](#multiline-strings) + - [Paths](#paths) + - [Advanced test cases](#advanced-test-cases) + - [Providing standard input to commands](#providing-standard-input-to-commands) + - [Using a different output encoding](#using-a-different-output-encoding) + - [Working with binary output](#working-with-binary-output) + - [Setting and modifying environment variables](#setting-and-modifying-environment-variables) + - [Creating file copies](#creating-file-copies) + - [Customizing the output check](#customizing-the-output-check) + - [Running all commands under valgrind](#running-all-commands-under-valgrind) + - [Manually expanding variables in strings](#manually-expanding-variables-in-strings) + - [Hooks](#hooks) + - [Possible pitfalls](#possible-pitfalls) + - [Bash test cases](#bash-test-cases) + -If you just want to write a simple test case, check out the file -`writing_tests.md`. # Introduction @@ -11,6 +31,95 @@ especially useful for a regression test suite, but can be also used for testing of new features where unit testing is not feasible, e.g. to test new command line parameters. +## Running the test suite + +The test suite is written for Python 3 and is not compatible with Python 2, thus +it must be run with `python3` and not with `python` (which is usually an alias +for Python 2). + +Then navigate to the `tests/` subdirectory and run: +``` shell +python3 runner.py +``` + +One can supply the script with a directory where the suite should look for the +tests (it will search the directory recursively). If omitted, the runner will +look in the directory where the configuration file is located. It is also +possible to instead pass a file as the parameter, the test suite will then only +run the tests from this file. + +The runner script also supports the optional arguments `--config_file` which +allows to provide a different test suite configuration file than the default +`suite.conf`. It also forwards the verbosity setting via the `-v`/`--verbose` +flags to Python's unittest module. + +Optionally one can provide the `--debug` flag which will instruct test suite to +print all command invocations and all expected and obtained outputs to the +standard output. + +## Writing new tests + +The test suite is intended to run a binary and compare its standard output, +standard error and return value against provided values. This is implemented +using Python's [unittest](https://docs.python.org/3/library/unittest.html) module and thus all test files are Python files. + +The simplest test has the following structure: +``` python +# -*- coding: utf-8 -*- + +import system_tests + + +class GoodTestName(metaclass=system_tests.CaseMeta): + + filename = "$data_path/test_file" + commands = ["$exiv2 $filename", "$exiv2 $filename" + '_2'] + stdout = [""] * 2 + stderr = ["""$exiv2_exception_msg $filename: +$kerFailedToReadImageData +"""] * 2 + retval = [1] * 2 +``` + +The test suite will run the provided commands in `commands` and compare them to +the output in `stdout` and `stderr` and it will compare the return values. + +The strings after a `$` are variables either defined in this test's class or are +taken from the suite's configuration file (see `doc.md` for a complete +explanation). + +When creating new tests, follow roughly these steps: + +1. Choose an appropriate subdirectory where the test belongs. If none fits + create a new one and put an empty `__init__.py` file there. + +2. Create a new file with a name matching `test_*.py`. Copy the class definition + from the above example and choose an appropriate class name. + +3. Run the test suite via `python3 runner.py` and ensure that your test case is + actually run! Either run the suite with the `-v` option which will output all + test cases that were run or simply add an error and check if errors occur. + +## Test suite + +The test suite itself uses the builtin `unittest` module of Python to discover +and run the individual test cases. The test cases themselves are implemented in +Python source files, but the required Python knowledge is minimal. + +The test suite is configured via one configuration file whose location +automatically sets the root directory of the test suite. The `unittest` module +then recursively searches all sub-directories with a `__init__.py` file for +files of the form `test_*.py`, which it automatically interprets as test cases +(more about these in the next section). Python will automatically interpret each +directory as a module and use this to format the output, e.g. the test case +`regression/crashes/test_bug_15.py` will be interpreted as the module +`regression.crashes.test_bug_15`. Thus one can use the directory structure to +group test cases. + +### Configuration + +#### INI style + The test suite is configured via `INI` style files using Python's builtin [ConfigParser](https://docs.python.org/3/library/configparser.html) module. Such a configuration file looks roughly like this: @@ -55,21 +164,7 @@ is equivalent to this: some_var:some value with whitespaces before and after ``` -The test suite itself uses the builtin `unittest` module of Python to discover -and run the individual test cases. The test cases themselves are implemented in -Python source files, but the required Python knowledge is minimal. - -## Test suite - -The test suite is configured via one configuration file whose location -automatically sets the root directory of the test suite. The `unittest` module -then recursively searches all sub-directories with a `__init__.py` file for -files of the form `test_*.py`, which it automatically interprets as test cases -(more about these in the next section). Python will automatically interpret each -directory as a module and use this to format the output, e.g. the test case -`regression/crashes/test_bug_15.py` will be interpreted as the module -`regression.crashes.test_bug_15`. Thus one can use the directory structure to -group test cases. +#### Parameters The test suite's configuration file should have the following form: @@ -161,7 +256,7 @@ Please note that while the `INI` file allows for variables with whitespaces or variable names in Python. -## Test cases +### Test cases The test cases are defined in Python source files utilizing the unittest module, thus every file must also be a valid Python file. Each file defining a test case @@ -692,28 +787,7 @@ class AnInformativeName(metaclass=system_tests.CaseMeta): for expansion. -## Running the test suite +## Bash test cases -The test suite is written for Python 3 and is not compatible with Python 2, thus -it must be run with `python3` and not with `python` (which is usually an alias -for Python 2). - -Then navigate to the `tests/` subdirectory and run: -``` shell -python3 runner.py -``` - -One can supply the script with a directory where the suite should look for the -tests (it will search the directory recursively). If omitted, the runner will -look in the directory where the configuration file is located. It is also -possible to instead pass a file as the parameter, the test suite will then only -run the tests from this file. - -The runner script also supports the optional arguments `--config_file` which -allows to provide a different test suite configuration file than the default -`suite.conf`. It also forwards the verbosity setting via the `-v`/`--verbose` -flags to Python's unittest module. - -Optionally one can provide the `--debug` flag which will instruct test suite to -print all command invocations and all expected and obtained outputs to the -standard output. +- Previously, Exiv2 had some bash test scripts, which were saved as the file `EXIV2_DIR/test/*.sh`. We're going to rewrite them as Python test scripts and save them to the directory `EXIV2_DIR/tests/bash_test`. +- These Python test scripts are based on [unittest](https://docs.python.org/3/library/unittest.html) and written in a common format, which is different from the format described in [Writing new tests](#writing-new-tests), but can be executed compatibly by `python3 runner.py`. diff --git a/tests/WRITING_TESTS.md b/tests/WRITING_TESTS.md deleted file mode 100644 index 1f9a2377..00000000 --- a/tests/WRITING_TESTS.md +++ /dev/null @@ -1,42 +0,0 @@ -## Writing new tests - -The test suite is intended to run a binary and compare its standard output, -standard error and return value against provided values. This is implemented -using Python's `unittest` module and thus all test files are Python files. - -The simplest test has the following structure: -``` python -# -*- coding: utf-8 -*- - -import system_tests - - -class GoodTestName(metaclass=system_tests.CaseMeta): - - filename = "$data_path/test_file" - commands = ["$exiv2 $filename", "$exiv2 $filename" + '_2'] - stdout = [""] * 2 - stderr = ["""$exiv2_exception_msg $filename: -$kerFailedToReadImageData -"""] * 2 - retval = [1] * 2 -``` - -The test suite will run the provided commands in `commands` and compare them to -the output in `stdout` and `stderr` and it will compare the return values. - -The strings after a `$` are variables either defined in this test's class or are -taken from the suite's configuration file (see `doc.md` for a complete -explanation). - -When creating new tests, follow roughly these steps: - -1. Choose an appropriate subdirectory where the test belongs. If none fits - create a new one and put an empty `__init__.py` file there. - -2. Create a new file with a name matching `test_*.py`. Copy the class definition - from the above example and choose an appropriate class name. - -3. Run the test suite via `python3 runner.py` and ensure that your test case is - actually run! Either run the suite with the `-v` option which will output all - test cases that were run or simply add an error and check if errors occur. diff --git a/tests/bash_test/README.md b/tests/bash_test/README.md deleted file mode 100644 index 103490dc..00000000 --- a/tests/bash_test/README.md +++ /dev/null @@ -1,21 +0,0 @@ -# README - -- Plan to convert the bash test scripts `../../test/*.sh` into Python scripts and save them to this directory. -- The test cases in this directory are based on [unittest](https://docs.python.org/3/library/unittest.html) and written in a common format, which is different from the format described in `../writing_tests.md`, but can be executed compatibly. - - -## Running the test cases - -- Use runner.py to collect test cases and execute them: - ```sh - cd EXIV2_DIR/tests - python3 runner.py -v unit_test - ``` - -- Also, you can use pytest to execute the test cases for unittest, so that the test report looks better: - ```sh - cd EXIV2_DIR/tests - pytest -v unit_test - ``` - -- If no exception occurs during the execution of the test case, it passes. Otherwise it fails. diff --git a/tests/bash_test/utils.py b/tests/bash_test/utils.py index 30d1325b..ead606fe 100644 --- a/tests/bash_test/utils.py +++ b/tests/bash_test/utils.py @@ -13,7 +13,6 @@ EXIV2_DIR = os.path.normpath(os.path.join(os.path.abspath(__file__), '../../../' BIN_DIR = os.path.join(EXIV2_DIR, 'build/bin') DATA_DIR = os.path.join(EXIV2_DIR, 'test/data') TEST_DIR = os.path.join(EXIV2_DIR, 'test/tmp') -BIN_SUFFIX = '' # TODO: Determine if the suffix is '.exe' def log_info(s):