-## 4 Running the test suite
+## 4 Test Suite
+
+The test suite is implemented using CTest. CMake adds the target 'test' to the build. You can run ctest with the command `$ cmake --build . --target test`, or simply with `$ ctest`. The build creates 5 tests: bashTests, bugfixes, tiffTests, unit_tests and versionTest. You can run all tests or a subset.
+
+```bash
+$ cmake --build . --target test
+$ ctest # run all tests and display summary
+$ ctest --output-on-failure # run all tests and output failures
+$ ctest -R bugfixes # run only bugfixes and display summary
+$ ctest -R bugfixes --verbose # run only bugfixes and display all output
+```
-#### Different kinds of tests:
+#### Test Architecture
-| Description | Language | Location | Command
_(in build directory)_ | CMake Option to Build |
-|:-- |:-- |:-- |:-- |:-- |
-| Run all tests | | | $ make tests | |
-| Run all tests | | **Visual Studio Users** | > cmake --build . --target tests | |
-| Bash tests | python | tests/bash\_tests | $ make bash_tests | -DEXIV2\_BUILD\_SAMPLES=On |
-| Python tests | python | tests | $ make python_tests | -DEXIV2\_BUILD\_SAMPLES=On |
-| Unit tests | C++ | unitTests | $ make unit_test | -DEXIV2\_BUILD\_UNIT\_TESTS=On |
-| Version test | C++ | src/version.cpp | $ make version_test | Always in library |
+| Name | Language | Location | Command
_(in build directory)_ | CMake Option to Build |
+|:-- |:-- |:-- |:-- |:-- |
+| bashTests | python | tests/bash\_tests | $ ctest -R bash | -DEXIV2\_BUILD\_SAMPLES=On |
+| bugfixes | python | tests/bugfixes | $ ctest -R bugfixes | -DEXIV2\_BUILD\_SAMPLES=On |
+| tiffTests | python | tests/tiff_test | $ ctest -R tiff | -DEXIV2\_BUILD\_SAMPLES=On |
+| unit_tests | C++ | unitTests | $ ctest -R unit | -DEXIV2\_BUILD\_UNIT\_TESTS=On |
+| versionTest | C++ | src/version.cpp | $ ctest -R version | Always in library |
The term _**bash scripts**_ is historical. The implementation of the tests in this collection originally required bash. These
scripts have been rewritten in python. Visual Studio Users will appreciate the python implementation as it avoids the
@@ -899,17 +905,17 @@ The Variable EXIV2\_PORT or EXIV2\_HTTP can be set to None to skip http tests.
[TOC](#TOC)
-
-### 4.1 Running tests on a UNIX-like system
+### 4.1 Running tests on Unix-like systems
You can run tests directly from the build:
```bash
$ cmake .. -G "Unix Makefiles" -DEXIV2_BUILD_UNIT_TESTS=On
-$ make
-... lots of output ...
-$ make tests
+... lots of output and build summary ...
+$ cmake --build .
... lots of output ...
+$ cmake --build . --target test
+... test summary ...
$
```
@@ -917,13 +923,13 @@ You can run individual tests in the `test` directory. **Caution:** If you build
```bash
$ cd
/build
-$ make bash_tests
+$ ctest -R bash --verbose
addmoddel_test (testcases.TestCases) ... ok
....
Ran 176 tests in 9.526s
OK (skipped=6)
-$ make python_tests
+$ ctest -R bugfixes --verbose
... lots of output ...
test_run (tiff_test.test_tiff_test_program.TestTiffTestProg) ... ok
----------------------------------------------------------------------
@@ -934,10 +940,9 @@ $
[TOC](#TOC)
-
### 4.2 Running tests on Visual Studio builds from cmd.exe
-**Caution:** _The python3 interpreter must be on the PATH, build for DOS, and called python3.exe. I copied the python.exe program:
+**Caution:** _The python3 interpreter must be on the PATH, build for DOS, and called python3.exe._ I copied the python.exe program:
```cmd
> copy c:\Python37\python.exe c:\Python37\python3.exe
@@ -945,13 +950,11 @@ $
```
You can execute the test suite as described for UNIX-like systems.
-The main difference is that you must use cmake to initiate the test
-as make is not a system utility on Windows.
```bash
> cd
/build
-> cmake --build . --target tests
-> cmake --build . --target python_tests
+> cmake --build . --target test
+> ctest -R bugfixes --verbose
```
##### Running tests from cmd.exe
@@ -974,13 +977,12 @@ If you wish to use an environment variables, use set:
```
set VERBOSE=1
-cmake --build . --config Release --target tests
+cmake --build . --config Release --target test
set VERBOSE=
```
[TOC](#TOC)
-
### 4.3 Unit tests
The code for the unit tests is in `
/unitTests`. To include unit tests in the build, use the *cmake* option `-DEXIV2_BUILD_UNIT_TESTS=On`.
@@ -1000,24 +1002,23 @@ $ popd
[TOC](#TOC)
+### 4.4 Bugfix tests
-### 4.4 Python tests
-
-You can run the python tests from the build directory:
+You can run the bugfix tests from the build directory:
```bash
$ cd
/build
-$ make python_tests
+$ ctest -R bugfixes
```
If you wish to run in verbose mode:
```bash
$ cd /build
-$ make python_tests VERBOSE=1
+$ $ ctest -R bugfixes --verbose
```
-The python tests are stored in the directory tests and you can run them all with the command:
+The bugfix tests are stored in the directory tests and you can run them all with the command:
```bash
$ cd /tests
@@ -1036,24 +1037,23 @@ You may wish to get a brief summary of failures with commands such as:
```bash
$ cd /build
-$ make python_tests 2>&1 | grep FAIL
+$ ctest -R bugfixes --verbose 2>&1 | grep FAIL
```
[TOC](#TOC)
-
### 4.5 Test Summary
-| *Tests* | Unix Style Platforms _(bash)_ | Visual Studio _(cmd.exe)_ |
+| *Tests* | Execute using ctest | Execute using cmake |
|:-- |:--- |:-- |
-| | $ cd \
/build | \> cd \/build |
-| tests | $ make tests | \> cmake --build . --config Release --target tests |
-| bash_tests | $ make bash_tests | \> cmake --build . --config Release --target bash_tests |
-| python_tests | $ make python_tests | \> cmake --build . --config Release --target python_tests |
-| unit_test | $ make unit_test | \> cmake --build . --config Release --target unit_test |
-| version_test | $ make version_test | \> cmake --build . --config Release --target version_test |
-
-The name **bash_tests** is historical. They are implemented in python.
+| | $ cd \/build | \> cd \/build |
+| test | $ ctest | \> cmake --build . --config Release --target test |
+| bash_tests | $ ctest -R bash | \> cmake --build . --config Release --target bash_tests |
+| bugfixes | $ ctest -R bugfixes | \> cmake --build . --config Release --target bugfix_tests |
+| unit_test | $ ctest -R unit | \> cmake --build . --config Release --target unit_test |
+| version_test | $ ctest -R version | \> cmake --build . --config Release --target version_test |
+
+The name **bashTests** is historical. The tests are implemented in python.
[TOC](#TOC)