CMake/Tests/RunCMake
Reiner Herrmann edcccde7d6 file: Sort GLOB results to make it deterministic (#14491)
Even though the `file(GLOB)` documentation specifically warns against
using it to collect a list of source files, projects often do it anyway.
Since it uses `readdir()`, the list of files will be unsorted.
This list is often passed directly to add_executable / add_library.
Linking binaries with an unsorted list will make it unreproducible,
which means that the produced binary will differ depending on the
unpredictable `readdir()` order.

To solve those reproducibility issues in a lot of programs (which don't
explicitly `list(SORT)` the list manually), sort the resulting list of
the `file(GLOB)` command.

A more detailed rationale about reproducible builds is available
[here](https://reproducible-builds.org/).
2016-05-17 10:12:11 -04:00
..
AutoExportDll Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
BuildDepends Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
CMP0004 Cygwin: Avoid legacy warnings in RunCMake.* tests 2013-08-13 09:50:15 -04:00
CMP0019 cmMakefile: Remove special handling of LINK_DIRECTORIES property. 2015-07-18 13:56:48 +02:00
CMP0022 Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
CMP0026 Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
CMP0027 Record more policies on targets when created 2014-03-06 11:05:47 -05:00
CMP0028 RunCMake: Remove unneeded files. 2015-02-09 19:15:04 +01:00
CMP0037 Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
CMP0038 Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
CMP0039 RunCMake: Remove unneeded files. 2015-02-09 19:15:04 +01:00
CMP0040 Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
CMP0041 Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
CMP0042 Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
CMP0043 Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
CMP0045 Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
CMP0046 Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
CMP0049 Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
CMP0050 Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
CMP0051 Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
CMP0053 tests: test CMP0053 in WARN mode when watching variables 2014-05-22 11:13:29 -04:00
CMP0054 cmIfCommand: Issue CMP0054 warning with appropriate context. (#15802) 2015-10-20 23:40:12 +02:00
CMP0055 RunCMake: Remove unneeded files. 2015-02-09 19:15:04 +01:00
CMP0057 if: Implement new IN_LIST operator 2015-04-30 10:21:19 -04:00
CMP0059 CMP0059: Fix typo in policy description 2016-05-02 15:16:07 -04:00
CMP0060 Link libraries by full path even in implicit directories 2015-04-09 11:29:18 -04:00
CMP0064 CMP0064: Fix recorded version of introduction to be 3.4 2015-09-21 10:05:02 -04:00
CMP0065 Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
COMPILE_LANGUAGE-genex Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
CPack Merge topic 'cpack-rpm-adding-dist-to-release-tag' 2016-05-16 10:11:35 -04:00
CPackConfig Introduction of CPACK_VERBATIM_VARIABLES variable 2015-09-20 23:39:03 +02:00
CPackInstallProperties Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
CPackSymlinks Tests: Add case for CPack source package with symlinks 2015-04-06 08:52:54 -04:00
CTest Tests: Fix RunCMake.CTest to tolerate source line number changes 2015-04-27 10:46:52 -04:00
CTestCommandLine CTest: Fix regression in handling of a RUN_SERIAL test that fails 2015-10-23 09:55:25 -04:00
CTestTimeoutAfterMatch CTest: Optionally use a secondary test timeout after matching output 2016-03-22 11:17:55 -04:00
CheckModules RunCMake: Expect empty output by default. 2014-12-31 16:34:34 +01:00
ClangTidy Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
CommandLine Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
CommandLineTar Tests: Consolidate, refactor and extend -E tar tests 2015-04-10 08:32:32 -04:00
CompatibleInterface Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
CompileDefinitions CompileDefinitions: Add unit test for setting empty content. 2015-07-22 21:23:49 +02:00
CompileFeatures Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
CompilerChange Drop use of configure_file IMMEDIATE option 2013-11-13 10:12:17 -05:00
CompilerLauncher Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
CompilerNotFound nmake/jom: Only warn about bad VS environment if compiler not found. 2015-08-06 17:32:43 +00:00
Configure cmTarget: Do not mistake a preceding error for a CMP0049 failure 2014-07-30 13:48:33 -04:00
CrosscompilingEmulator Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
DisallowedCommands RunCMake: Remove unneeded files. 2015-02-09 19:15:04 +01:00
ExportWithoutLanguage Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
ExternalData ExternalData: Add option to recursively match under directories 2015-03-27 14:44:54 -04:00
ExternalProject ExternalProject: Added new USES_TERMINAL options 2015-07-06 14:51:40 -04:00
FPHSA RunCMake: Expect empty output by default. 2014-12-31 16:34:34 +01:00
FeatureSummary FeatureSummary: Print each feature info only once 2015-02-23 11:39:17 -05:00
File_Generate Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
FindMatlab Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
FindPkgConfig FindPkgConfig: optionally create imported target for the found libraries 2016-05-14 09:31:41 +02:00
Framework Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
GNUInstallDirs GNUInstallDirs: Add special cases for certain prefixes 2015-06-18 08:59:34 -04:00
GeneratorExpression Genex: Add a SHELL_PATH expression 2015-09-28 10:37:33 -04:00
GeneratorPlatform Tests: Test setting a generator platform in a toolchain file 2014-09-15 10:27:00 -04:00
GeneratorToolset VS14: Add Visual Studio 14 generator (#14982) 2014-06-25 14:51:19 -04:00
IfacePaths Tests: Fix failures when running under the default install prefix (#15566) 2015-05-14 10:07:37 -04:00
IncludeWhatYouUse Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
IncompatibleQt Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
Languages Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
LinkStatic Tests: Move LINK_SEARCH_{START,END}_STATIC case to dedicated test 2015-09-02 16:11:29 -04:00
Make Tests: Add test for TARGET_MESSAGES global property 2015-07-14 13:58:53 -04:00
Ninja Ninja: Add test for `$subdir/all` targets 2016-03-22 08:01:14 -04:00
ObjectLibrary Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
ObsoleteQtMacros Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
PolicyScope cmListFileBacktrace: Refactor storage to provide efficient value semantics 2016-04-18 09:21:19 -04:00
PositionIndependentCode Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
Swift Swift: Remove positive Swift language tests 2015-09-06 18:14:35 +02:00
Syntax cmListFileBacktrace: Refactor storage to provide efficient value semantics 2016-04-18 09:21:19 -04:00
TargetObjects Allow INTERFACE_SOURCES to specify $<TARGET_OBJECTS> (#14970) 2014-07-14 09:09:32 -04:00
TargetPolicies Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
TargetPropertyGeneratorExpressions Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
TargetSources Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
ToolchainFile Diagnose recursive project/enable_language without crashing (#15999) 2016-03-07 13:52:38 -05:00
VS10Project Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
VSSolution VS: Fix default target support for targets nested inside a folder 2016-03-23 13:22:15 -04:00
VisibilityPreset CMP0063: Split unit test by target type. 2015-10-20 23:58:48 +02:00
WriteCompilerDetectionHeader WCDH: Make it possible to generate multiple files. 2014-11-24 21:54:41 +01:00
XcodeProject Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
add_custom_command add_custom_command: Clarify error when TARGET is out of scope (#15681) 2016-01-28 10:33:26 -05:00
add_custom_target Add an option for explicit BYPRODUCTS of custom commands (#14963) 2014-11-14 16:16:00 -05:00
add_dependencies Cygwin: Avoid legacy warnings in RunCMake.* tests 2013-08-13 09:50:15 -04:00
add_subdirectory Tests: Add case for add_subdirectory inside a function 2015-11-23 10:52:51 -05:00
alias_targets Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
build_command CTest: Stop telling 'make' to ignore errors with -i 2015-05-12 09:06:49 -04:00
cmake_minimum_required Drop compatibility with CMake < 2.4 2013-10-23 08:54:31 -04:00
cmake_parse_arguments cmake_parse_arguments: Additional regression tests 2016-02-24 10:07:30 -05:00
configure_file Tests: Add more signature tests to RunCMake.configure_file test 2015-03-09 09:17:59 -04:00
continue continue: Add a new CMake language command for loop continuation (#14013) 2014-12-01 09:50:49 -05:00
ctest_build CTest: Optionally add a ChangeId attribute on XML Site tags 2015-07-17 10:26:28 -04:00
ctest_configure ctest_configure: Add QUIET option 2015-02-23 10:01:59 -05:00
ctest_coverage ctest_coverage: Add QUIET option 2015-02-23 10:02:00 -05:00
ctest_memcheck Tests: Guard more CTest tests against parallel execution. 2015-04-27 22:03:40 +02:00
ctest_start Tests: Move CTestTestConfigFileInBuildDir into RunCMake.ctest_start 2015-07-14 09:32:53 -04:00
ctest_submit Tests: Extend ctest_submit host lookup failure matching (#15958) 2016-02-08 09:25:21 -05:00
ctest_test Tests: Disable parallel test execution while running ctest_test 2015-10-27 21:44:04 +01:00
ctest_upload ctest_upload: Add QUIET option 2015-02-23 10:02:00 -05:00
execute_process execute_process: Improve stdout/stderr merging 2015-05-07 14:40:38 -04:00
export export: Reject custom target exports earlier (#15657) 2015-07-21 14:39:10 -04:00
file file: Sort GLOB results to make it deterministic (#14491) 2016-05-17 10:12:11 -04:00
find_dependency Tests: Check find_dependency empty extra arguments 2014-03-13 15:57:03 -04:00
find_file Drop find_(library|file|path) prefixes from PATH on non-Windows 2016-05-09 12:58:08 -04:00
find_library Drop find_(library|file|path) prefixes from PATH on non-Windows 2016-05-09 12:58:08 -04:00
find_package cmListFileBacktrace: Refactor storage to provide efficient value semantics 2016-04-18 09:21:19 -04:00
find_path Drop find_(library|file|path) prefixes from PATH on non-Windows 2016-05-09 12:58:08 -04:00
find_program Merge topic 'fix-find_program-regression' 2015-11-02 09:09:46 -05:00
get_filename_component get_filename_component: Teach new BASE_DIR parameter. 2015-08-17 23:55:38 -04:00
get_property Add SOURCE_DIR and BINARY_DIR target properties 2015-07-21 14:19:35 -04:00
if cmConditionEvaluator: Fix matching of `CMAKE_MATCH_*` values (#15944) 2016-02-01 10:05:10 -05:00
include Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
include_directories include_directories: Fix regression in BEFORE option (#15693) 2015-08-21 15:21:37 -04:00
include_external_msproject Tests: Add generator platform support 2014-09-10 11:21:50 -04:00
install Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
interface_library Allow add_dependencies() on INTERFACE libraries (#15414) 2015-03-27 16:25:09 -04:00
list list: Add FILTER subcommand (#3986) 2016-02-03 11:13:17 -05:00
message Add -Werror and -Wno-error command-line options 2016-01-12 14:02:51 -05:00
no_install_prefix RunCMake: Remove unneeded files. 2015-02-09 19:15:04 +01:00
project RunCMake: Remove unneeded files. 2015-02-09 19:15:04 +01:00
return return: Add test for returning inside a foreach loop 2014-11-25 14:37:39 -05:00
set Merge branch 'parent-scope-tests' into variable-pull-failure 2014-10-24 13:34:30 -04:00
set_property Tests: Cover set_property for buildsystem directory properties 2015-09-11 19:13:25 +02:00
string string: add APPEND subcommand 2015-07-07 09:23:21 -04:00
target_compile_features Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
target_link_libraries Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
try_compile Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
try_run Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
variable_watch Cygwin: Avoid legacy warnings in RunCMake.* tests 2013-08-13 09:50:15 -04:00
while cmListFileBacktrace: Refactor storage to provide efficient value semantics 2016-04-18 09:21:19 -04:00
CMakeLists.txt CustomCommandGenerator: Add support for CROSSCOMPILING_EMULATOR 2016-05-09 08:56:27 -04:00
README.rst Tests: Split RunCTest helper out of RunCMake.CTest(Submit|Memcheck) 2015-02-19 11:26:44 -05:00
RunCMake.cmake Deprecate Visual Studio 7 .NET 2003 generator 2016-04-28 10:09:20 -04:00
RunCTest.cmake Tests: Split RunCTest helper out of RunCMake.CTest(Submit|Memcheck) 2015-02-19 11:26:44 -05:00
pseudo_emulator.c Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
pseudo_emulator_custom_command.c Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00
pseudo_iwyu.c Add options to run include-what-you-use with the compiler 2015-05-19 13:16:29 -04:00
pseudo_tidy.c Revise C++ coding style using clang-format 2016-05-16 16:05:19 -04:00

README.rst

This directory contains tests that run CMake to configure a project
but do not actually build anything.  To add a test:

1. Add a subdirectory named for the test, say ``<Test>/``.

2. In ``./CMakeLists.txt`` call ``add_RunCMake_test`` and pass the
   test directory name ``<Test>``.

3. Create script ``<Test>/RunCMakeTest.cmake`` in the directory containing::

    include(RunCMake)
    run_cmake(SubTest1)
    ...
    run_cmake(SubTestN)

   where ``SubTest1`` through ``SubTestN`` are sub-test names each
   corresponding to an independent CMake run and project configuration.

   One may also add calls of the form::

    run_cmake_command(SubTestI ${CMAKE_COMMAND} ...)

   to fully customize the test case command-line.

   Alternatively, if the test is to cover running ``ctest -S`` then use::

    include(RunCTest)
    run_ctest(SubTest1)
    ...
    run_ctest(SubTestN)

   and create ``test.cmake.in``, ``CTestConfig.cmake.in``, and
   ``CMakeLists.txt.in`` files to be configured for each case.

4. Create file ``<Test>/CMakeLists.txt`` in the directory containing::

    cmake_minimum_required(...)
    project(${RunCMake_TEST} NONE) # or languages needed
    include(${RunCMake_TEST}.cmake)

   where ``${RunCMake_TEST}`` is literal.  A value for ``RunCMake_TEST``
   will be passed to CMake by the ``run_cmake`` macro when running each
   sub-test.

5. Create a ``<Test>/<SubTest>.cmake`` file for each sub-test named
   above containing the actual test code.  Optionally create files
   containing expected test results:

   ``<SubTest>-result.txt``
    Process result expected if not "0"
   ``<SubTest>-stdout.txt``
    Regex matching expected stdout content
   ``<SubTest>-stderr.txt``
    Regex matching expected stderr content, if not "^$"
   ``<SubTest>-check.cmake``
    Custom result check.

   Note that trailing newlines will be stripped from actual and expected
   test output before matching against the stdout and stderr expressions.
   The code in ``<SubTest>-check.cmake`` may use variables

   ``RunCMake_TEST_SOURCE_DIR``
    Top of test source tree
   ``RunCMake_TEST_BINARY_DIR``
    Top of test binary tree

   and an failure must store a message in ``RunCMake_TEST_FAILED``.