When testing under Bullseye coverage, some tests get lines on stderr of
the form:
... Bullseye Testing Technology ...
Remove such lines from output before matching because they are not
representative of the actual test output.
On some OS X machines some tests get lines on stderr of the form:
... attempting to exclude an item from Time Machine by path ...
produced by the system. Remove such lines from output before matching
because they are not representative of the actual test output.
Some tests may need to read "cmake --build" output passed through from
native build tools and do not know if it will be on stdout or stderr.
Optionally use the same variable for the execute_process output so
that it merges them and we can always match using expected stdout.
When RunCMake tests run on Solaris, the output may contain unexpected
lines of the form "Error kstat returned...". These lines are printed by
SystemInformationImplementation::RunProcess when called from
SystemInformationImplementation::ParseValueFromKStat (see issue #12066).
Until someone investigates why kstat returns values outside the range it
documents, simply remove such lines from the actual output before
matching it against the expected output.
When RunCMake tests run under Xcode Guard Malloc, Guard Malloc may add
lines of the form "<tool>(<pid>) malloc:..." to the output. Remove such
lines from the actual output before matching it against the expected
output.
When RunCMake tests run under dynamic analysis, Bullseye may add lines
of the form "BullseyeCoverage..." to the output. Remove such lines from the
actual output before matching it against the expected output.
Expect tests to specify stderr content if it is present.
Fix the CMP0019 test, which has only been testing the WARN status
until now. Specify in the CommandLine and FPHSA tests that content
is at least one character.
Set policies in the Language and CheckModules tests, which have empty
test output, modulo unrelated policies on some platforms.
Define the 'cmake -A' option to set CMAKE_GENERATOR_PLATFORM
without having to spell out the whole variable name. We choose
the name '-A' for "platform" because '-P' is already taken, and
in the common use case the "platform" is actually an architecture
(e.g. x64).
Teach the RunCMake test infrastructure to use -A to pass the generator
platform. Extend the RunCMake.GeneratorPlatform test with a case to
verify that the -A option cannot be repeated.
We cannot use -DRunCMake_TEST=${RunCMake_TEST_FILE} because test cases
use project(${RunCMake_TEST}) and the project name cannot be a path to a
file. The parent and grandparent commits removed the only uses of the
option. Drop it now.
This will allow decoupling the name of the test from the name and
location of the source file under test, which means one source
file can be used for multiple tests.
Rename the PARENT_SCOPE test in RunCMake.set to not use a keyword
of the if() command as a file name. As the filename is now used
with an if condition, this causes a conflict.
On OS X we pass -DCMAKE_POLICY_DEFAULT_CMP0025=NEW to all tests.
Once tests start using cmake_minimum_required(VERSION 3.0) then
CMake will warn that CMAKE_POLICY_DEFAULT_CMP0025 is unused.
These warnings are not part of the expected test output and can
cause such tests to fail. Pass --no-warn-unused-cli to each test
cmake invocation to tell it not to produce these warnings.
Add a 'run_cmake_command' function that can be used by tests to run a
given command-line and check the results rather than always running a
CMake configuration process. This can be used in the future to test
'cmake -E' for example.
1763c31 Set policy CMP0025 to NEW while building CMake itself
aa53ee5 Add policy CMP0025 for Apple Clang compiler id compatibility
ab65862 Clang: Add separate "AppleClang" compiler id
CMake is aware of the policy's NEW behavior and the AppleClang compiler
id. Set the policy to NEW explicitly to avoid the warning and get the
NEW behavior.
Also teach the RunCMake test infrastructure to build tests with
-DCMAKE_POLICY_DEFAULT_CMP0025=NEW to avoid the policy warning
in test output that must match specific regular expressions.
When RunCMake tests run during dynamic analysis, valgrind may add lines
of the form "==[0-9]+==..." to the output. Remove such lines from the
actual output before matching it against the expected output.
Reject the option by default. It will be implemented on a per-generator
basis. Pass the setting into try_compile project generation. Add cache
entry CMAKE_GENERATOR_TOOLSET and associated variable documentation to
hold the value persistently.
Add a RunCMake.GeneratorToolset test to cover basic "-T" option cases.
Verify that CMAKE_GENERATOR_TOOLSET is empty without -T, that -T is
rejected when the generator doesn't support it, and that two -T options
are always rejected.
Look for a <SubTest>-check.cmake script and load it to check side
effects of the sub test. Provide it with the test source and build tree
paths in variables. Check for a failure message in a result variable.
The CMakeOnly directory added by commit 9a20abf0 (Add infrastructure for
CMake-only tests, 2012-01-11) was sufficient only for tests that always
run CMake to successfully configure a project. Later commit eeaaffcb
(find_package: Test error and warning messages in failure cases,
2012-02-28) added a sample test that covers failure cases.
Generalize the above to create new "RunCMake" test infrastructure that
can run CMake multiple times for a single project with different
variations and check for expected result/stdout/stderr. Allow for both
successful and failing CMake project configuration cases. This will be
useful to test error messages and failure behavior.
Brad King
Stephen Kelly