CTest: Add CAPTURE_CMAKE_ERROR val to `ctest_*` commands

If a `ctest_*` command has CAPTURE_CMAKE_ERROR then any errors generated
by cmake during that command will cause the value to be assigned `-1`.
This will prevent a `ctest -S` script from returning non-zero unless the
script explicitly calls `message(FATAL_ERROR)`.
This commit is contained in:
Bill Hoffman 2016-07-28 15:12:43 -04:00 committed by Brad King
parent 9ac2e18960
commit d328dc6853
18 changed files with 192 additions and 6 deletions

View File

@ -13,6 +13,7 @@ Perform the :ref:`CTest Build Step` as a :ref:`Dashboard Client`.
[NUMBER_ERRORS <num-err-var>]
[NUMBER_WARNINGS <num-warn-var>]
[RETURN_VALUE <result-var>]
[CAPTURE_CMAKE_ERROR <result-var>]
)
Build the project and store results in ``Build.xml``
@ -66,6 +67,10 @@ The options are:
``RETURN_VALUE <result-var>``
Store the return value of the native build tool in the given variable.
``CAPTURE_CMAKE_ERROR <result-var>``
Store in the ``<result-var>`` variable -1 if there are any errors running
the command and prevent ctest from returning non-zero if an error occurs.
``QUIET``
Suppress any CTest-specific non-error output that would have been
printed to the console otherwise. The summary of warnings / errors,

View File

@ -6,7 +6,8 @@ Perform the :ref:`CTest Configure Step` as a :ref:`Dashboard Client`.
::
ctest_configure([BUILD <build-dir>] [SOURCE <source-dir>] [APPEND]
[OPTIONS <options>] [RETURN_VALUE <result-var>] [QUIET])
[OPTIONS <options>] [RETURN_VALUE <result-var>] [QUIET]
[CAPTURE_CMAKE_ERROR <result-var>])
Configure the project build tree and record results in ``Configure.xml``
for submission with the :command:`ctest_submit` command.
@ -33,6 +34,10 @@ The options are:
Store in the ``<result-var>`` variable the return value of the native
configuration tool.
``CAPTURE_CMAKE_ERROR <result-var>``
Store in the ``<result-var>`` variable -1 if there are any errors running
the command and prevent ctest from returning non-zero if an error occurs.
``QUIET``
Suppress any CTest-specific non-error messages that would have
otherwise been printed to the console. Output from the underlying

View File

@ -8,6 +8,7 @@ Perform the :ref:`CTest Coverage Step` as a :ref:`Dashboard Client`.
ctest_coverage([BUILD <build-dir>] [APPEND]
[LABELS <label>...]
[RETURN_VALUE <result-var>]
[CAPTURE_CMAKE_ERROR <result-var]
[QUIET]
)
@ -33,6 +34,10 @@ The options are:
Store in the ``<result-var>`` variable ``0`` if coverage tools
ran without error and non-zero otherwise.
``CAPTURE_CMAKE_ERROR <result-var>``
Store in the ``<result-var>`` variable -1 if there are any errors running
the command and prevent ctest from returning non-zero if an error occurs.
``QUIET``
Suppress any CTest-specific non-error output that would have been
printed to the console otherwise. The summary indicating how many

View File

@ -18,6 +18,7 @@ Perform the :ref:`CTest Test Step` as a :ref:`Dashboard Client`.
[SCHEDULE_RANDOM <ON|OFF>]
[STOP_TIME <time-of-day>]
[RETURN_VALUE <result-var>]
[CAPTURE_CMAKE_ERROR <result-var>]
[QUIET]
)
@ -80,6 +81,10 @@ The options are:
Store in the ``<result-var>`` variable ``0`` if all tests passed.
Store non-zero if anything went wrong.
``CAPTURE_CMAKE_ERROR <result-var>``
Store in the ``<result-var>`` variable -1 if there are any errors running
the command and prevent ctest from returning non-zero if an error occurs.
``QUIET``
Suppress any CTest-specific non-error messages that would have otherwise
been printed to the console. Output from the underlying test command is not

View File

@ -5,7 +5,7 @@ Upload files to a dashboard server as a :ref:`Dashboard Client`.
::
ctest_upload(FILES <file>... [QUIET])
ctest_upload(FILES <file>... [QUIET] [CAPTURE_CMAKE_ERROR <result-var>])
The options are:
@ -16,3 +16,7 @@ The options are:
``QUIET``
Suppress any CTest-specific non-error output that would have been
printed to the console otherwise.
``CAPTURE_CMAKE_ERROR <result-var>``
Store in the ``<result-var>`` variable -1 if there are any errors running
the command and prevent ctest from returning non-zero if an error occurs.

View File

@ -31,6 +31,7 @@ cmCTestHandlerCommand::cmCTestHandlerCommand()
this->Arguments.push_back(CM_NULLPTR);
}
this->Arguments[ct_RETURN_VALUE] = "RETURN_VALUE";
this->Arguments[ct_CAPTURE_CMAKE_ERROR] = "CAPTURE_CMAKE_ERROR";
this->Arguments[ct_SOURCE] = "SOURCE";
this->Arguments[ct_BUILD] = "BUILD";
this->Arguments[ct_SUBMIT_INDEX] = "SUBMIT_INDEX";
@ -39,15 +40,71 @@ cmCTestHandlerCommand::cmCTestHandlerCommand()
this->Quiet = false;
}
namespace {
// class to save and restore the error state for ctest_* commands
// if a ctest_* command has a CAPTURE_CMAKE_ERROR then put the error
// state into there and restore the system wide error to what
// it was before the command ran
class SaveRestoreErrorState
{
public:
SaveRestoreErrorState()
{
this->InitialErrorState = cmSystemTools::GetErrorOccuredFlag();
cmSystemTools::ResetErrorOccuredFlag(); // rest the error state
this->CaptureCMakeErrorValue = false;
}
// if the function has a CAPTURE_CMAKE_ERROR then we should restore
// the error state to what it was before the function was run
// if not then let the error state be what it is
void CaptureCMakeError() { this->CaptureCMakeErrorValue = true; }
~SaveRestoreErrorState()
{
// if we are not saving the return value then make sure
// if it was in error it goes back to being in error
// otherwise leave it be what it is
if (!this->CaptureCMakeErrorValue) {
if (this->InitialErrorState) {
cmSystemTools::SetErrorOccured();
}
return;
}
// if we have saved the error in a return variable
// then put things back exactly like they were
bool currentState = cmSystemTools::GetErrorOccuredFlag();
// if the state changed during this command we need
// to handle it, if not then nothing needs to be done
if (currentState != this->InitialErrorState) {
// restore the initial error state
if (this->InitialErrorState) {
cmSystemTools::SetErrorOccured();
} else {
cmSystemTools::ResetErrorOccuredFlag();
}
}
}
private:
bool InitialErrorState;
bool CaptureCMakeErrorValue;
};
}
bool cmCTestHandlerCommand::InitialPass(std::vector<std::string> const& args,
cmExecutionStatus& /*unused*/)
{
// save error state and restore it if needed
SaveRestoreErrorState errorState;
// Allocate space for argument values.
this->Values.clear();
this->Values.resize(this->Last, CM_NULLPTR);
// Process input arguments.
this->ArgumentDoing = ArgumentDoingNone;
// look at all arguments and do not short circuit on the first
// bad one so that CAPTURE_CMAKE_ERROR can override setting the
// global error state
bool foundBadArgument = false;
for (unsigned int i = 0; i < args.size(); ++i) {
// Check this argument.
if (!this->CheckArgumentKeyword(args[i]) &&
@ -55,14 +112,36 @@ bool cmCTestHandlerCommand::InitialPass(std::vector<std::string> const& args,
std::ostringstream e;
e << "called with unknown argument \"" << args[i] << "\".";
this->SetError(e.str());
return false;
foundBadArgument = true;
}
// Quit if an argument is invalid.
// note bad argument
if (this->ArgumentDoing == ArgumentDoingError) {
return false;
foundBadArgument = true;
}
}
bool capureCMakeError = (this->Values[ct_CAPTURE_CMAKE_ERROR] &&
*this->Values[ct_CAPTURE_CMAKE_ERROR]);
// now that arguments are parsed check to see if there is a
// CAPTURE_CMAKE_ERROR specified let the errorState object know.
if (capureCMakeError) {
errorState.CaptureCMakeError();
}
// if we found a bad argument then exit before running command
if (foundBadArgument) {
// store the cmake error
if (capureCMakeError) {
this->Makefile->AddDefinition(this->Values[ct_CAPTURE_CMAKE_ERROR],
"-1");
const char* err = this->GetError();
if (err && !cmSystemTools::FindLastString(err, "unknown error.")) {
cmCTestLog(this->CTest, ERROR_MESSAGE, err << " error from command\n");
}
// return success because failure is recorded in CAPTURE_CMAKE_ERROR
return true;
}
// return failure because of bad argument
return false;
}
// Set the config type of this ctest to the current value of the
// CTEST_CONFIGURATION_TYPE script variable if it is defined.
@ -117,6 +196,15 @@ bool cmCTestHandlerCommand::InitialPass(std::vector<std::string> const& args,
if (!handler) {
cmCTestLog(this->CTest, ERROR_MESSAGE, "Cannot instantiate test handler "
<< this->GetName() << std::endl);
if (capureCMakeError) {
this->Makefile->AddDefinition(this->Values[ct_CAPTURE_CMAKE_ERROR],
"-1");
const char* err = this->GetError();
if (err && !cmSystemTools::FindLastString(err, "unknown error.")) {
cmCTestLog(this->CTest, ERROR_MESSAGE, err << " error from command\n");
}
return true;
}
return false;
}
@ -147,6 +235,22 @@ bool cmCTestHandlerCommand::InitialPass(std::vector<std::string> const& args,
this->Makefile->AddDefinition(this->Values[ct_RETURN_VALUE],
str.str().c_str());
}
// log the error message if there was an error
if (capureCMakeError) {
const char* returnString = "0";
if (cmSystemTools::GetErrorOccuredFlag()) {
returnString = "-1";
const char* err = this->GetError();
// print out the error if it is not "unknown error" which means
// there was no message
if (err && !cmSystemTools::FindLastString(err, "unknown error.")) {
cmCTestLog(this->CTest, ERROR_MESSAGE, err);
}
}
// store the captured cmake error state 0 or -1
this->Makefile->AddDefinition(this->Values[ct_CAPTURE_CMAKE_ERROR],
returnString);
}
cmSystemTools::ChangeDirectory(current_dir);
return true;
}

View File

@ -47,6 +47,7 @@ public:
{
ct_NONE,
ct_RETURN_VALUE,
ct_CAPTURE_CMAKE_ERROR,
ct_BUILD,
ct_SOURCE,
ct_SUBMIT_INDEX,

View File

@ -45,11 +45,19 @@ bool cmCTestUploadCommand::CheckArgumentKeyword(std::string const& arg)
this->Quiet = true;
return true;
}
if (arg == "CAPTURE_CMAKE_ERROR") {
this->ArgumentDoing = ArgumentDoingCaptureCMakeError;
return true;
}
return false;
}
bool cmCTestUploadCommand::CheckArgumentValue(std::string const& arg)
{
if (this->ArgumentDoing == ArgumentDoingCaptureCMakeError) {
this->Values[ct_CAPTURE_CMAKE_ERROR] = arg.c_str();
return true;
}
if (this->ArgumentDoing == ArgumentDoingFiles) {
if (cmSystemTools::FileExists(arg.c_str())) {
this->Files.insert(arg);

View File

@ -61,6 +61,7 @@ protected:
enum
{
ArgumentDoingFiles = Superclass::ArgumentDoingLast1,
ArgumentDoingCaptureCMakeError,
ArgumentDoingLast2
};

View File

@ -184,6 +184,7 @@ add_RunCMake_test(cmake_minimum_required)
add_RunCMake_test(cmake_parse_arguments)
add_RunCMake_test(continue)
add_RunCMake_test(ctest_build)
add_RunCMake_test(ctest_cmake_error)
add_RunCMake_test(ctest_configure)
if(COVERAGE_COMMAND)
add_RunCMake_test(ctest_coverage -DCOVERAGE_COMMAND=${COVERAGE_COMMAND})

View File

@ -0,0 +1,4 @@
cmake_minimum_required(VERSION 3.1)
project(CTestCoverage@CASE_NAME@ NONE)
include(CTest)
add_test(NAME RunCMakeVersion COMMAND "${CMAKE_COMMAND}" --version)

View File

@ -0,0 +1,4 @@
.*ctest_configure called with unknown argument "junk".*
.*ctest_build called with unknown argument "junk".*
.*ctest_test called with unknown argument "junk".*
.*ctest_coverage called with unknown argument "junk".*

View File

@ -0,0 +1,4 @@
.*ctest_configure called with unknown argument "junk".*
.*ctest_build called with unknown argument "junk".*
.*ctest_test called with unknown argument "junk".*
.*ctest_coverage called with unknown argument "junk".*

View File

@ -0,0 +1 @@
.*Run dashboard with model Experimental.*

View File

@ -0,0 +1 @@
set(CTEST_PROJECT_NAME "CTestCoverage@CASE_NAME@")

View File

@ -0,0 +1 @@
sec$

View File

@ -0,0 +1,10 @@
include(RunCTest)
set(CASE_CTEST_COVERAGE_ARGS "")
function(run_ctest_coverage CASE_NAME)
set(CASE_CTEST_COVERAGE_ARGS "${ARGN}")
run_ctest(${CASE_NAME})
endfunction()
run_ctest_coverage(CTestCaptureErrorNonZero junk CAPTURE_CMAKE_ERROR val)

View File

@ -0,0 +1,22 @@
cmake_minimum_required(VERSION 3.1)
set(CTEST_SITE "test-site")
set(CTEST_BUILD_NAME "test-build-name")
set(CTEST_SOURCE_DIRECTORY "@RunCMake_BINARY_DIR@/@CASE_NAME@")
set(CTEST_BINARY_DIRECTORY "@RunCMake_BINARY_DIR@/@CASE_NAME@-build")
set(CTEST_CMAKE_GENERATOR "@RunCMake_GENERATOR@")
set(CTEST_CMAKE_GENERATOR_PLATFORM "@RunCMake_GENERATOR_PLATFORM@")
set(CTEST_CMAKE_GENERATOR_TOOLSET "@RunCMake_GENERATOR_TOOLSET@")
set(CTEST_BUILD_CONFIGURATION "$ENV{CMAKE_CONFIG_TYPE}")
set(CTEST_COVERAGE_COMMAND "@COVERAGE_COMMAND@")
set(ctest_coverage_args "@CASE_CTEST_COVERAGE_ARGS@")
ctest_start(Experimental)
ctest_configure( ${ctest_coverage_args} )
ctest_build( ${ctest_coverage_args} )
ctest_test( ${ctest_coverage_args} )
ctest_coverage( ${ctest_coverage_args} )
ctest_upload(junk CAPTURE_CMAKE_ERROR val)
if(NOT val EQUAL -1)
message(FATAL_ERROR "CAPTURE_CMAKE_ERROR should be -1 is [${val}]")
endif()