CTest: Add CAPTURE_CMAKE_ERROR val to `ctest_*` commands

If a `ctest_*` command has CAPTURE_CMAKE_ERROR then any errors generated
by cmake during that command will cause the value to be assigned `-1`.
This will prevent a `ctest -S` script from returning non-zero unless the
script explicitly calls `message(FATAL_ERROR)`.

Help/command/ctest_build.rst

@@ -13,6 +13,7 @@ Perform the :ref:`CTest Build Step` as a :ref:`Dashboard Client`.
               [NUMBER_ERRORS <num-err-var>]
               [NUMBER_WARNINGS <num-warn-var>]
               [RETURN_VALUE <result-var>]
+              [CAPTURE_CMAKE_ERROR <result-var>]
               )
 
 Build the project and store results in ``Build.xml``
@@ -66,6 +67,10 @@ The options are:
 ``RETURN_VALUE <result-var>``
   Store the return value of the native build tool in the given variable.
 
+``CAPTURE_CMAKE_ERROR <result-var>``
+  Store in the ``<result-var>`` variable -1 if there are any errors running
+  the command and prevent ctest from returning non-zero if an error occurs.
+
 ``QUIET``
   Suppress any CTest-specific non-error output that would have been
   printed to the console otherwise.  The summary of warnings / errors,

Help/command/ctest_configure.rst

@@ -6,7 +6,8 @@ Perform the :ref:`CTest Configure Step` as a :ref:`Dashboard Client`.
 ::
 
   ctest_configure([BUILD <build-dir>] [SOURCE <source-dir>] [APPEND]
-                  [OPTIONS <options>] [RETURN_VALUE <result-var>] [QUIET])
+                  [OPTIONS <options>] [RETURN_VALUE <result-var>] [QUIET]
+                  [CAPTURE_CMAKE_ERROR <result-var>])
 
 Configure the project build tree and record results in ``Configure.xml``
 for submission with the :command:`ctest_submit` command.
@@ -33,6 +34,10 @@ The options are:
   Store in the ``<result-var>`` variable the return value of the native
   configuration tool.
 
+``CAPTURE_CMAKE_ERROR <result-var>``
+  Store in the ``<result-var>`` variable -1 if there are any errors running
+  the command and prevent ctest from returning non-zero if an error occurs.
+
 ``QUIET``
   Suppress any CTest-specific non-error messages that would have
   otherwise been printed to the console.  Output from the underlying

Help/command/ctest_coverage.rst

@@ -8,6 +8,7 @@ Perform the :ref:`CTest Coverage Step` as a :ref:`Dashboard Client`.
   ctest_coverage([BUILD <build-dir>] [APPEND]
                  [LABELS <label>...]
                  [RETURN_VALUE <result-var>]
+                 [CAPTURE_CMAKE_ERROR <result-var]
                  [QUIET]
                  )
 
@@ -33,6 +34,10 @@ The options are:
   Store in the ``<result-var>`` variable ``0`` if coverage tools
   ran without error and non-zero otherwise.
 
+``CAPTURE_CMAKE_ERROR <result-var>``
+  Store in the ``<result-var>`` variable -1 if there are any errors running
+  the command and prevent ctest from returning non-zero if an error occurs.
+
 ``QUIET``
   Suppress any CTest-specific non-error output that would have been
   printed to the console otherwise.  The summary indicating how many

Help/command/ctest_test.rst

@@ -18,6 +18,7 @@ Perform the :ref:`CTest Test Step` as a :ref:`Dashboard Client`.
              [SCHEDULE_RANDOM <ON|OFF>]
              [STOP_TIME <time-of-day>]
              [RETURN_VALUE <result-var>]
+             [CAPTURE_CMAKE_ERROR <result-var>]
              [QUIET]
              )
 
@@ -80,6 +81,10 @@ The options are:
   Store in the ``<result-var>`` variable ``0`` if all tests passed.
   Store non-zero if anything went wrong.
 
+``CAPTURE_CMAKE_ERROR <result-var>``
+  Store in the ``<result-var>`` variable -1 if there are any errors running
+  the command and prevent ctest from returning non-zero if an error occurs.
+
 ``QUIET``
   Suppress any CTest-specific non-error messages that would have otherwise
   been printed to the console.  Output from the underlying test command is not

Help/command/ctest_upload.rst

@@ -5,7 +5,7 @@ Upload files to a dashboard server as a :ref:`Dashboard Client`.
 
 ::
 
-  ctest_upload(FILES <file>... [QUIET])
+  ctest_upload(FILES <file>... [QUIET] [CAPTURE_CMAKE_ERROR <result-var>])
 
 The options are:
 
@@ -16,3 +16,7 @@ The options are:
 ``QUIET``
   Suppress any CTest-specific non-error output that would have been
   printed to the console otherwise.
+
+``CAPTURE_CMAKE_ERROR <result-var>``
+  Store in the ``<result-var>`` variable -1 if there are any errors running
+  the command and prevent ctest from returning non-zero if an error occurs.

Source/CTest/cmCTestHandlerCommand.cxx

@@ -31,6 +31,7 @@ cmCTestHandlerCommand::cmCTestHandlerCommand()
     this->Arguments.push_back(CM_NULLPTR);
   }
   this->Arguments[ct_RETURN_VALUE] = "RETURN_VALUE";
+  this->Arguments[ct_CAPTURE_CMAKE_ERROR] = "CAPTURE_CMAKE_ERROR";
   this->Arguments[ct_SOURCE] = "SOURCE";
   this->Arguments[ct_BUILD] = "BUILD";
   this->Arguments[ct_SUBMIT_INDEX] = "SUBMIT_INDEX";
@@ -39,15 +40,71 @@ cmCTestHandlerCommand::cmCTestHandlerCommand()
   this->Quiet = false;
 }
 
+namespace {
+// class to save and restore the error state for ctest_* commands
+// if a ctest_* command has a CAPTURE_CMAKE_ERROR then put the error
+// state into there and restore the system wide error to what
+// it was before the command ran
+class SaveRestoreErrorState
+{
+public:
+  SaveRestoreErrorState()
+  {
+    this->InitialErrorState = cmSystemTools::GetErrorOccuredFlag();
+    cmSystemTools::ResetErrorOccuredFlag(); // rest the error state
+    this->CaptureCMakeErrorValue = false;
+  }
+  // if the function has a CAPTURE_CMAKE_ERROR then we should restore
+  // the error state to what it was before the function was run
+  // if not then let the error state be what it is
+  void CaptureCMakeError() { this->CaptureCMakeErrorValue = true; }
+  ~SaveRestoreErrorState()
+  {
+    // if we are not saving the return value then make sure
+    // if it was in error it goes back to being in error
+    // otherwise leave it be what it is
+    if (!this->CaptureCMakeErrorValue) {
+      if (this->InitialErrorState) {
+        cmSystemTools::SetErrorOccured();
+      }
+      return;
+    }
+    // if we have saved the error in a return variable
+    // then put things back exactly like they were
+    bool currentState = cmSystemTools::GetErrorOccuredFlag();
+    // if the state changed during this command we need
+    // to handle it, if not then nothing needs to be done
+    if (currentState != this->InitialErrorState) {
+      // restore the initial error state
+      if (this->InitialErrorState) {
+        cmSystemTools::SetErrorOccured();
+      } else {
+        cmSystemTools::ResetErrorOccuredFlag();
+      }
+    }
+  }
+
+private:
+  bool InitialErrorState;
+  bool CaptureCMakeErrorValue;
+};
+}
+
 bool cmCTestHandlerCommand::InitialPass(std::vector<std::string> const& args,
                                         cmExecutionStatus& /*unused*/)
 {
+  // save error state and restore it if needed
+  SaveRestoreErrorState errorState;
   // Allocate space for argument values.
   this->Values.clear();
   this->Values.resize(this->Last, CM_NULLPTR);
 
   // Process input arguments.
   this->ArgumentDoing = ArgumentDoingNone;
+  // look at all arguments and do not short circuit on the first
+  // bad one so that CAPTURE_CMAKE_ERROR can override setting the
+  // global error state
+  bool foundBadArgument = false;
   for (unsigned int i = 0; i < args.size(); ++i) {
     // Check this argument.
     if (!this->CheckArgumentKeyword(args[i]) &&
@@ -55,14 +112,36 @@ bool cmCTestHandlerCommand::InitialPass(std::vector<std::string> const& args,
       std::ostringstream e;
       e << "called with unknown argument \"" << args[i] << "\".";
       this->SetError(e.str());
-      return false;
+      foundBadArgument = true;
     }
-
-    // Quit if an argument is invalid.
+    // note bad argument
     if (this->ArgumentDoing == ArgumentDoingError) {
-      return false;
+      foundBadArgument = true;
     }
   }
+  bool capureCMakeError = (this->Values[ct_CAPTURE_CMAKE_ERROR] &&
+                           *this->Values[ct_CAPTURE_CMAKE_ERROR]);
+  // now that arguments are parsed check to see if there is a
+  // CAPTURE_CMAKE_ERROR specified let the errorState object know.
+  if (capureCMakeError) {
+    errorState.CaptureCMakeError();
+  }
+  // if we found a bad argument then exit before running command
+  if (foundBadArgument) {
+    // store the cmake error
+    if (capureCMakeError) {
+      this->Makefile->AddDefinition(this->Values[ct_CAPTURE_CMAKE_ERROR],
+                                    "-1");
+      const char* err = this->GetError();
+      if (err && !cmSystemTools::FindLastString(err, "unknown error.")) {
+        cmCTestLog(this->CTest, ERROR_MESSAGE, err << " error from command\n");
+      }
+      // return success because failure is recorded in CAPTURE_CMAKE_ERROR
+      return true;
+    }
+    // return failure because of bad argument
+    return false;
+  }
 
   // Set the config type of this ctest to the current value of the
   // CTEST_CONFIGURATION_TYPE script variable if it is defined.
@@ -117,6 +196,15 @@ bool cmCTestHandlerCommand::InitialPass(std::vector<std::string> const& args,
   if (!handler) {
     cmCTestLog(this->CTest, ERROR_MESSAGE, "Cannot instantiate test handler "
                  << this->GetName() << std::endl);
+    if (capureCMakeError) {
+      this->Makefile->AddDefinition(this->Values[ct_CAPTURE_CMAKE_ERROR],
+                                    "-1");
+      const char* err = this->GetError();
+      if (err && !cmSystemTools::FindLastString(err, "unknown error.")) {
+        cmCTestLog(this->CTest, ERROR_MESSAGE, err << " error from command\n");
+      }
+      return true;
+    }
     return false;
   }
 
@@ -147,6 +235,22 @@ bool cmCTestHandlerCommand::InitialPass(std::vector<std::string> const& args,
     this->Makefile->AddDefinition(this->Values[ct_RETURN_VALUE],
                                   str.str().c_str());
   }
+  // log the error message if there was an error
+  if (capureCMakeError) {
+    const char* returnString = "0";
+    if (cmSystemTools::GetErrorOccuredFlag()) {
+      returnString = "-1";
+      const char* err = this->GetError();
+      // print out the error if it is not "unknown error" which means
+      // there was no message
+      if (err && !cmSystemTools::FindLastString(err, "unknown error.")) {
+        cmCTestLog(this->CTest, ERROR_MESSAGE, err);
+      }
+    }
+    // store the captured cmake error state 0 or -1
+    this->Makefile->AddDefinition(this->Values[ct_CAPTURE_CMAKE_ERROR],
+                                  returnString);
+  }
   cmSystemTools::ChangeDirectory(current_dir);
   return true;
 }

Source/CTest/cmCTestHandlerCommand.h

@@ -47,6 +47,7 @@ public:
   {
     ct_NONE,
     ct_RETURN_VALUE,
+    ct_CAPTURE_CMAKE_ERROR,
     ct_BUILD,
     ct_SOURCE,
     ct_SUBMIT_INDEX,

Source/CTest/cmCTestUploadCommand.cxx

@@ -45,11 +45,19 @@ bool cmCTestUploadCommand::CheckArgumentKeyword(std::string const& arg)
     this->Quiet = true;
     return true;
   }
+  if (arg == "CAPTURE_CMAKE_ERROR") {
+    this->ArgumentDoing = ArgumentDoingCaptureCMakeError;
+    return true;
+  }
   return false;
 }
 
 bool cmCTestUploadCommand::CheckArgumentValue(std::string const& arg)
 {
+  if (this->ArgumentDoing == ArgumentDoingCaptureCMakeError) {
+    this->Values[ct_CAPTURE_CMAKE_ERROR] = arg.c_str();
+    return true;
+  }
   if (this->ArgumentDoing == ArgumentDoingFiles) {
     if (cmSystemTools::FileExists(arg.c_str())) {
       this->Files.insert(arg);

Source/CTest/cmCTestUploadCommand.h

@@ -61,6 +61,7 @@ protected:
   enum
   {
     ArgumentDoingFiles = Superclass::ArgumentDoingLast1,
+    ArgumentDoingCaptureCMakeError,
     ArgumentDoingLast2
   };
 

Tests/RunCMake/CMakeLists.txt

@@ -184,6 +184,7 @@ add_RunCMake_test(cmake_minimum_required)
 add_RunCMake_test(cmake_parse_arguments)
 add_RunCMake_test(continue)
 add_RunCMake_test(ctest_build)
+add_RunCMake_test(ctest_cmake_error)
 add_RunCMake_test(ctest_configure)
 if(COVERAGE_COMMAND)
   add_RunCMake_test(ctest_coverage -DCOVERAGE_COMMAND=${COVERAGE_COMMAND})

Tests/RunCMake/ctest_cmake_error/CMakeLists.txt.in

@@ -0,0 +1,4 @@
+cmake_minimum_required(VERSION 3.1)
+project(CTestCoverage@CASE_NAME@ NONE)
+include(CTest)
+add_test(NAME RunCMakeVersion COMMAND "${CMAKE_COMMAND}" --version)

Tests/RunCMake/ctest_cmake_error/CTestCaptureErrorNonZero-stderr.txt

@@ -0,0 +1,4 @@
+.*ctest_configure called with unknown argument "junk".*
+.*ctest_build called with unknown argument "junk".*
+.*ctest_test called with unknown argument "junk".*
+.*ctest_coverage called with unknown argument "junk".*

Tests/RunCMake/ctest_cmake_error/CTestCaptureErrorZero-stderr.txt

@@ -0,0 +1,4 @@
+.*ctest_configure called with unknown argument "junk".*
+.*ctest_build called with unknown argument "junk".*
+.*ctest_test called with unknown argument "junk".*
+.*ctest_coverage called with unknown argument "junk".*

Tests/RunCMake/ctest_cmake_error/CTestCaptureErrorZero-stdout.txt

@@ -0,0 +1 @@
+.*Run dashboard with model Experimental.*

Tests/RunCMake/ctest_cmake_error/CTestConfig.cmake.in

@@ -0,0 +1 @@
+set(CTEST_PROJECT_NAME "CTestCoverage@CASE_NAME@")

Tests/RunCMake/ctest_cmake_error/CoverageQuiet-stdout.txt

@@ -0,0 +1 @@
+sec$

Tests/RunCMake/ctest_cmake_error/RunCMakeTest.cmake

@@ -0,0 +1,10 @@
+include(RunCTest)
+
+set(CASE_CTEST_COVERAGE_ARGS "")
+
+function(run_ctest_coverage CASE_NAME)
+  set(CASE_CTEST_COVERAGE_ARGS "${ARGN}")
+  run_ctest(${CASE_NAME})
+endfunction()
+
+run_ctest_coverage(CTestCaptureErrorNonZero junk CAPTURE_CMAKE_ERROR val)

Tests/RunCMake/ctest_cmake_error/test.cmake.in

@@ -0,0 +1,22 @@
+cmake_minimum_required(VERSION 3.1)
+
+set(CTEST_SITE                          "test-site")
+set(CTEST_BUILD_NAME                    "test-build-name")
+set(CTEST_SOURCE_DIRECTORY              "@RunCMake_BINARY_DIR@/@CASE_NAME@")
+set(CTEST_BINARY_DIRECTORY              "@RunCMake_BINARY_DIR@/@CASE_NAME@-build")
+set(CTEST_CMAKE_GENERATOR               "@RunCMake_GENERATOR@")
+set(CTEST_CMAKE_GENERATOR_PLATFORM      "@RunCMake_GENERATOR_PLATFORM@")
+set(CTEST_CMAKE_GENERATOR_TOOLSET       "@RunCMake_GENERATOR_TOOLSET@")
+set(CTEST_BUILD_CONFIGURATION           "$ENV{CMAKE_CONFIG_TYPE}")
+set(CTEST_COVERAGE_COMMAND              "@COVERAGE_COMMAND@")
+
+set(ctest_coverage_args "@CASE_CTEST_COVERAGE_ARGS@")
+ctest_start(Experimental)
+ctest_configure( ${ctest_coverage_args} )
+ctest_build(  ${ctest_coverage_args} )
+ctest_test( ${ctest_coverage_args} )
+ctest_coverage( ${ctest_coverage_args} )
+ctest_upload(junk CAPTURE_CMAKE_ERROR val)
+if(NOT val EQUAL -1)
+  message(FATAL_ERROR "CAPTURE_CMAKE_ERROR should be -1 is [${val}]")
+endif()