e0e75a72 Help: Add notes for topic 'ctest-memcheck-sanitizers'
7345a1f7 tests: Add a test for ctest_memcheck MemorySanitizer
0c6330da ctest_memcheck: Add support for MemorySanitizer msan
9ba8bf12 tests: add a test for ctest_memcheck UndefinedBehaviorSanitizer
816c100a ctest_memcheck: Add support for UndefinedBehaviorSanitizer ubsan
b67ef537 ctest_memcheck: Order sanitizer type code consistently
f48a2968 Tests: Organize CTestTestMemcheck inner test code
f25e431d tests: set sanitizer options properly
f0661bf3 tests: fix copy/paste from tsan -> asan comments
ca9cc25c ctest: add support for additional sanitizer options
0b9ffffc ctest: update documentation for CTEST_MEMORYCHECK_TYPE
When the file is not found in $CMAKE_SOURCE_DIR/<file path>,
the CurFileName pointer remained set to the previous file. This
caused the new file's coverage data to populate the into the
wrong object giving incorrect results and occasionally resulting
in a seg fault.
The ctest_configure command already reads the CTEST_CMAKE_GENERATOR
variable to get the value for the cmake -G option. Read new variable
CTEST_CMAKE_GENERATOR_PLATFORM to pass on as CMAKE_GENERATOR_PLATFORM.
The "ctest --build-and-test" mode already has "--build-generator" to
specify the -G option to CMake. Add a "--build-generator-platform" option
to specify a value to pass on as CMAKE_GENERATOR_PLATFORM.
Prior to this change / was not allowed in the build name. This was tested
with a CDash server and worked. In addition the safe build name was not
used everywhere. This caused mismatched build names to be in the xml
files going to CDash which caused different rows to be created for the
same build.
This allows ctest_update to get the current version without actually
changing the repository. This is useful when using Jenkins or an
external project to update the source to a specific version, but you
still want the current version to show up in CDash.
The current file parser for a MUMPS routine uses a period "." as the
one of the signals that a line of MUMPS code is executable. This is not
a correct assumption. Add the period to the list of characters that CTest
will not consider the start of a line of code.
Update the test routine to have an entry point with code to match the scenario
mentioned above.
This showed up because scan-build noticed outputFull was basically
a constant. Logic should be output all valgrind output. Then output
regular test output until output limit is reached.
No call sites pass NULL to the output argument, so take it by
reference to avoid the if(output) conditions. Propagate the
change through the TryCompile APIs that call it.
Add a new additional entry to the FilePaths array when a
"package" tag has been found. This path should consist of the package
information found appended to the projects source directory.
This change will allow code held in a /src/main/java/* directory off of the
projects source directory to be found, unlike now which assumes a subdirectory
contains the code.
Since the Sanitizers write out one log file per process, a single
test might have more than one log file. This commit allows ctest
to read all of the log files found for a particual test.
Before this commit, you would have to run ctest -S mode to get
MemoryCheckType to work. This is because CMAKE_COMMAND was not set.
The fix is to use cmSystemTools::GetCMakeCommand instead.
This adds support for memory and leak sanitizers. This is built into
clang and gcc 4.8 and new compilers. It is activated with a -f switch
during compile.
This commit adds support for ThreadSanitizer to ctest. ThreadSanitizer
is part of the clang compiler and also gcc 4.8 and later. You have to
compile the code with special flags. Then your code gets the the
ThreadSanitizer ability built into it. To pass options to the
ThreadSanitizer you use an environment variable. This commit teaches
ctest to parse the output from ThreadSanitizer and send it to CDash.
Send status messages to the CTest HANDLER_OUTPUT log since they are part
of the script handler output. This also ensures they appear inline with
other test command handler output.
The ctest launcher code did not respect the number of errors and
warnings limits. Limit the number of launcher report fragments that we
report in the final submission.
For testing purposes CMake creates dummy memory checkers. The dummy checkers
are in the CMake build tree. Before this change when the path contained the
string valgrind, such as CMake-valgrind, all the checkers were thought to
be valgrind, and this caused tests to fail.
Add the ability to parse the XML output of the Jacoco tool.
Jacoco (www.eclemma.org/jacoco) is a Java coverage tool.
Add and integrate a class for the parser and
include a test which utilizes the new parser.
Add support for Cobertura coverage files written by Java.
Add a test which uses the report from a Java run of Cobertura to calculate coverage.
In the documentation of CTEST_COVERAGE_COMMAND, give a sample .sh file to merge
the Cobertura .ser files and generate the XML report from the merged file.
The coverage.py tool writes out an XML that conforms to the Cobertura
Coverage tool standard. Rename the cmParsePythonCoverage files to
instead be cmParseCoberturaCoverage.
If the coverage.py source file is not found in the source directory, the
build directory is first searched before raising an error.
This is necessary because it is a valid workflow to build a Python
package from source, then install this package to a virtualenv that
lives in the build directory. Tests will run against this deployed
package and therefore the covered source files will be found in a
subdirectory of the build directory, and not anywhere in the source
directory.
Fix the MUMPS coverage parser:
* Account for tabs after entry points
* Stop double incrementing lines that have explicit calls to the 0 line
* If a line has been previously marked as non executable, but then
contains a count, increment it an extra one to push it back into
the executable code set.
Add a custom routine and corresponding coverage files in the test case.
This file is smaller and has cmcov/mcov files that have data for only
that routine.
Factor a common callback type out of StdoutCallback. Add an equivalent
StderrCallback. While at it, use "size_t" for the data length instead
of "int".
Teach "ctest --build-and-test" to capture the Stderr callback because
output sent through it is part of the logical CMake process output.
If creation of the global generator fails, return early with an error
message instead of trying to use the generator and crashing.
Add a CTestTestBadGenerator test to cover this case.
Reported-by: Mathieu Malaterre <malat@debian.org>
Bug-Debian: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=747306
By default, Intel compiler coverage tools generate HTML files as
reports, but the option -txtlcov can be given to codecov to output a
coverage file with LCov format.
To use Intel coverage:
* build the project with coverage flags
* run the application
* run profmerge
* run codecov
The output file will be "build_dir/CodeCoverage/SRCFILEDIR.LCOV".
Ask users to compile with -prof-dir${BUILD_DIR} instead of searching
the entire build tree recursively to find coverage files.
Use the clang RemoveCStrCalls tool to automatically migrate the
code. This was only run on linux, so does not have any positive or
negative effect on other platforms.
Casts from std::string -> cmStdString were high on the list of things
taking up time. Avoid such implicit casts across function calls by just
using std::string everywhere.
The comment that the symbol name is too long is no longer relevant since
modern debuggers alias the templates anyways and the size is a
non-issue since the underlying methods are generated since it's
inherited.
My last related commit e5e3f3d4 (CTest: filter /showIncludes output from
ninja compile launcher, 2013-12-01) filtered /showIncludes messages from
the generated xml output but they also need to be filtered in
ScrapeLog(). Otherwise they are being detected as warnings when using
compilers withs english diagnostics.
Mark unknown revisions as such and fail instead of reporting revision 0.
Otherwise CTest reports massive file updates between revisions when the
server timeouts while trying to fetch the current revision number.
The first regression resulted in endless looping due to unrun test
dependencies. The second regression prioritized all tests with dependencies
in serial test runs.
The --build-options option consumes all following arguments until either
--build-target or --test-command. Fix the logic to allow this to be
zero options.
GenerateBuildCommand now knows how to lookup CMAKE_MAKE_PROGRAM or
choose a generator-provided default build tool. Therefore the
--build-makeprogram can now be optional and simply override the
default selection when provided.
Note that with --build-nocmake we now need to load the cache in order to
make the CMAKE_MAKE_PROGRAM entry available to GenerateBuildCommand.
This avoids requiring cmGlobalGenerator::GenerateBuildCommand to produce
a string so that it can be later refactored to produce a vector with no
quoting or escaping. It also makes the ctest_build command match what
"ctest -T Build" would run in a build tree configured with the new
build_command() command behavior. It also ensures that the native build
tool used matches that selected by the configuration of the tree to be
built.
Rename cmSystemTools::FindExecutableDirectory to FindCMakeResources.
Teach it to compute the locations of cmake, ctest, cpack, ccmake, and
cmake-gui executables, and the location of CMAKE_ROOT. Provide this
information from static cmSystemTools::Get<resource>() methods.
Refactor code that needs these locations to use the new APIs.
Teach FindCMakeResources to use the OS X system API to lookup the
executable location. When running from the CMake build tree itself,
leave a file in the tree that FindCMakeResources can use to read the
location of the source tree. This avoids the need to compile the source
tree location into a binary that may be installed and used without the
source tree.
Teach the QtDialog on OS X to create a "cmake-gui" symlink in the build
tree next to "cmake" and the other tools, as is already done in the
install tree for the application bundle. This ensures a consistent set
of executables are available in one directory.
Make sure that CMakeCache.txt is the last file being removed since
the binary directory may be left in a state that is no longer
removable otherwise.
Also retry removal a couple of times which makes this more robust
on windows where file locks may temporarily prevent removal.
Instead of removing the depot name, which causes problems when looking
at the file change list in CDash, make the Update test remove a given
prefix from the files retrieved from Update.xml.
ff59365 CTest: fix dashboard issues associated with the ctest-fix-run-serial topic
7a665ae CTest: added test for RUN_SERIAL issue #14484384beff CTest: added comments that describe the basic test sorting approach
adbe00d CTest: removed redundant copy of test dependency set
1b750cb CTest: perform cycle test early
6d4d7ca CTest: consider previously failed tests before all others
e809d8c CTest: prioritize tests by their depth in the dependency graph
44017a4 CTest: handle dependent and non dependent test requirements equally
Teach the ctest_update implementation to use the p4 command-line
client to perform updates and extract the list of changes.
Add a CTest.UpdateP4 test like those that exist already for the other
version control tools. Make the test available when p4 and the p4d
server are found. During the test launch p4d in the background to
serve a repository from the test directory. Then direct the client
toward this server for the duration of the test.
We will no longer support full documentation generation from executables
and will instead generate documentation with other tools. Disable (with
a warning left behind) the command-line options:
--copyright
--help-compatcommands
--help-full
--help-html
--help-man
Drop supporting code. Drop manual sections generation from executables.
Remove internal documentation construction APIs. Drop unused sections
See Also, Author, Copyright, Compat Commands, Custom Modules.
This assumes that coverage.py has been run in such a way to produce its
standard XML output. This uses the Cobertura schema and should be somewhat
generalizable.
Add a new command line argument to ctest. This allows users to
rerun tests that failed during the previous call to ctest. This
is accomplished by analyzing the most recently modified file named
"^LastTestsFailed*" in the Testing/Temporary subdirectory of the
project's binary directory.
The commits 9db31162 (Remove CMake-language block-end command
arguments, 2012-08-13) and 77543bde (Convert CMake-language
commands to lower case, 2012-08-13) changed most cmake code
to use lowercase commands and no parameters in termination
commands. However, those changes excluded cmake code generated
in c++ by cmake.
Make a similar style change to code generated by cmake.
The output file used for memory checker runs must be unique for every test run
in parallel, so simply make them unique for every test run. Simply use the test
index to avoid collisions.
This reverts commit 6187876dea.
It was actually possible before to have paths with spaces in them, the spaces
just need to be quoted. This way spaces will work as argument separators.
10bc50e Tests: ignore Guard Malloc messages in MemChecker tests
159c3e9 Tests: add a test with custom options passed to valgrind
61ddb93 CTest: fix comment documenting cmBoundsCheckerParser class
cbdfcb0 Tests: add test for non-existent Valgrind suppression file
3b5b758 CTest: drop suppression for gcc 2.9.6 errors from default Valgrind flags
7752253 Tests: verify that memory checker output files are always present
abf1df4 Tests: remove code duplication in CTestTestMemCheck tests
f499422 CTest: remove unreachable code and CTestTestMemcheckUnknown test
dde6306 CTest: use an output file for Valgrind (#14110)
bcc0f3f Tests: create output files for all memory checkers
At this point we do not care what the current directory is, as nothing that
requires a specific directory is done. Just make sure that at the end we are
back in the same directory as at the start.
PASS_REGULAR_EXPRESSION and FAIL_REGULAR_EXPRESSION both take a list of expressions.
Stop searching if the first of those has matched the output as the result will not
change anymore.
Because of clock scew between processors or just because of someone changing
the system time the end timestamp may be before the start time. Reporting a
negative time doesn't any sense, just report zero there as it already happens
for really fast tests.
The --workaround-gcc296-bugs has been part of the default Valgrind flags since
Valgrind support was added in commit 5b232ded15
(ENH: Add initial memory check support which works for Valgrind, 2003-12-15).
The Valgrind manpage says that this option should be avoided if not really
needed as it may cause real errors to get ignored. If someone uses a compiler
that really needs the flag this flag should be set by the user explicitely.
Most users will never set any flags and probably never notice that they use a
flag they shouldn't.
The memory checker command can't be quoted at this point, because previously it
has been tested that the given file exists, which will fail if the name is
quoted. The CTestTestMemcheckUnknown test aimed to test this case, has always
failed to do so and serves no useful purpose therefore.
This allows the ctest_build command's TARGET option to name a target
in a subdirectory and still build properly with msbuild.
Add test case covering use of ctest_build() with such a TARGET.
In cmCTestHandlerCommand::InitialPass call InitializeHandler after setting
the CTest "SourceDirectory" and "BuildDirectory" configuration values
instead of before. This makes the values available to InitializeHandler
methods in subclasses, so also drop the duplicate configuration of those
values from cmCTestConfigureCommand::InitializeHandler.
Extend the cmGlobalGenerator::GenerateBuildCommand virtual method
signature with a "projectDir" parameter specifying the top of the
project build tree for which the build command will be generated.
Populate it from call sites in cmGlobalGenerator::Build where a
fully-generated build tree should be available.
The ctest_configure command already reads the CTEST_CMAKE_GENERATOR
variable to get the value for the cmake -G option. Read new variable
CTEST_CMAKE_GENERATOR_TOOLSET for -T.
The "ctest --build-and-test" mode already has "--build-generator" to
specify the -G option to CMake. Add a "--build-generator-toolset" option
to specify the -T value.
Note it is still possible for CTest to start more than the number of
processes specified by PARALLEL_LEVEL, but this prevents the number of
tests to start from being unbounded because of overflow.
This is for consistency throughout cmake. The cmsys version exists
becaues uses of auto_ptr types as return types does not work with
some implementations in ancient compilers.
995a35f CTest: add a check with a quoted memory checker
de8bffc CTest: add a test for CTEST_CUSTOM_MEMCHECK_IGNORE
d26c9b6 CTest: improve memory checker type detection
fcae1da CTest: add tests that simulate memcheck runs
6187876 CTest: fix pre and post test commands with spaces
95bc8aa CTest: fix usage of memory checker with spaces in path
If the checker was explicitely set with a "TypeCommand" variable (e.g.
ValgrindCommand) then we now just believe that this is valgrind, even if
nothing in the path matches "valgrind". Only when "MemoryCheckCommand" was used
we still scan the path to find out what checker we have.
Hack access to cmCTestVC::Revision on this compiler to avoid errors like
Source/CTest/cmCTestSVN.cxx, line 23: Error: cmCTestSVN::SVNInfo is not accessible from cmCTestSVN::Revision.
Source/CTest/cmCTestSVN.cxx, line 22: Error: cmCTestVC::Revision is not accessible from file level.
If the pre or post memcheck or test commands have spaces in the path these
were never escaped, leading to broken commands. This was not covered in the
test suite so it went unnoticed.
The filename was escaped in cmCTestMemCheckHandler::InitializeMemoryChecking()
and again before it was written to output in
cmCTestRunTest::ComputeArguments().
Once someone uses e.g. a valgrind path with spaces this leads to double escaping
making the memory checker fail completely because of the invalid path.
This call 'svn status' and parse the result to get the list of externals
repositories. The external repositories found are added to the
Repositories list.
- Suppress the URL, Root, Base fields from the cmCTestSVN class
- Update the code to use RootInfo instead
- LoadInfo(), GuessBase(), and a new LoadRevision() functions work on a given SVNInfo
- Use the implementation from the base class for LocalPath(), correct path is built by SVNInfo::BuildLocalPath() instead
The Repository list will contain the SVNInfo of all the repositories
(root and external ones). The RootInfo pointer will point to the
SVNInfo structure of the root repository.
c806b23 CDash now supports lots of files in coverage. So, show all files.
761d931 Do not try to run bullseye coverage if COVFILE env is empty.
5b69ce4 Update test data to match new coverage format.
1b418f1 Change GT.M Coverage Parser global
b0c07a1 Disable bullseye coverage for mumps coverage test.
0a169e6 Remove uncovered files from cache coverage data.
a7abf5e Add ability to specify more than one package directory or coverage directory.
220afca Use <TARGET_FILE> expression to run ctest so it works with Xcode and VS IDE.
62f6bce Use a script to run the test because WORKING_DIRECTORY is not in 2.8.2.
f5c5db0 Fix some warnings and a bug where it went past the length of a vector.
7955e99 Add support for Cache coverage.
a86cd33 Add virutal destructor to silence warning.
319eeb0 Add test for mumps coverage. Also refactor code to prepare for cache coverage.
72210c2 Fix line length.
dd07161 Fix warning about char* instead of const char*.
e6412e0 Add support to ctest for GTM mumps coverage.
The coverage global should be in the local namespace. This means the global
will be ^ZZCOVERAGE instead of ^COVERAGE. Change the parser to look for ^ZZCOVERAGE
instead of the old ^COVERAGE
Replace use of AppendEnv/RestoreEnv pairs with instances of
SaveRestoreEnvironment. Simplify the signature of AppendEnv and use it
in place of similar loops elsewhere. Move the RestoreEnv implementation
inside the SaveRestoreEnvironment destructor which is the only place
left that calls it.
Commit ffbe61bb (make sure english is used for output of gcov, 2008-05-10)
taught ctest_coverage to set LC_ALL=POSIX to get English output from gcov.
Use the more portable value LC_ALL=C and restore the original value when
finished.
Even if there are no lines covered in the file the gcov coverage report still
contains valueable information, the amount of uncovered lines and which exactly
they are. Set 'Covered="true"' for files we have a gcov report for even if they
have no lines covered. Otherwise CDash will neither show the uncovered line
count nor the detailed coverage report for this file.
When CTEST_EXTRA_COVERAGE_GLOB was used to collect otherwise uncovered files
'Covered="true"' was unconditionally set, so this can't be worse here.
Fix the regex meant to match errors of the form:
C:\some\dir\source.cpp(17): catastrophic error: could not open source file "some_header.h"
to make the space between ')' and ':' optional.
COVERAGE_EXTRA_FLAGS is a space separated value of extra flags
that will be passed to gcov when ctest's coverage handler invokes
gcov to do coverage analysis.
Map to CoverageExtraFlags in the CTest ini file. Use default value
of "-l" to match the coverage handler's earlier behavior from ctest
2.8.4 and earlier. The fix for related issue #11717 had added a " -p"
which was the cause of both #12415 and #12490. Here, we revert that
change to the default value, so -p is no longer there by default.
The people that care to add -p may do so in their own build trees
by appending " -p" to the new cache variable COVERAGE_EXTRA_FLAGS.
Previous code was missing some matches in the output. This commit
fixes the regular expressions used for output matching to detect
numbers reported with commas in them, too.
This deterministic ordering makes it easier for a developer
comparing the ctest output of two test runs to see what changed
from run to run.
Thanks to Fraser Hutchison for the patch.
When add_test(NAME) is called without the CONFIGURATIONS argument then
the test is intended to run in any configuration. In multi-config
generators like the VS IDE and Xcode tests created by add_test(NAME) can
only be run when testing a known configuration (otherwise there is no
way to generate the test command line). If no test command line is
known for a particular configuration, or if no configuration is given to
ctest, report the test as not run instead of silently skipping it.
Also fix CMake's own TestsWorkingDirectory test invocation to correct a
previously silent failure exposed by this change.
3e32db7 cmCTestUploadCommand::CheckArgumentKeyword should return false if not FILES
6b6f309 Add the FILES keyword to ctest_upload command
28cdd0a Don't tar/gz ctest_upload() files
fbe4356 Change 'Files' tag to 'Upload' in Upload.xml
350546d Implement ctest_upload command
Some samples of things that got unnoticed by our nightly builds:
$ JAVA_HOME= mvn
Warning: JAVA_HOME environment variable is not set.
...
$ mvn
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD FAILURE
...
Several major Linux distributions still do not provide Git >= 1.6.5.0 in
their stable package lists. Prior to commit 1173cc4a (Update Git
submodules with --recursive, 2011-02-22) CTest was able to use older Git
versions but simply silently failed to update submodules recursively.
Instead of failing with older Git versions preserve the status quo and
add a warning in the update log. Users testing projects with recursive
submodules may simply update to a Git new enough to support them.
Using sort results in a possibly-modified sorting
when all elements are "tied" - use stable_sort instead
to preserve the original ordering of tied elements.
Joseph Snyder
Brad King
Stephen Kelly
Nils Gladitz
Bill Hoffman
Ben Boeckel
Clinton Stimpson
Jonathan Beezley
Zach Mullen
Roni Choudhury
Volkan Gezer
Zack Galbreath
Alex Neundorf
Pedro Navarro
Rolf Eike Beer
Patrick Reynolds
Sean McBride
Andreas Mohr
Petr Kmoch
Casey Goodlett
David Cole
Nils Gladitz
Xavier Besseron
Kitware Robot
Alexandru Ciobanu
Yury G. Kudryashov
Nicolas Despres
Johannes Stallkamp
Thomas Jarosch
Johan Björk