kolan
/
CMake
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Help: Document ctest dashboard client usage 

Extend the ctest(1) manual with a section describing configuration
and usage of CTest as a CDash software quality dashboard client.
This commit is contained in:
Brad King 2014-05-30 14:16:47 -04:00
parent 28f73dffbf
commit 2d57a390da
2 changed files with 603 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
Help/manual/cmake-commands.7.rst
View File

@ -128,6 +128,8 @@ versions of CMake. Do not use them in new code.
/command/variable_requires /command/variable_requires
/command/write_file /command/write_file
.. _`CTest Commands`:
CTest Commands CTest Commands
============== ==============

601
Help/manual/ctest.1.rst
View File

@ -365,6 +365,607 @@ Options
.. include:: OPTIONS_HELP.txt .. include:: OPTIONS_HELP.txt
Dashboard Client
================
CTest can operate as a client for the `CDash`_ software quality dashboard
application. As a dashboard client, CTest performs a sequence of steps
to configure, build, and test software, and then submits the results to
a `CDash`_ server.
.. _`CDash`: http://cdash.org/
Dashboard Client Steps
----------------------
CTest defines an ordered list of testing steps of which some or all may
be run as a dashboard client:
``Start``
Start a new dashboard submission to be composed of results recorded
by the following steps.
See the `CTest Start Step`_ section below.
``Update``
Update the source tree from its version control repository.
Record the old and new versions and the list of updated source files.
See the `CTest Update Step`_ section below.
``Configure``
Configure the software by running a command in the build tree.
Record the configuration output log.
See the `CTest Configure Step`_ section below.
``Build``
Build the software by running a command in the build tree.
Record the build output log and detect warnings and errors.
See the `CTest Build Step`_ section below.
``Test``
Test the software by loading a ``CTestTestfile.cmake``
from the build tree and executing the defined tests.
Record the output and result of each test.
See the `CTest Test Step`_ section below.
``Coverage``
Compute coverage of the source code by running a coverage
analysis tool and recording its output.
See the `CTest Coverage Step`_ section below.
``MemCheck``
Run the software test suite through a memory check tool.
Record the test output, results, and issues reported by the tool.
See the `CTest MemCheck Step`_ section below.
``Submit``
Submit results recorded from other testing steps to the
software quality dashboard server.
See the `CTest Submit Step`_ section below.
Dashboard Client Modes
----------------------
CTest defines three modes of operation as a dashboard client:
``Nightly``
This mode is intended to be invoked once per day, typically at night.
It enables the ``Start``, ``Update``, ``Configure``, ``Build``, ``Test``,
``Coverage``, and ``Submit`` steps by default. Selected steps run even
if the ``Update`` step reports no changes to the source tree.
``Continuous``
This mode is intended to be invoked repeatedly throughout the day.
It enables the ``Start``, ``Update``, ``Configure``, ``Build``, ``Test``,
``Coverage``, and ``Submit`` steps by default, but exits after the
``Update`` step if it reports no changes to the source tree.
``Experimental``
This mode is intended to be invoked by a developer to test local changes.
It enables the ``Start``, ``Configure``, ``Build``, ``Test``, ``Coverage``,
and ``Submit`` steps by default.
Dashboard Client via CTest Command-Line
---------------------------------------
CTest can perform testing on an already-generated build tree.
Run the ``ctest`` command with the current working directory set
to the build tree and use one of these signatures::
ctest -D <mode>[<step>]
ctest -M <mode> [ -T <step> ]...
The ``<mode>`` must be one of the above `Dashboard Client Modes`_,
and each ``<step>`` must be one of the above `Dashboard Client Steps`_.
CTest reads the `Dashboard Client Configuration`_ settings from
a file in the build tree called either ``CTestConfiguration.ini``
or ``DartConfiguration.tcl`` (the names are historical). The format
of the file is::
# Lines starting in '#' are comments.
# Other non-blank lines are key-value pairs.
<setting>: <value>
where ``<setting>`` is the setting name and ``<value>`` is the
setting value.
In build trees generated by CMake, this configuration file is
generated by the :module:`CTest` module if included by the project.
The module uses variables to obtain a value for each setting
as documented with the settings below.
.. _`CTest Script`:
Dashboard Client via CTest Script
---------------------------------
CTest can perform testing driven by a :manual:`cmake-language(7)`
script that creates and maintains the source and build tree as
well as performing the testing steps. Run the ``ctest`` command
with the current working directory set outside of any build tree
and use one of these signatures::
ctest -S <script>
ctest -SP <script>
The ``<script>`` file must call :ref:`CTest Commands` commands
to run testing steps explicitly as documented below. The commands
obtain `Dashboard Client Configuration`_ settings from their
arguments or from variables set in the script.
Dashboard Client Configuration
==============================
The `Dashboard Client Steps`_ may be configured by named
settings as documented in the following sections.
.. _`CTest Start Step`:
CTest Start Step
----------------
Start a new dashboard submission to be composed of results recorded
by the following steps.
In a `CTest Script`_, the :command:`ctest_start` command runs this step.
Arguments to the command may specify some of the step settings.
The command first runs the command-line specified by the
``CTEST_CHECKOUT_COMMAND`` variable, if set, to initialize the source
directory.
Configuration settings include:
``BuildDirectory``
The full path to the project build tree.
* `CTest Script`_ variable: ``CTEST_BINARY_DIRECTORY``
* :module:`CTest` module variable: :variable:`PROJECT_BINARY_DIR`
``SourceDirectory``
The full path to the project source tree.
* `CTest Script`_ variable: ``CTEST_SOURCE_DIRECTORY``
* :module:`CTest` module variable: :variable:`PROJECT_SOURCE_DIR`
.. _`CTest Update Step`:
CTest Update Step
-----------------
In a `CTest Script`_, the :command:`ctest_update` command runs this step.
Arguments to the command may specify some of the step settings.
Configuration settings to specify the version control tool include:
``BZRCommand``
``bzr`` command-line tool to use if source tree is managed by Bazaar.
* `CTest Script`_ variable: ``CTEST_BZR_COMMAND``
* :module:`CTest` module variable: none
``BZRUpdateOptions``
Command-line options to the ``BZRCommand`` when updating the source.
* `CTest Script`_ variable: ``CTEST_BZR_UPDATE_OPTIONS``
* :module:`CTest` module variable: none
``CVSCommand``
``cvs`` command-line tool to use if source tree is managed by CVS.
* `CTest Script`_ variable: ``CTEST_CVS_COMMAND``
* :module:`CTest` module variable: ``CVSCOMMAND``
``CVSUpdateOptions``
Command-line options to the ``CVSCommand`` when updating the source.
* `CTest Script`_ variable: ``CTEST_CVS_UPDATE_OPTIONS``
* :module:`CTest` module variable: ``CVS_UPDATE_OPTIONS``
``GITCommand``
``git`` command-line tool to use if source tree is managed by Git.
* `CTest Script`_ variable: ``CTEST_GIT_COMMAND``
* :module:`CTest` module variable: ``GITCOMMAND``
``GITUpdateCustom``
Specify a semicolon-separated list of custom command lines to run
in the source tree (Git work tree) to update it instead of running
the ``GITCommand``.
* `CTest Script`_ variable: ``CTEST_GIT_UPDATE_CUSTOM``
* :module:`CTest` module variable: ``CTEST_GIT_UPDATE_CUSTOM``
``GITUpdateOptions``
Command-line options to the ``GITCommand`` when updating the source.
* `CTest Script`_ variable: ``CTEST_GIT_UPDATE_OPTIONS``
* :module:`CTest` module variable: ``GIT_UPDATE_OPTIONS``
``HGCommand``
``hg`` command-line tool to use if source tree is managed by Mercurial.
* `CTest Script`_ variable: ``CTEST_HG_COMMAND``
* :module:`CTest` module variable: none
``HGUpdateOptions``
Command-line options to the ``HGCommand`` when updating the source.
* `CTest Script`_ variable: ``CTEST_HG_UPDATE_OPTIONS``
* :module:`CTest` module variable: none
``P4Client``
Value of the ``-c`` option to the ``P4Command``.
* `CTest Script`_ variable: ``CTEST_P4_CLIENT``
* :module:`CTest` module variable: ``CTEST_P4_CLIENT``
``P4Command``
``p4`` command-line tool to use if source tree is managed by Perforce.
* `CTest Script`_ variable: ``CTEST_P4_COMMAND``
* :module:`CTest` module variable: ``P4COMMAND``
``P4Options``
Command-line options to the ``P4Command`` for all invocations.
* `CTest Script`_ variable: ``CTEST_P4_OPTIONS``
* :module:`CTest` module variable: ``CTEST_P4_OPTIONS``
``P4UpdateCustom``
Specify a semicolon-separated list of custom command lines to run
in the source tree (Perforce tree) to update it instead of running
the ``P4Command``.
* `CTest Script`_ variable: none
* :module:`CTest` module variable: ``CTEST_P4_UPDATE_CUSTOM``
``P4UpdateOptions``
Command-line options to the ``P4Command`` when updating the source.
* `CTest Script`_ variable: ``CTEST_P4_UPDATE_OPTIONS``
* :module:`CTest` module variable: ``CTEST_P4_UPDATE_OPTIONS``
``SVNCommand``
``svn`` command-line tool to use if source tree is managed by Subversion.
* `CTest Script`_ variable: ``CTEST_SVN_COMMAND``
* :module:`CTest` module variable: ``SVNCOMMAND``
``SVNOptions``
Command-line options to the ``SVNCommand`` for all invocations.
* `CTest Script`_ variable: ``CTEST_SVN_OPTIONS``
* :module:`CTest` module variable: ``CTEST_SVN_OPTIONS``
``SVNUpdateOptions``
Command-line options to the ``SVNCommand`` when updating the source.
* `CTest Script`_ variable: ``CTEST_SVN_UPDATE_OPTIONS``
* :module:`CTest` module variable: ``SVN_UPDATE_OPTIONS``
``UpdateCommand``
Specify the version-control command-line tool to use without
detecting the VCS that manages the source tree.
* `CTest Script`_ variable: ``CTEST_UPDATE_COMMAND``
* :module:`CTest` module variable: ``<VCS>COMMAND``
when ``UPDATE_TYPE`` is ``<vcs>``, else ``UPDATE_COMMAND``
``UpdateOptions``
Command-line options to the ``UpdateCommand``.
* `CTest Script`_ variable: ``CTEST_UPDATE_OPTIONS``
* :module:`CTest` module variable: ``<VCS>_UPDATE_OPTIONS``
when ``UPDATE_TYPE`` is ``<vcs>``, else ``UPDATE_OPTIONS``
``UpdateType``
Specify the version-control system that manages the source
tree if it cannot be detected automatically.
The value may be ``bzr``, ``cvs``, ``git``, ``hg``,
``p4``, or ``svn``.
* `CTest Script`_ variable: none, detected from source tree
* :module:`CTest` module variable: ``UPDATE_TYPE`` if set,
else ``CTEST_UPDATE_TYPE``
Additional configuration settings include:
``NightlyStartTime``
In the ``Nightly`` dashboard mode, specify the "nightly start time".
With centralized version control systems (``cvs`` and ``svn``),
the ``Update`` step checks out the version of the software as of
this time so that multiple clients choose a common version to test.
This is not well-defined in distributed version-control systems so
the setting is ignored.
* `CTest Script`_ variable: ``CTEST_NIGHTLY_START_TIME``
* :module:`CTest` module variable: ``NIGHTLY_START_TIME`` if set,
else ``CTEST_NIGHTLY_START_TIME``
.. _`CTest Configure Step`:
CTest Configure Step
--------------------
In a `CTest Script`_, the :command:`ctest_configure` command runs this step.
Arguments to the command may specify some of the step settings.
Configuration settings include:
``ConfigureCommand``
Command-line to launch the software configuration process.
It will be executed in the location specified by the
``BuildDirectory`` setting.
* `CTest Script`_ variable: ``CTEST_CONFIGURE_COMMAND``
* :module:`CTest` module variable: :variable:`CMAKE_COMMAND`
followed by :variable:`PROJECT_SOURCE_DIR`
.. _`CTest Build Step`:
CTest Build Step
----------------
In a `CTest Script`_, the :command:`ctest_build` command runs this step.
Arguments to the command may specify some of the step settings.
Configuration settings include:
``DefaultCTestConfigurationType``
When the build system to be launched allows build-time selection
of the configuration (e.g. ``Debug``, ``Release``), this specifies
the default configuration to be built when no ``-C`` option is
given to the ``ctest`` command. The value will be substituted into
the value of ``MakeCommand`` to replace the literal string
``${CTEST_CONFIGURATION_TYPE}`` if it appears.
* `CTest Script`_ variable: ``CTEST_CONFIGURATION_TYPE``
* :module:`CTest` module variable: ``DEFAULT_CTEST_CONFIGURATION_TYPE``,
initialized by the ``CMAKE_CONFIG_TYPE`` environment variable
``MakeCommand``
Command-line to launch the software build process.
It will be executed in the location specified by the
``BuildDirectory`` setting.
* `CTest Script`_ variable: ``CTEST_BUILD_COMMAND``
* :module:`CTest` module variable: ``MAKECOMMAND``,
initialized by the :command:`build_command` command
``UseLaunchers``
For build trees generated by CMake using a Makefile generator
or the :generator:`Ninja` generator, specify whether the
``CTEST_USE_LAUNCHERS`` feature is enabled by the
:module:`CTestUseLaunchers` module (also included by the
:module:`CTest` module). When enabled, the generated build
system wraps each invocation of the compiler, linker, or
custom command line with a "launcher" that communicates
with CTest via environment variables and files to report
granular build warning and error information. Otherwise,
CTest must "scrape" the build output log for diagnostics.
* `CTest Script`_ variable: ``CTEST_USE_LAUNCHERS``
* :module:`CTest` module variable: ``CTEST_USE_LAUNCHERS``
.. _`CTest Test Step`:
CTest Test Step
---------------
In a `CTest Script`_, the :command:`ctest_test` command runs this step.
Arguments to the command may specify some of the step settings.
Configuration settings include:
``TimeOut``
The default timeout for each test if not specified by the
:prop_test:`TIMEOUT` test property.
* `CTest Script`_ variable: ``CTEST_TEST_TIMEOUT``
* :module:`CTest` module variable: ``DART_TESTING_TIMEOUT``
.. _`CTest Coverage Step`:
CTest Coverage Step
-------------------
In a `CTest Script`_, the :command:`ctest_coverage` command runs this step.
Arguments to the command may specify some of the step settings.
Configuration settings include:
``CoverageCommand``
Command-line tool to perform software coverage analysis.
It will be executed in the location specified by the
``BuildDirectory`` setting.
* `CTest Script`_ variable: ``CTEST_COVERAGE_COMMAND``
* :module:`CTest` module variable: ``COVERAGE_COMMAND``
``CoverageExtraFlags``
Specify command-line options to the ``CoverageCommand`` tool.
* `CTest Script`_ variable: ``CTEST_COVERAGE_EXTRA_FLAGS``
* :module:`CTest` module variable: ``COVERAGE_EXTRA_FLAGS``
.. _`CTest MemCheck Step`:
CTest MemCheck Step
-------------------
In a `CTest Script`_, the :command:`ctest_memcheck` command runs this step.
Arguments to the command may specify some of the step settings.
Configuration settings include:
``MemoryCheckCommand``
Command-line tool to perform dynamic analysis. Test command lines
will be launched through this tool.
* `CTest Script`_ variable: ``CTEST_MEMORYCHECK_COMMAND``
* :module:`CTest` module variable: ``MEMORYCHECK_COMMAND``
``MemoryCheckCommandOptions``
Specify command-line options to the ``MemoryCheckCommand`` tool.
They will be placed prior to the test command line.
* `CTest Script`_ variable: ``CTEST_MEMORYCHECK_COMMAND_OPTIONS``
* :module:`CTest` module variable: ``MEMORYCHECK_COMMAND_OPTIONS``
``MemoryCheckSuppressionFile``
Specify a file containing suppression rules for the
``MemoryCheckCommand`` tool. It will be passed with options
appropriate to the tool.
* `CTest Script`_ variable: ``CTEST_MEMORYCHECK_SUPPRESSIONS_FILE``
* :module:`CTest` module variable: ``MEMORYCHECK_SUPPRESSIONS_FILE``
Additional configuration settings include:
``BoundsCheckerCommand``
Specify a ``MemoryCheckCommand`` that is known to be command-line
compatible with Bounds Checker.
* `CTest Script`_ variable: none
* :module:`CTest` module variable: none
``PurifyCommand``
Specify a ``MemoryCheckCommand`` that is known to be command-line
compatible with Purify.
* `CTest Script`_ variable: none
* :module:`CTest` module variable: ``PURIFYCOMMAND``
``ValgrindCommand``
Specify a ``MemoryCheckCommand`` that is known to be command-line
compatible with Valgrind.
* `CTest Script`_ variable: none
* :module:`CTest` module variable: ``VALGRIND_COMMAND``
``ValgrindCommandOptions``
Specify command-line options to the ``ValgrindCommand`` tool.
They will be placed prior to the test command line.
* `CTest Script`_ variable: none
* :module:`CTest` module variable: ``VALGRIND_COMMAND_OPTIONS``
.. _`CTest Submit Step`:
CTest Submit Step
-----------------
In a `CTest Script`_, the :command:`ctest_submit` command runs this step.
Arguments to the command may specify some of the step settings.
Configuration settings include:
``BuildName``
Describe the dashboard client platform with a short string.
(Operating system, compiler, etc.)
* `CTest Script`_ variable: ``CTEST_BUILD_NAME``
* :module:`CTest` module variable: ``BUILDNAME``
``CDashVersion``
Specify the version of `CDash`_ on the server.
* `CTest Script`_ variable: none, detected from server
* :module:`CTest` module variable: ``CTEST_CDASH_VERSION``
``CTestSubmitRetryCount``
Specify a number of attempts to retry submission on network failure.
* `CTest Script`_ variable: none,
use the :command:`ctest_submit` ``RETRY_COUNT`` option.
* :module:`CTest` module variable: ``CTEST_SUBMIT_RETRY_COUNT``
``CTestSubmitRetryDelay``
Specify a delay before retrying submission on network failure.
* `CTest Script`_ variable: none,
use the :command:`ctest_submit` ``RETRY_DELAY`` option.
* :module:`CTest` module variable: ``CTEST_SUBMIT_RETRY_DELAY``
``CurlOptions``
Specify a semicolon-separated list of options to control the
Curl library that CTest uses internally to connect to the
server. Possible options are ``CURLOPT_SSL_VERIFYPEER_OFF``
and ``CURLOPT_SSL_VERIFYHOST_OFF``.
* `CTest Script`_ variable: ``CTEST_CURL_OPTIONS``
* :module:`CTest` module variable: ``CTEST_CURL_OPTIONS``
``DropLocation``
The path on the dashboard server to send the submission.
* `CTest Script`_ variable: ``CTEST_DROP_LOCATION``
* :module:`CTest` module variable: ``DROP_LOCATION`` if set,
else ``CTEST_DROP_LOCATION``
``DropMethod``
Specify the method by which results should be submitted to the
dashboard server. The value may be ``cp``, ``ftp``, ``http``,
``https``, ``scp``, or ``xmlrpc`` (if CMake was built with
support for it).
* `CTest Script`_ variable: ``CTEST_DROP_METHOD``
* :module:`CTest` module variable: ``DROP_METHOD`` if set,
else ``CTEST_DROP_METHOD``
``DropSite``
The dashboard server name
(for ``ftp``, ``http``, and ``https``, ``scp``, and ``xmlrpc``).
* `CTest Script`_ variable: ``CTEST_DROP_SITE``
* :module:`CTest` module variable: ``DROP_SITE`` if set,
else ``CTEST_DROP_SITE``
``DropSitePassword``
The dashboard server login password, if any
(for ``ftp``, ``http``, and ``https``).
* `CTest Script`_ variable: ``CTEST_DROP_SITE_PASSWORD``
* :module:`CTest` module variable: ``DROP_SITE_PASSWORD`` if set,
else ``CTEST_DROP_SITE_PASWORD``
``DropSiteUser``
The dashboard server login user name, if any
(for ``ftp``, ``http``, and ``https``).
* `CTest Script`_ variable: ``CTEST_DROP_SITE_USER``
* :module:`CTest` module variable: ``DROP_SITE_USER`` if set,
else ``CTEST_DROP_SITE_USER``
``IsCDash``
Specify whether the dashboard server is `CDash`_ or an older
dashboard server implementation requiring ``TriggerSite``.
* `CTest Script`_ variable: ``CTEST_DROP_SITE_CDASH``
* :module:`CTest` module variable: ``CTEST_DROP_SITE_CDASH``
``ScpCommand``
``scp`` command-line tool to use when ``DropMethod`` is ``scp``.
* `CTest Script`_ variable: ``CTEST_SCP_COMMAND``
* :module:`CTest` module variable: ``SCPCOMMAND``
``Site``
Describe the dashboard client host site with a short string.
(Hostname, domain, etc.)
* `CTest Script`_ variable: ``CTEST_SITE``
* :module:`CTest` module variable: ``SITE``,
initialized by the :command:`site_name` command
``TriggerSite``
Legacy option to support older dashboard server implementations.
Not used when ``IsCDash`` is true.
* `CTest Script`_ variable: ``CTEST_TRIGGER_SITE``
* :module:`CTest` module variable: ``TRIGGER_SITE`` if set,
else ``CTEST_TRIGGER_SITE``
See Also See Also
======== ========