kolan
/
CMake
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

ExternalData: Split documentation into sections 

Also explicitly mark functions and variables.
This commit is contained in:
Brad King 2015-01-12 14:38:22 -05:00
parent 4ab5c652b6
commit f3884b47ec
1 changed files with 107 additions and 63 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

146
Modules/ExternalData.cmake
View File

@ -2,8 +2,15 @@
ExternalData ExternalData
------------ ------------
.. only:: html
.. contents::
Manage data files stored outside source tree Manage data files stored outside source tree
Introduction
^^^^^^^^^^^^
Use this module to unambiguously reference data files stored outside Use this module to unambiguously reference data files stored outside
the source tree and fetch them at build time from arbitrary local and the source tree and fetch them at build time from arbitrary local and
remote content-addressed locations. Functions provided by this module remote content-addressed locations. Functions provided by this module
@ -19,10 +26,13 @@ For example, the argument ``DATA{img.png}`` may be satisfied by either a
real ``img.png`` file in the current source directory or a ``img.png.md5`` real ``img.png`` file in the current source directory or a ``img.png.md5``
file containing its MD5 sum. file containing its MD5 sum.
The ``ExternalData_Expand_Arguments`` function evaluates ``DATA{}`` Module Functions
references in its arguments and constructs a new list of arguments: ^^^^^^^^^^^^^^^^
.. code-block:: cmake .. command:: ExternalData_Expand_Arguments
The ``ExternalData_Expand_Arguments`` function evaluates ``DATA{}``
references in its arguments and constructs a new list of arguments::
ExternalData_Expand_Arguments( ExternalData_Expand_Arguments(
<target> # Name of data management target <target> # Name of data management target
@ -30,39 +40,42 @@ references in its arguments and constructs a new list of arguments:
[args...] # Input arguments, DATA{} allowed [args...] # Input arguments, DATA{} allowed
) )
It replaces each ``DATA{}`` reference in an argument with the full path of It replaces each ``DATA{}`` reference in an argument with the full path of
a real data file on disk that will exist after the ``<target>`` builds. a real data file on disk that will exist after the ``<target>`` builds.
The ``ExternalData_Add_Test`` function wraps around the CMake .. command:: ExternalData_Add_Test
:command:`add_test` command but supports ``DATA{}`` references in
its arguments:
.. code-block:: cmake The ``ExternalData_Add_Test`` function wraps around the CMake
:command:`add_test` command but supports ``DATA{}`` references in
its arguments::
ExternalData_Add_Test( ExternalData_Add_Test(
<target> # Name of data management target <target> # Name of data management target
... # Arguments of add_test(), DATA{} allowed ... # Arguments of add_test(), DATA{} allowed
) )
It passes its arguments through ``ExternalData_Expand_Arguments`` and then It passes its arguments through ``ExternalData_Expand_Arguments`` and then
invokes the :command:`add_test` command using the results. invokes the :command:`add_test` command using the results.
The ``ExternalData_Add_Target`` function creates a custom target to .. command:: ExternalData_Add_Target
manage local instances of data files stored externally:
.. code-block:: cmake The ``ExternalData_Add_Target`` function creates a custom target to
manage local instances of data files stored externally::
ExternalData_Add_Target( ExternalData_Add_Target(
<target> # Name of data management target <target> # Name of data management target
) )
It creates custom commands in the target as necessary to make data It creates custom commands in the target as necessary to make data
files available for each ``DATA{}`` reference previously evaluated by files available for each ``DATA{}`` reference previously evaluated by
other functions provided by this module. A list of URL templates may other functions provided by this module. A list of URL templates may
be provided in the variable ``ExternalData_URL_TEMPLATES`` using the be provided in the variable ``ExternalData_URL_TEMPLATES`` using the
placeholders ``%(algo)`` and ``%(hash)`` in each template. Data fetch placeholders ``%(algo)`` and ``%(hash)`` in each template. Data fetch
rules try each URL template in order by substituting the hash rules try each URL template in order by substituting the hash
algorithm name for ``%(algo)`` and the hash value for ``%(hash)``. algorithm name for ``%(algo)`` and the hash value for ``%(hash)``.
Hash Algorithms
^^^^^^^^^^^^^^^
The following hash algorithms are supported:: The following hash algorithms are supported::
@ -78,7 +91,8 @@ The following hash algorithms are supported::
Note that the hashes are used only for unique data identification and Note that the hashes are used only for unique data identification and
download verification. download verification.
Example usage: Example Usage
^^^^^^^^^^^^^
.. code-block:: cmake .. code-block:: cmake
@ -98,6 +112,9 @@ replaced by the full path to a real instance of the data file
such as ``MyInput.png.md5`` then the ``MyData`` target creates a real such as ``MyInput.png.md5`` then the ``MyData`` target creates a real
``MyInput.png`` in the build tree. ``MyInput.png`` in the build tree.
Referencing File Series
^^^^^^^^^^^^^^^^^^^^^^^
The ``DATA{}`` syntax can be told to fetch a file series using the form The ``DATA{}`` syntax can be told to fetch a file series using the form
``DATA{<name>,:}``, where the ``:`` is literal. If the source tree ``DATA{<name>,:}``, where the ``:`` is literal. If the source tree
contains a group of files or content links named like a series then a contains a group of files or content links named like a series then a
@ -126,6 +143,9 @@ Configure series number matching with a regex that matches the
Note that the ``<suffix>`` of a series does not include a hash-algorithm Note that the ``<suffix>`` of a series does not include a hash-algorithm
extension. extension.
Referencing Associated Files
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The ``DATA{}`` syntax can alternatively match files associated with the The ``DATA{}`` syntax can alternatively match files associated with the
named file and contained in the same directory. Associated files may named file and contained in the same directory. Associated files may
be specified by options using the syntax be specified by options using the syntax
@ -139,6 +159,9 @@ syntax ``REGEX:<regex>``. For example, the arguments::
will pass ``MyInput.mha`` and ``MyFrames00.png`` on the command line but will pass ``MyInput.mha`` and ``MyFrames00.png`` on the command line but
ensure that the associated files are present next to them. ensure that the associated files are present next to them.
Referencing Directories
^^^^^^^^^^^^^^^^^^^^^^^
The ``DATA{}`` syntax may reference a directory using a trailing slash and The ``DATA{}`` syntax may reference a directory using a trailing slash and
a list of associated files. The form ``DATA{<name>/,<opt1>,<opt2>,...}`` a list of associated files. The form ``DATA{<name>/,<opt1>,<opt2>,...}``
adds rules to fetch any files in the directory that match one of the adds rules to fetch any files in the directory that match one of the
@ -148,39 +171,60 @@ directory on the command line and ensure that the directory contains
files corresponding to every file or content link in the ``MyDataDir`` files corresponding to every file or content link in the ``MyDataDir``
source directory. source directory.
The variable ``ExternalData_LINK_CONTENT`` may be set to the name of a Module Variables
supported hash algorithm to enable automatic conversion of real data ^^^^^^^^^^^^^^^^
files referenced by the ``DATA{}`` syntax into content links. For each
such ``<file>`` a content link named ``<file><ext>`` is created. The
original file is renamed to the form ``.ExternalData_<algo>_<hash>`` to
stage it for future transmission to one of the locations in the list
of URL templates (by means outside the scope of this module). The
data fetch rule created for the content link will use the staged
object if it cannot be found using any URL template.
The variable ``ExternalData_OBJECT_STORES`` may be set to a list of local The following variables configure behavior. They should be set before
directories that store objects using the layout ``<dir>/%(algo)/%(hash)``. calling any of the functions provided by this module.
These directories will be searched first for a needed object. If the
object is not available in any store then it will be fetched remotely
using the URL templates and added to the first local store listed. If
no stores are specified the default is a location inside the build
tree.
The variable ``ExternalData_SOURCE_ROOT`` may be set to the highest source .. variable:: ExternalData_LINK_CONTENT
directory containing any path named by a ``DATA{}`` reference. The
default is ``CMAKE_SOURCE_DIR``. ``ExternalData_SOURCE_ROOT`` and
``CMAKE_SOURCE_DIR`` must refer to directories within a single source
distribution (e.g. they come together in one tarball).
The variable ``ExternalData_BINARY_ROOT`` may be set to the directory to The ``ExternalData_LINK_CONTENT`` variable may be set to the name of a
hold the real data files named by expanded ``DATA{}`` references. The supported hash algorithm to enable automatic conversion of real data
default is ``CMAKE_BINARY_DIR``. The directory layout will mirror that of files referenced by the ``DATA{}`` syntax into content links. For each
content links under ``ExternalData_SOURCE_ROOT``. such ``<file>`` a content link named ``<file><ext>`` is created. The
original file is renamed to the form ``.ExternalData_<algo>_<hash>`` to
stage it for future transmission to one of the locations in the list
of URL templates (by means outside the scope of this module). The
data fetch rule created for the content link will use the staged
object if it cannot be found using any URL template.
Variables ``ExternalData_TIMEOUT_INACTIVITY`` and .. variable:: ExternalData_OBJECT_STORES
``ExternalData_TIMEOUT_ABSOLUTE`` set the download inactivity and absolute
timeouts, in seconds. The defaults are 60 seconds and 300 seconds, The ``ExternalData_OBJECT_STORES`` variable may be set to a list of local
respectively. Set either timeout to 0 seconds to disable enforcement. directories that store objects using the layout ``<dir>/%(algo)/%(hash)``.
These directories will be searched first for a needed object. If the
object is not available in any store then it will be fetched remotely
using the URL templates and added to the first local store listed. If
no stores are specified the default is a location inside the build
tree.
.. variable:: ExternalData_SOURCE_ROOT
The ``ExternalData_SOURCE_ROOT`` variable may be set to the highest source
directory containing any path named by a ``DATA{}`` reference. The
default is ``CMAKE_SOURCE_DIR``. ``ExternalData_SOURCE_ROOT`` and
``CMAKE_SOURCE_DIR`` must refer to directories within a single source
distribution (e.g. they come together in one tarball).
.. variable:: ExternalData_BINARY_ROOT
The ``ExternalData_BINARY_ROOT`` variable may be set to the directory to
hold the real data files named by expanded ``DATA{}`` references. The
default is ``CMAKE_BINARY_DIR``. The directory layout will mirror that of
content links under ``ExternalData_SOURCE_ROOT``.
.. variable:: ExternalData_TIMEOUT_INACTIVITY
The ``ExternalData_TIMEOUT_INACTIVITY`` variable sets the download
inactivity timeout, in seconds, with a default of ``60`` seconds.
Set to ``0`` to disable enforcement.
.. variable:: ExternalData_TIMEOUT_ABSOLUTE
The ``ExternalData_TIMEOUT_ABSOLUTE`` variable sets the download
absolute timeout, in seconds, with a default of ``300`` seconds.
Set to ``0`` to disable enforcement.
#]=======================================================================] #]=======================================================================]
#============================================================================= #=============================================================================