The CTestTestFailedSubmit-http test was failing on the
hut11 Experimental dashboards with "Empty reply from
server" due to a localhost settings change.
If GIT_EXECUTABLE is not passed in, and is not available from
DartConfiguration.tcl or CTestConfiguration.ini, then make one
more last ditch attempt to get it from Update.xml, if there is
an Update.xml. For dashboards that have successfully done a
ctest_update call, there should be an Update.xml in the Testing
subdir of the binary tree. Parse that file for the git executable
recorded in the <UpdateCommand> element.
And make this test pass on those RogueResearch dashboard machines!
Git's diff-tree format has no '\n'-terminated blank line at the end of
its commit message body block if there are no diff lines. Instead the
message body is terminated by '\0' and there is no diff section. Teach
CTest to parse the format in this case.
If GIT_EXECUTABLE is not passed into the test, but the source
tree is a git checkout, then use GitCommand or UpdateCommand
from the ctest ini file to set its value. Presumably, a dashboard
running the test suite had to have set this properly in order
to do the ctest_update step.
Optionally hide the output of each external project build step by
wrapping its command in a launcher script. Make output available in log
files instead. This simplifies the output of external project builds.
At this point, CTestTest3 causes more problems than it's worth.
It uses CVS to grab a remote (over the network) copy of kwsys
code for testing. This causes some sort of problem nearly every
night on the nightly CMake dashboards. Worse: it causes problems
on different machines on different nights, then the next day, it's
fine again. So: remove this test and monitor the coverage.
If we lose a significant portion of code coverage, I will revert
this commit and re-activate the test. However, if we do not lose
a significant portion of code coverage, I will remove the code
for the test as well as removing it from the CMakeLists.txt file.
Brad King and I discussed this over the last few weeks, and we both
think we have sufficient coverage of all the checkout and update code
in other locally (non-network) based tests.
On the other hand, even if we do take a mild hit on coverage temporarily,
it should be relatively easy to increase our coverage again by adding
bits to those other locally based tests.
The bootstrap script works under MSYS, so test it. Use a launcher batch
file since 'ctest --build-and-test' is a Windows program and will not
honor the shebang line in the script.
Even though this test is checking that the ctest running it can handle
test output without newlines we should run the just-built CMake binary.
This allows the MemCheck test mode to check the correct CMake.
Use 'git fetch' followed by 'git reset' to update the source tree. This
is better than 'git pull' because it can handle a rewritten upstream
branch and does not leave local modifications. After fetch, parse
FETCH_HEAD to find the merge head that 'git pull' would choose to track
the upstream branch. Then reset to the selected head.
In the normal fast-forward case the behavior remains unchanged.
However, now local modifications and commits will be erased, and
upstream rewrites are handled smoothly. This ensures that the upstream
branch is tested as expected.
Teach (create|run)_dashboard_script macros to treat the argument as the
name of a build tree. Append '.cmake' to generate the dashboard script
name. This allows future re-use of the macros for multiple test
scripts.
Use it from ExternalProject and the ExternalProject test's
CMakeLists file rather than having duplicate find_program calls.
Add logic so that we do not try to use *.cmd variants of git
programs when using the MSYS Makefiles generator. Should fix
the last remaining dashboard issue with the new ExternalProject
git support additions.
Also, correct minor problem regarding placement of the local git
repo during test execution. On clean builds, it was being placed
incorrectly because of the ../.. relative reference. Use an absolute
path to place the local git repo in the proper directory, and only
use the relative reference when referring to it.
The Intel C compiler for Linux does not seem to reject any bad flags or
object files on its link lines. Work around the problem by using a
preprocessor #error directive to ensure that BADFLAG appears in the
build output. This does not really achieve the purpose of the tests but
it allows them to pass.
Double quote executable names that may have spaces in them.
Do not run the new git portions of the test on machines that
have git < version 1.6.5 on them.
The Borland librarian actually creates a BADFLAG.obj when the object is
missing the first time! This causes later tests to not reject it.
Instead use a Borland-specific variation on the flag.
Add archives of these file types and add to the test
cases covered in the ExternalProject test.
Also add an "Example" directory in the Tests/ExternalProject
directory containing the canonical simplest example of
ExternalProject usage.
Add a LinkFlags test series to check that these properties work. Since
no link flag is accepted everywhere we test for presence of flags by
adding a bad flag and looking for the complaint in the test output.
Echo results of calling git status before exiting with
an error. Add one special case so that the test may pass
on the dashmacmini2 continuous dashboard, despite a 'git
status' non-zero return code. More logic like this may
be required. I will re-evaluate based on tomorrow's
nightly dashboard runs.
Additionally, output some more information in both cvs
and git cases. When it is a cvs checkout, echo the contents
of CVS/Root and CVS/Repository to the test output. When it
is a git checkout, echo the output of 'git branch -a'.
This will allow us to see more details about any given CMake
source tree right in the CDash results for this test.
Improve FILE(DOWNLOAD ...):
- Add percent complete progress output to the FILE DOWNLOAD
command. This progress output is off by default to
preserve existing behavior. To turn it on, pass
SHOW_PROGRESS as an argument.
- Add EXPECTED_MD5 argument. Verify that the downloaded
file has the expected md5 sum after download is complete.
- Add documentation for SHOW_PROGRESS and EXPECTED_MD5.
When the destination file exists already and has the
expected md5 sum, then do not bother re-downloading
the file. ("Short circuit" return.)
Also, add a test that checks for the status output
indicating that the short circuit behavior is actually
occurring. Use a binary file for the test so that the
md5 sum is guaranteed to be the same on all platforms
regardless of "shifting text file line ending" issues.
Improve ExternalProject:
- Add argument URL_MD5.
- Add verify step that compares md5 sum of .tar.gz file
before extracting it.
- Add md5 check to download step, too, to prevent
unnecessary downloads.
- Emit a warning message when a file is not verified.
Indicate that the file may be corrupt or that no
checksum was specified.
Add the <Email>...</Email> element in Update.xml for each commit
reported. This field was defined by Dart but never really used.
Distributed version control systems use author name and email
instead of a user id, so now it makes sense to use this field.
Some Mac linkers produce the message
"file was built for unsupported file format which is not the
architecture being linked"
for this test. Update the test output regex to match it.
Git does not automatically checkout the matching version of a submodule
when it checks out a new version of the parent project in the work tree.
If the submodule reference changed in the parent project then we were
reporting the submodule path as a local modification. Work around the
problem in ctest_update using "git submodule update" after "git pull".
For projects with no submodules this is a no-op. See issue #10662.
Also add a submodule to the test project for CTest.UpdateGIT to test the
work-around.
For builds from Git repositories, add "-g<commit>" to the end of the
version number. If the source tree is modified, append "-dirty".
For builds from CVS checkouts, add "-cvs-<branch>".
If defined and non-empty, the value of CMAKE_TESTS_CDASH_SERVER should point
to a CDash server willing to accept submissions for a project named
PublicDashboard. On machines that also run a CDash dashboard, set this
variable to "http://localhost/CDash-trunk-Testing" so that the CMake tests
that submit dashboards do not have to send those submissions over the wire.
The CTestSubmitLargeOutput test runs a dashboard that has a test that produces
very large amount of output on stdout/stderr. Since we do not even want to
attempt to send such large output over the wire, this test is off by default
unless the CMAKE_TESTS_CDASH_SERVER server is localhost. This test is expected
to cause a submission failure when sent to CDash. It passes if the submit
results contain error output. It fails if the submit succeeds.
CMAKE_TESTS_CDASH_SERVER: CDash server used by CMake/Tests.
If not defined or "", this variable defaults to the server at
http://www.cdash.org/CDash.
If set explicitly to "NOTFOUND", curl tests and ctest tests that use the
network are skipped.
If set to something starting with "http://localhost/", the CDash is expected
to be an instance of CDash used for CDash testing, pointing to a
cdash4simpletest database. In these cases, the CDash dashboards should be
run first.
Commit "Teach CTest.Update tests to strongly check entries" (2010-02-09)
started checking Update.xml entries strongly. This revealed that some
cvs clients report "U CTestConfig.cmake" during update even though the
file did not change and it selects the same revision. As a result the
test fails with
Update.xml has extra unexpected entries:
Updated{CTestConfig.cmake}
We fix the test to tolerate this particular extra entry without failing.
We wrap the git executable in a shell script that touches one source
file after 'git pull'. This makes the file newer than the index even
though it has not actually changed. If CTest does not refresh the index
properly then the test will fail with a bogus modified file.
Previously these tests just checked for matching file names in the
Update.xml files. Now we check the update types (Updated, Modified, or
Conflicting) and reject unexpected extra entries.
At least one Fortran compiler does not provide a preprocessor symbol to
identify itself. Instead we try running unknown compilers with version
query flags known for each vendor and look for known output. Future
commits will add vendor-specific flags/output table entries.
The PathScale compiler silently accepts unknown options that start in
more than one '-':
$ touch foo.c
$ pathcc -c foo.c --junk
$ echo $?
0
$ pathcc -c foo.c ---junk
$ echo $?
0
$ pathcc -c foo.c -junk
pathcc ERROR parsing -junk: unknown flag
$ echo $?
2
We teach the TryCompile to pass a bogus flag with only one '-' instead
of three '-'s for this compiler.
We disable this test because PathScale Fortran mangles module symbols as
"MYSUB.in.MYMODULE" so we cannot interface with it from C. We already
did this for SunPro and MIPSpro.
Makefile dependencies must be escaped using cmLocalGenerator::Convert
with the cmLocalGenerator::MAKEFILE option. This fixes Fortran module
dependencies with spaces in the path. We test the fix by adding a space
to one of the module paths in the Fortran test.
If APPEND is given to ctest_start, it will read the tag from the current existing Testing/TAG file rather than creating a new one based on the current time stamp. This allows a developer to run several dashboard scripts in a row, all of which will share the same tag/stamp/buildid when they finally get submitted to CDash. Now you can split the running of build phases and test phases for the same dashboard row into multiple scripts.
The commit "Make CTestTestTimeout time configurable" added a CMake cache
variable CTestTestTimeout_TIME to configure the length of the timeout
used by the inner CTestTestTimeout test. The reason was to allow users
on slow machines to give this test some extra time without extending the
timeout for everyone. However, Cygwin CMake seems to load slowly enough
that it is worth a longer default timeout for that platform.
The commit "Submit Subversion directory path in Update.xml" added the
element <SVNPath>...</SVNPath> to Update.xml for Subversion work trees.
This commit teaches the CTest.UpdateSVN test to verify the presence of
the element.
We teach CTest to report in a <Revision> element the revision of the
source tree that was tested. This makes sense for all modern VCS tools
because they version the whole tree. We simply omit this element for
CVS because it only versions files. See issue #7541.
We teach ADD_TEST_MACRO to transform names of the form "Namespace.Name"
to the directory "Namespace/Name" and the project name "Name". This
will allow new tests to be better organized.
The test overrides the CMAKE_C_FLAGS and CMAKE_Fortran_FLAGS to test
passing a specific flag to the compiler wrapper scripts. We fix it to
honor any outside flags needed for the real compiler.
The commit "FortranCInterface: Honor language flags in checks" taught
the FortranCInterface module to pass C and Fortran flags into its
detection and verification checks. We improve on the change to allow
the '=' character in the language flags. This requires passing the
cache entry type with the -D options.
We create test FortranC.Flags to try passing per-language flags from a
project into its FortranCInterface detect/verify checks. We wrap the
compilers with scripts that enforce presence of expected flags.
CMake does not enable Fortran for its own build, but it needs to find a
Fortran compiler to know if it is possible to enable Fortran tests.
Previously we searched for a hard-coded list of Fortran compilers which
was duplicated from the CMakeDetermineFortranCompiler.cmake module. We
now run CMake on a small test project that enables the Fortran language
and reports the compiler it found. This represents a more realistic
check of whether the Fortran tests will be able to find a compiler.
Previously our EnforceConfig script that loads at test-time would only
enforce a non-empty CTEST_CONFIGURATION_TYPE for CMake 2.6.2 and lower.
Now we simply always enforce use of a configuration, and select one of
the configurations that was built if none is given.
This is necessary to run tests like CMake.Install that need to know the
configuration with which CMake was built.
We create option CMake_TEST_INSTALL to enable a new CMake.Install test.
It tests running the "make install" target to install CMake itself into
a test directory. We enable the option by default for dashboard builds.
We configure an EnforceConfig.cmake script to load at CTest time.
Previously we loaded it from Tests/CTestTestfile.cmake, but now we load
it from the top level so it applies to all tests.
Xcode 2.x forgets to create the target output directory before linking
the individual architecture pieces of a universal binary for the target
CMakeLibTests. Then it passes the directory to -L and -F options when
linking the and warns that the directory does not exist. We work around
the problem by using a pre-build rule on the target to create the output
directory.
We re-arrange EXECUTABLE_OUTPUT_PATH settings to avoid putting utility
and test executables in the 'bin' directory of the build tree. This
makes the directory look like that in the installation tree, except that
on multi-configuration generators we still use a per-config
subdirectory.
The DumpDocumentation executable and some supporting code and tests were
completely unused by CMake. Generation of documentation is done by the
individual executables with --help* options. In this commit we simply
remove the unused code, executable, and test.
In this test we start up a cmake script that runs a process that sleeps,
and the timeout for the script is shorter than the sleep time. However,
in order to properly detect that the sleeping grandchild is killed when
the script times out we need to give sufficient time for the script to
start the grandchild. Otherwise the log file for the grandchild is not
available.
On some (cygwin) builds our previous 1 second timeout for the script was
not long enough to let the interpreter load and start the grandchild.
We make the timeout time configurable by setting CTestTestTimeout_TIME
in the cache for CMake itself. It tells the test how long to let the
script run. The grandchild always sleeps for 4 seconds longer to ensure
a comfortable window during which the process tree can be killed.
This test requires that the dashboard script it drives be invoked with
"ctest -C <config> -S ...". We create a "CTestTest_CONFIG" variable to
hold a configuration selected at test time. We use the configuration
given to the outer CTest, if any, and then default to either Debug or
the CMAKE_BUILD_TYPE.
We extend the CTestTestTimeout test to check that when a test times out
its children (grandchildren of ctest) are killed. Instead of running
the timeout executable directly, we run it through a cmake script that
redirects the timeout executable output to a file. A second test later
runs and verifies that the timeout executable was unable to complete and
write data to the log file. Only if the first inner test times out and
the second inner test passes (log is empty) does the CTestTestTimeout
test pass.
The commit "Fake $HOME to isolate tests from user" started setting $HOME
in the CTest script environment. On some platforms tests depend on some
local configuration in the home directory, such as the "cvs login" for
KWSys in CTestTest3.
In this commit we now construct a fake home dir during CMake config step
and populate it with a .cvspass file needed by the test. We also check
CTEST_NO_TEST_HOME to optionally disable the test home.
See issue #9949.
On platforms with $HOME in the environment, some of our features use it
to store information in the user home directory. However, tests for
these features should not touch the real user home directory. Instead
we configure a fake $HOME that points inside the build tree for use
during testing.
See issue #9949.
The commit "Fix get_filename_component ABSOLUTE mode" broke REALPATH
treatment of relative paths because it stopped storing the absolute path
in local variable 'filename'. This commit fixes the call to GetRealPath
to use the proper local variable and adds a test.
Some compilers use implicit link options of the form
-lcrt*.o
-lgcc*
-lSystem (on Mac)
-lSystemStubs (on Mac)
that provide system-wide symbols not specific to any language.
These need not be listed explicitly for mixed-language linking.
We teach CMake to remove the above items from the implicit library list
of each language. This change makes it possible to mix GNU compiler
versions in some cases.
This allows for a built in bzip and zip capability, so external tools
will not be needed for these packagers. The cmake -E tar xf should be
able to handle all compression types now as well.
The commit "Test per-config OUTPUT_DIRECTORY properties" added this test
with a find_library() call in a CMake script, which requires an explicit
list of possible library prefixes and suffixes. This commit adds more
suffixes to match the libraries built on HP, MinGW, and Cygwin.
We test (ARCHIVE|LIBRARY|RUNTIME)_OUTPUT_DIRECTORY_<CONFIG> properties
by building COnly as a subdirectory and setting the properties to put
its files in specific locations. We build an executable that verifies
the targets actually appear where expected.
The BuildDepends test exercises incremental linking with MSVC and Intel
tools on Windows. In some cases the Intel compiler creates objects that
cause the MS linker it invokes to crash during incremental linking. We
avoid the problem for this test by disabling incremental linking.
We introduce the "CMake.If" test to try out conversion of constants and
variables to boolean values in the if() command. We cover both OLD and
NEW behavior for policy CMP0012.
The commit "Test all target types in Fortran" enabled a SHARED library
in the Fortran test. However, we do not yet implement support for
shared libraries with XL Fortran (it seems this requires using the C
compiler to link). Furthermore, the old g77 2.97 from Red Hat does not
support shared libs on Itanium because the g2c lib is not -fPIC.
For now we just disable SHARED libs in the test for these tools.
We add Intel and MinGW Fortran linker options to create the import
library portion of a DLL. This allows other binaries to link to a
Fortran DLL.
We also update the Fortran test to use a .def file to specify exports
since there is no __declspec(dllexport) markup syntax in Fortran.
To enable this test, the option TEST_KDE4_STABLE_BRANCH must be switched on.
It can only be switched on if CMAKE_RUN_LONG_TESTS is ON.
Then the test will only be added if Qt >= 4.5 can be found, Perl can be
found and ZLIB can be found.
Alex
This commit re-writes Borland compiler build rules. We split the rules
into modern <os>-<id>-<lang> information modules but share a common
macro between languages to avoid duplication.
We also address a bug in the previous rules that would build some target
types against the static Borland runtime and others against the shared
Borland runtime in one build tree. Now we always use the shared runtime
as is the default in the rules for MS tools.
Previously we passed inputs to the decision to each Complex test and let
the test source decide. This commit moves the decision out of the tests
and makes it an option() in their source. This makes it possible to
build the Complex tests from outside the CMake test tree.
When <pkg>_DIR is set to an incorrect version we search again and store
the result in the variable, even if it is <pkg>_DIR-NOTFOUND.
There was a bug in the case when the new search does not find anything
and the old value came from a cache entry with UNINITALIZED type. The
command used to try to load a package configuration file from the last
place searched, and would leave the old wrong value in the entry. This
commit fixes the behavior to avoid trying to load a missing file and to
set the value to <pkg>_DIR-NOTFOUND as expected.
The regex used by CMAKE_PARSE_IMPLICIT_LINK_INFO to detect link lines
should not match lines that happen to have ".../ld.../..." in them. A
linker name should match only as the last component of a path.
See issue #9666.
This commit teaches the CMAKE_PARSE_IMPLICIT_LINK_INFO function to log
its actions. We store the log in CMakeFiles/CMakeOutput.log at the top
of the project build tree. This will make diagnosis of implicit link
information parsing problems easier.
In the Fortran test we use a custom command to build another Fortran
project internally. The project provides a Fortran module and library
to which to link. This commit teaches the test to build the extra
project using the same build configuration as the main project.
We need to leave out the '%' character from tests with the Intel
compiler. Since '%' needs to be written '%%' in NMake when not using a
response file but just '%' when using a response file, we just skip the
character for now. It works with MSVC in NMake only because that
compiler expects '%%' inside response files, which do get used.
CMake defines MSVC only for a VS compiler, but the Intel compiler adds
the preprocessor definition _MSC_VER. Instead of relying on separate
tests to decide whether to use example_dll_2, we do one test in CMake
and then add our own preprocessor definition.
We test this by adding export(TARGETS) to the LinkLanguage test to
export the executable before the library is linked to it. Since
export(TARGETS) computes the link interface of the target (so that it
can export it), this ensures that the information is recomputed after
the link library is added.
This adds a "ModuleDefinition" test enabled when using MSVC tools. It
checks that .def files can be used to export .dll and .exe symbols and
create corresponding .lib files that can be linked. See issue #9613.
Policy CMP0002's OLD behavior allows duplicate non-custom targets. We
test it with a project that builds two executables of the same name by
setting CMP0002 to OLD.
The flag "-_this_is_not_a_flag_" was not rejected by GCC 4.0 on older
Mac OS X. We now use "---_this_is_not_a_flag_" instead, which will
hopefully be rejected by all compilers.
The CMake.File test runs several scripts through "cmake -P" and checks
the output and result against known good values. This commit factors
out the checking code into a separate CMakeCheckTest module. The module
may be used by new tests.
This commit teaches the FunctionTest to check variable scope behavior
when a subdirectory is added inside a function call. Any PARENT_SCOPE
sets in the subdirectory should affect only the function scope which
called add_subdirectory and not its parent scope.
CMake now looks for a Fortran compiler matching any C or C++ compiler
already enabled. We test this by enabling C and C++ first in the
Fortran test, which is what user projects will likely do.
Visual Studio 10 uses MSBuild to drive the build. Custom commands
appear in MSBuild files inside CustomBuild elements, which appear inside
ItemGroup elements. The Outputs and AdditionalInputs elements of each
CustomBuild element are evaluated according to timestamps on disk.
MSBuild does not use inputs/outputs to order CustomBuild steps within a
single ItemGroup or across multiple ItemGroup elements. Instead we must
put only unrelated CustomBuild elements in a single ItemGroup and order
the item groups from top to bottom using a topological order of the
custom command dependency graph.
This fixes CustomCommand and ExternalProject test failures, so we remove
the expectation of these failures.
Part of this test does "git pull" on a dirty work tree. We need to make
sure that 'branch.master.rebase' is false for the test repository.
Otherwise if it is true in the user configuration then pull will refuse
to rebase and the test will fail.
We teach the FindPackageTest to build a sample project that stores its
build tree in the user package registry using export(PACKAGE), and then
find it with find_package.
We test that LINK_INTERFACE_MULTIPLICITY propagates through export() and
install(EXPORT) into dependent projects. A simple cycle of two archives
that need to be scanned three times ensures that the importing project
uses the multiplicity correctly.
This function builds a simple test project using a combination of
Fortran and C (and optionally C++) to verify that the compilers are
compatible. The idea is to help projects report very early to users
that the compilers specified cannot mix languages.
This teaches the 'testing' test to try generator expressions in
arguments to add_test(NAME). This test case mimics a common use-case of
passing executables to test driver scripts. We excercise the syntax for
per-configuration target file names.
This is a new FortranCInterface.cmake module to replace the previous
prototype. All module support files lie in a FortranCInterface
directory next to it.
This module uses a new approach to detect Fortran symbol mangling. We
build a single test project which defines symbols in a Fortran library
(one per object-file) and calls them from a Fortran executable. The
executable links to a C library which defines symbols encoding all known
manglings (one per object-file). The C library falls back to the
Fortran library for symbols it cannot provide. Therefore the executable
will always link, but prefers the C-implemented symbols when they match.
These symbols store string literals of the form INFO:symbol[<name>] so
we can parse them out of the executable.
This module also provides a simpler interface. It always detects the
mangling as soon as it is included. A single macro is provided to
generate mangling macros and optionally pre-mangled symbols.
The try_compile command builds the cmTryCompileExec executable using the
cmTryCompileExec/fast target with Makefile generators in order to save
time since dependencies are not needed. However, in project mode the
command builds an entire source tree that may have dependencies.
Therefore we can use the /fast target approach only in one-source mode.
Previously the Fortran test created a single executable containing C,
C++, and Fortran sources. This commit divides the executable into three
libraries corresponding to each language, and two executables testing
Fortran/C only and Fortran/C/C++ together. The result tests more
combinations of using the languages together, and that language
requirements propagate through linking.
When building an entire source tree with try_compile instead of just a
single source file, it is possible that the CMakeLists.txt file in the
try-compiled project invokes try_compile. This commit fixes propagation
of language-initialization results from the outer-most project into any
number of try-compile levels.
The try_compile command project mode builds an entire source tree
instead of one source file. It uses an existing CMakeLists.txt file in
the given source tree instead of generating one. This commit creates a
test for the mode in the TryCompile test.
This adds sample linker invocation lines for the Intel compiler on
Linux. In particular, this exercises the case when "ld" appears without
a full path.
The Sun Fortran compiler passes -zallextract and -zdefaultextract to the
linker so that all objects from one of its archives are included in the
link. This teaches the implicit options parser to recognize the flags.
We need to pass them explicitly on C++ link lines when Fortran code is
linked.
This extends the Fortran-to-C interface test to add a C++ source file.
The executable can only link with the C++ linker and with the proper
Fortran runtime libraries. These libraries should be detected by CMake
automatically, so this tests verifies the detection functionality.
This hack was created to help the Fortran test executables link to the
implicit C libraries added by BullsEye. Now that implicit libraries
from all languages are detected and included automatically the hack is
no longer needed.
This teaches the SystemInformation test to report the CMake log files
CMakeOutput.log and CMakeError.log from the CMake build tree and from
the SystemInformation test build tree. These logs may help diagnose
dashboard problems remotely.
Kai Wasserbäch
David Cole
Zach Mullen
Brad King
Bill Hoffman
Alexander Neundorf