-
Notifications
You must be signed in to change notification settings - Fork 0
Testing
The UVCDAT project is tested nightly by a variety of machines to ensure that the program builds and functions correctly. This page describes the process and documents both the existing tests and the wish lists of features that we need to cover.
UVCDAT's CMake based build system includes a set of regression test targets. These targets, found in the testing subdirectory of the source tree are arbitrary scripts and programs that exercise various features and return status via their exit code. When UVCDAT is configured with the BUILD_TESTING flag turned on, these targets are built. To run them type "ctest" at a prompt from within the build directory.
A set of desktop machines have been configured via cronjobs to download the UVCDAT source code, build, run the tests, and submit the results every night to a CDash server running at kitware. By looking at the results one can easily see if any of the features described below are working properly on any given day.
The dashboard submitters run jobs that run the cmake script located in the UVCDAT source code's CMake/dashboard/dashboard.cmake script like so:
0 5 * * * export PATH=${PATH}:\
/Source/mpi/openmpi-1.4.4-install/bin:\
/Source/Qt/4.8.1/install/bin:\
/Users/demarle/bin:/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin: \
/usr/X11/bin; \
export DASHROOT=/Builds/CDAT; export DASHTYPE=Nightly; export DISPLAY=:0; \
/opt/local/bin/ctest -S /Source/CDAT/improve_build/CMake/dashboard/dashboard.cmake,\
"improve_build;kargad_maclion;krishna.cmake" 2>&1 /Users/demarle/Desktop/cdatcdash.log
Other dashboard submitters choose to run top level scripts which encapsulate most of the above into a shorter cron entry. The important features though are the same. The job must run at a particular time (5 AM above), set up all the environment needed to run uvcdat unattended, and run UVCDAT's dashboard.cmake script via ctest -S and give it a git branch name such as improve_build, a submitting machine name such as kargad_maclion, and a sample customization script name such as krishna.cmake (which you can find in CMake/dashboards/).
The tests can run arbitrary scripts and executables. It is anticipated that most of our tests will make use of UVCDAT's provenance based architecture to play back recorded actions taken in the UVCDAT GUI. To play back a recorded action we run uvcdat in scripted mode via:
uvcdat -b recorded_vistrail_file.vt:chosen_state_tag_name\
-a "datafile=${cdat_SOURCE_DIR}/path_to_datafile.nc\
-e directory_to_place_resulting_imagefile.png
To make a new test:
- Record a visualization: Take some actions. In the vistrails pipeline browser add aliases to any filenames you load so that we don't try to load full paths in your personal file system. In the vistrails history browser add a tag for the particular state that you want to play back. From the file menu save the vistrail.
- Run UVCDAT and load the vistrail and ensure that you can play back the state.
- Run it through the script interface described above as well.
- Add a new call to "add_test" to ${SRC}/testing/uvcdat/CMakeLists.txt, giving it your script and tag name, your aliased datafile name, and a image result filename.
- Run your test by compiling and running "ctest -V".
- Copy the generated test image file from ${BLD}/Testing/Temporary/testname.pngstripped.png to ${SRC}/testing/uvcdat/testname.png .
- Rerun you test and make sure it passes locally.
- Commit the modified CMakeList.txt file, your vistrail, the new baseline image and any new data files required, and push it all to github in the usual way.
- Watch the nightly dashboards to see if the test fails on any other platforms.
- successful build - indicated by 0 errors in the build Errors column of cdash
- run_uvcdat - starts and quits the uvcdat GUI to verify that the built program will run
- cdms_test - uses the VCS mesh fill operator to plot the testnc.nc file to verify we can do something meaningful
- cdms_verify - runs cdms_test and compares the produced image against a known good result to verify that we are doing it correctly
?