All Data Processing Tasks Stable

all existing scripts are stable including sciclone jobs for

• ndvi mosaic
• ndvi extract
• historic ndvi prep
• historic ndvi extract
• ndvi merge
• atap extract
• atap merge

...as well as various utility scripts

paths in scripts reflect semi-formal file structure that will be used for projects on sciclone

Stable Historic Prep

Easy. Runs in under a minute with a single c11 node using all 8 cores with 8gb memory per GDAL process.

Stable Mosaic

Stable version of mosaic job scripts.

Processed 8.5 years of data (2007-2015*) on 2 c11a nodes (16 cores) and successfully completed micro tests (sub-year data sets) without errors.

Error during initial full job (2001-2015*, 4 c11a nodes with 32 total cores) occurred roughly at the 2006-233 dataset causing job and multi-node failure. Crash may have been due to system issues occurring after a maintenance outage period. No errors have occurred since but no 4 node (32 core) jobs have been run since.

*2015 is a partial year dataset

Initial pre processing

Initial run with preprocessing individual frames before creating mosaic.

May cause I/O issues. Has unnecessary writes of unzipped data back to main filesystem.

Use for testing purposes only.

Mosaic post processing

For testing only - may cause severe I/O issues.

Builds mosaic from raw frames then processes the complete mosaic.

Testing purposes:

• Determine GDAL behavior when working with large rasters
• Raster size GDAL classifies as "large" and processes differently

Theory:

• GDAL is creating "empty" large rasters for each frame to use for processing then merges for final product