ASGS Troubleshooting

Troubleshooting By Topic

Below we have debugging and troubleshooting steps below, topics will be added as needed.

Suggested Debugging of an ASGS Installation Step

Based on comments provided directly in #1119.

Given,

prompt> ./init-asgs.sh -b -x "--list-steps"
           setup-env - Updates current environment with variables needed for subsequent steps. It only affects the environment within the asgs-brew.pl environment.
             openmpi - Downloads and builds OpenMPI on all platforms for ASGS. Note: gfortran is required, so any compiler option causes this step to be skipped.
                hdf5 - Downloads and builds the version of HDF5 that has been tested to work on all platforms for ASGS.
             netcdf4 - Downloads and builds the versions of NetCDF and NetCFD-Fortran that have been tested to work on all platforms for ASGS.
              wgrib2 - Downloads and builds wgrib2 on all platforms for ASGS. Note: gfortran is required, so any compiler option passed is overridden.
       cpra-postproc - Runs the makefile and builds associated utilities in the output/cpra_postproc directory
              output - Runs the makefile and builds associated utilities in the output/ directory.
                util - Runs the makefile and builds all associated utilities in the util/ directory.
          input-mesh - Runs the makefile and builds all associated util/input/mesh in the input-mesh/ directory.
     input-nodalattr - Runs the makefile and builds associated utilities in the util/input/nodalattr directory.
                perl - Install local Perl version used for ASGS.
        perl-modules - Install Perl modules used for ASGS.
        image-magick - Install local ImageMagick tools and Perl module Image::Magick.
             python3 - install python 3 locally and install required modules
              ffmpeg - Install ffmpeg and required libraries (nasm)
             gnuplot - Install gnuplot (commandline only)
               units - Install GNU Units utility
                 nco - Install The netCDF Operators (NCO) Toolkit
                pigz - Install pigz, unpigz - parallel gzip
              adcirc - Builds ADCIRC and SWAN if $HOME/adcirc-cg exists.

You can straight up skip breaking steps for debugging purposes. For example,

./init-asgs.sh -x "--skip-steps input-mesh"

You can skip multiple steps, via:

./init-asgs.sh -x "--skip-steps input-mesh,image-magick,..."

This is how I debug. Basically I skip whatever is breaking, then the given command build; then you can run ./asgsh and manually debug the "build" commands present in ./cloud/general/asgs-brew.pl inside of the proper environment.

Also, look at the created ./update-asgs script - it contains the full asgs-brew.pl, which lets you then run it instead of ./init-asgs.sh,

./update-asgs "flags passed to asgs-brew.pl command ..."

An extreme command to immediately get to the shell, with nothing built but the environment set up would be:

prompt> ./init-asgs.sh -b -x "--update-shell"
...
prompt> ./asgsh

Detecting Model Failure

Using NCO (netCDF Operators), specifically ncap2 (netCDF arithmetic processor) and ncks (netCDF kitchen sink) as a concise method of creating diagnostics.

Determine the number of nodes

np=$(ncks --trd -m -M fort.63.nc | grep -E -i ": node, size =" | cut -f 7 -d ' ' | tr -d "," | uniq)

Counting the number of datasets in the file:

ndset=$(ncks --trd -M fort.63.nc | grep -E -i "^Root record dimension 0:" | cut -f 10- -d ' ')

Counting the number of missing values in the first dataset to see if all values are missing:

num_missing=$(ncap2 -A -C -v -s "print(zeta(0:0:1,0:$(($np - 1)):1).number_miss(),\"%d\");" fort.63.nc diagnostics.nc)

Count the number of missing values in each dataset in a file:

for snap in $(seq 0 $(($ndset - 1))) ; do 
   ncap2 -O -C -v -s "print(zeta($snap:$snap:1,0:$(($np - 1)):1).number_miss(),\"%d\");" fort.63.nc diagnostics.nc
done

Count the number of missing values in each dataset in a file and store the number of missing values in an array:

declare -a num_missing
for snap in $(seq 0 $(($ndset - 1))) ; do 
   num_missing+=( $(ncap2 -O -C -v -s "print(zeta($snap:$snap:1,0:$(($np - 1)):1).number_miss(),\"%d\");" fort.63.nc diagnostics.nc) )
done

Determine the max water level within a lat/lon bounding box for one dataset, automatically excluding dry values:

ncap2 -O -v -s "bbox_msk[node]=0.0;where((y >= 20.0 && y <= 30.1) && (x >= -95.0 && x <= -65.1)) bbox_msk=zeta(0:0:1,0:$(($np - 1)):1); elsewhere bbox_msk=zeta@_FillValue; zeta_msk_max=bbox_msk.max(); print(zeta_msk_max,\"%f\"); " fort.63.nc foo.nc