ASGS Troubleshooting

Troubleshooting By Topic

Below we have debugging and troubleshooting steps below, topics will be added as needed.

Suggested Debugging of an ASGS Installation Step

Based on comments provided directly in #1119.

Given,

prompt> ./init-asgs.sh -b -x "--list-steps"
           setup-env - Updates current environment with variables needed for subsequent steps. It only affects the environment within the asgs-brew.pl environment.
             openmpi - Downloads and builds OpenMPI on all platforms for ASGS. Note: gfortran is required, so any compiler option causes this step to be skipped.
                hdf5 - Downloads and builds the version of HDF5 that has been tested to work on all platforms for ASGS.
             netcdf4 - Downloads and builds the versions of NetCDF and NetCFD-Fortran that have been tested to work on all platforms for ASGS.
              wgrib2 - Downloads and builds wgrib2 on all platforms for ASGS. Note: gfortran is required, so any compiler option passed is overridden.
       cpra-postproc - Runs the makefile and builds associated utilities in the output/cpra_postproc directory
              output - Runs the makefile and builds associated utilities in the output/ directory.
                util - Runs the makefile and builds all associated utilities in the util/ directory.
          input-mesh - Runs the makefile and builds all associated util/input/mesh in the input-mesh/ directory.
     input-nodalattr - Runs the makefile and builds associated utilities in the util/input/nodalattr directory.
                perl - Install local Perl version used for ASGS.
        perl-modules - Install Perl modules used for ASGS.
        image-magick - Install local ImageMagick tools and Perl module Image::Magick.
             python3 - install python 3 locally and install required modules
              ffmpeg - Install ffmpeg and required libraries (nasm)
             gnuplot - Install gnuplot (commandline only)
               units - Install GNU Units utility
                 nco - Install The netCDF Operators (NCO) Toolkit
                pigz - Install pigz, unpigz - parallel gzip
              adcirc - Builds ADCIRC and SWAN if $HOME/adcirc-cg exists.

You can straight up skip breaking steps for debugging purposes. For example,

./init-asgs.sh -x "--skip-steps input-mesh"

You can skip multiple steps, via:

./init-asgs.sh -x "--skip-steps input-mesh,image-magick,..."

This is how I debug. Basically I skip whatever is breaking, then the given command build; then you can run ./asgsh and manually debug the "build" commands present in ./cloud/general/asgs-brew.pl inside of the proper environment.

Also, look at the created ./update-asgs script - it contains the full asgs-brew.pl, which lets you then run it instead of ./init-asgs.sh,

./update-asgs "flags passed to asgs-brew.pl command ..."

An extreme command to immediately get to the shell, with nothing built but the environment set up would be:

prompt> ./init-asgs.sh -b -x "--update-shell"
...
prompt> ./asgsh

Detecting Model Failure

Using NCO (netCDF Operators), specifically ncap2 (netCDF arithmetic processor) and ncks (netCDF kitchen sink) as a concise method of creating diagnostics.

Determine the number of nodes

np=$(ncks --trd -m -M fort.63.nc | grep -E -i ": node, size =" | cut -f 7 -d ' ' | tr -d "," | uniq)

Count the number of nodes that would be considered dry on cold start based on topobathy depth:

ncap2 -O -v -s "land_msk[node]=0.0;land_msk.set_miss(-99999.0);where(depth < 0.0) land_msk=-99999.0; elsewhere land_msk=depth; print(land_msk.number_miss(),\"%d\n\");" fort.63.nc foo.nc

Counting the number of datasets in the file:

ndset=$(ncks --trd -M fort.63.nc | grep -E -i "^Root record dimension 0:" | cut -f 10- -d ' ')

Counting the number of missing values in the first dataset to see if all values are missing:

num_missing=$(ncap2 -O -C -v -s "print(zeta(0:0:1,0:$(($np - 1)):1).number_miss(),\"%d\");" fort.63.nc diagnostics.nc)

Count the number of missing values in each dataset in a file:

for snap in $(seq 0 $(($ndset - 1))) ; do 
   ncap2 -O -C -v -s "print(zeta($snap:$snap:1,0:$(($np - 1)):1).number_miss(),\"%d\");" fort.63.nc diagnostics.nc
done

Count the number of missing values in each dataset in a file and store the number of missing values in an array:

declare -a num_missing
for snap in $(seq 0 $(($ndset - 1))) ; do 
   num_missing+=( $(ncap2 -O -C -v -s "print(zeta($snap:$snap:1,0:$(($np - 1)):1).number_miss(),\"%d\");" fort.63.nc diagnostics.nc) )
done

Determine the max water level within a lat/lon bounding box for one dataset:

ncap2 -O -v -s "bbox_msk[node]=0.0;where((y >= 20.0 && y <= 30.1) && (x >= -95.0 && x <= -65.1)) bbox_msk=zeta(0:0:1,0:$(($np - 1)):1); elsewhere bbox_msk=zeta@_FillValue; zeta_msk_max=bbox_msk.max(); print(zeta_msk_max,\"%f\"); " fort.63.nc foo.nc

Compute the min, max, average, and standard deviation of water level within a lat/lon bounding box excluding the -99999.0 _FillValue along with the number of dry nodes for one dataset :

ncap2 -O -v -s "bbox_msk[node]=0.0;bbox_msk.set_miss(-99999.0);where((y >= 20.0 && y <= 30.1) && (x >= -95.0 && x <= -65.1)) bbox_msk=zeta(0:0:1,0:$(($np - 1)):1); elsewhere bbox_msk=zeta@_FillValue; print(bbox_msk.max());print(bbox_msk.min());print(bbox_msk.avg());bbox_msk_stdev=(bbox_msk-bbox_msk.avg()).rmssdn();print(bbox_msk_stdev);print(bbox_msk.number_miss());" fort.63.nc foo.nc

Compute the min, max, average, and standard deviation of water level within a lat/lon bounding box excluding the -99999.0 _FillValue, along with the number of dry nodes, for one dataset:

for snap in $(seq 0 $(($ndset - 1))) ; do 
   ncap2 -O -v -s "bbox_msk[node]=0.0;bbox_msk.set_miss(-99999.0);where((y >= 20.0 && y <= 30.1) && (x >= -95.0 && x <= -65.1)) bbox_msk=zeta($snap:$snap:1,0:$(($np - 1)):1); elsewhere bbox_msk=zeta@_FillValue; print(bbox_msk.min(),\"%f \");print(bbox_msk.max(),\"%f \");print(bbox_msk.avg(),\"%f \");print((bbox_msk-bbox_msk.avg()).rmssdn(),\"%f \");print(bbox_msk.number_miss(),\"%d\n\");" fort.63.nc foo.nc
done

sample output from the above:

-0.968176 1.927551 0.473537 0.137439 912724
-0.968176 1.927551 0.503579 0.136248 907240
-0.968176 1.927551 0.528694 0.134726 901804
-0.968176 1.927551 0.548272 0.132958 897026
-0.968176 1.927551 0.561440 0.131201 893015
-0.968176 1.927551 0.568402 0.130402 889561
-0.968176 1.927551 0.569775 0.130022 887166
-0.968176 1.927551 0.565483 0.129697 885613
...

Compute the min, max, average, and standard deviation of water level excluding the -99999.0 _FillValue along with the number of dry nodes over the full domain for one dataset:

for snap in $(seq 0 $(($ndset - 1))) ; do 
   ncap2 -O -v -s "data[node]=0.0;data.set_miss(-99999.0);data=zeta($snap:$snap:1,0:$(($np - 1)):1);print(data.min(),\"%f \");print(data.max(),\"%f \");print(data.avg(),\"%f \");print((data-data.avg()).rmssdn(),\"%f \");print(data.number_miss(),\"%d\n\");" fort.63.nc foo.nc
done

Compute the min, max, average, and standard deviation of water level excluding the -99999.0 _FillValue along with the number of dry nodes over the full domain for all datasets, writing the results to a file called statistics_${file}.txt:

file=fort.63.nc
if [[ -e statistics_${file}.txt ]]; then 
   rm statistics_${file}.txt
fi
for snap in $(seq 0 $(($ndset - 1))) ; do 
   ncap2 -O -v -s "data[node]=0.0;data.set_miss(-99999.0);data=zeta($snap:$snap:1,0:$(($np - 1)):1);print(data.min(),\"%f \");print(data.max(),\"%f \");print(data.avg(),\"%f \");print((data-data.avg()).rmssdn(),\"%f \");print(data.number_miss(),\"%d\n\");" $file foo.nc >> statistics_${file}.txt
done