Streamlined LW calculations, OpenMP GPU support, small efficiency cha…

…nges, updates to continuous integration (#110)

OpenMP GPU offload, tested with  CCE 11.0.0. Longwave Jacobians aren't computed if the optional arguments aren't provided. Finalization methods for optical properties arrays are introduced. Make is streamlined to use environment variables and omit config files. Small changes for efficiency and robustness. Continuous integration is evolved; includes gfortran 9 and 10 on Github actions; current Nvidia and Intel HPC compilers in containers; Cray CPU and OpenMP GPU and PGI 20 OpenACC. 

Co-authored-by: Nichols Romero <naromero@anl.gov> 
Co-authored-by: Dustin Swales <Dustin.Swales@noaa.gov> 
Co-authored-by: Peter Ukkonen <peterukk@gmail.com>
Co-authored-by: Chiel van Heerwaarden <chiel.vanheerwaarden@wur.nl>

.github/workflows/containerized-ci.yml

@@ -0,0 +1,77 @@
+name: Continuous integration in a box
+on: [push, pull_request]
+
+jobs:
+  Containerized-CI:
+    runs-on: ubuntu-20.04
+    strategy:
+      matrix:
+        rte-kernels: [default, openacc]
+        container: ["earthsystemradiation/rte-rrtmgp-ci:ifort","earthsystemradiation/rte-rrtmgp-ci:nvfortran"]
+    container:
+      image: ${{ matrix.container }}
+    env:
+      NCHOME: /home/runner/netcdf-c
+      NFHOME: /home/runner/netcdf-fortran
+      RFMIP_DIR: /home/runner/rfmip-files
+    steps:
+    ############################################################################
+    # Checks out repository under $GITHUB_WORKSPACE
+    - name: Check out code
+      uses: actions/checkout@v2
+    - name: Environmental variables
+      # This might be able to be set in the ENV section above
+      run: echo "RRTMGP_ROOT=${GITHUB_WORKSPACE}"        >> $GITHUB_ENV
+    - name: Environmental variables - ifort
+      if: contains(matrix.container, 'ifort')
+      run: echo "FCFLAGS=-m64 -g  -traceback -heap-arrays -assume realloc_lhs -extend-source 132 -check bounds,uninit,pointers,stack -stand f08" >> $GITHUB_ENV
+    - name: Environmental variables - nvfortran
+      if: contains(matrix.container, 'nvfortran')
+      run: echo "FCFLAGS=-Mallocatable=03 -Mstandard -Mbounds -Mchkptr -Kieee -Mchkstk" >> $GITHUB_ENV
+
+    - name: Make library, examples, tests
+      shell: bash
+      env:
+        RTE_KERNELS: ${{ matrix.rte-kernels }}
+      run: |
+        source /opt/intel/oneapi/setvars.sh || true
+        cd ${RRTMGP_ROOT}
+        ${FC} --version
+        make libs
+    ############################################################################
+    - name: Cache RFMIP files
+      id: cache-rfmip-files
+      uses: actions/cache@v2
+      with:
+        path: /home/runner/rfmip-files # Same as #{RFMIP_DIR}
+        key: rfmip-files
+
+    - name: Stage RFMIP files
+      if: steps.cache-rfmip-files.outputs.cache-hit != 'true'
+      run: |
+        mkdir -p ${RFMIP_DIR}
+        cd ${RFMIP_DIR}
+        python ${RRTMGP_ROOT}/examples/rfmip-clear-sky/stage_files.py
+    ############################################################################
+    - name: Run examples, tests
+      shell: bash
+      env:
+        LD_LIBRARY_PATH: /home/runner/netcdf-c/lib
+      run: |
+        source /opt/intel/oneapi/setvars.sh || true
+        export LD_LIBRARY_PATH=${NFHOME}/lib:${LD_LIBRARY_PATH}
+        make tests
+    - name: Comparison
+      run: make check
+    ############################################################################
+    - name: Validation plots
+      if: contains(matrix.container, 'ifort') && contains(matrix.rte-kernels, 'default')
+      run: |
+        cd ${RRTMGP_ROOT}/tests
+        python validation-plots.py
+    - name: Upload plots
+      if: contains(matrix.container, 'ifort') && contains(matrix.rte-kernels, 'default')
+      uses: actions/upload-artifact@v2
+      with:
+        name: valdiation-plot
+        path: tests/validation-figures.pdf

.github/workflows/continuous-integration.yml

@@ -0,0 +1,142 @@
+name: Continuous Integration
+on: [push, pull_request]
+
+jobs:
+  CI:
+    runs-on: ubuntu-20.04
+    strategy:
+      matrix:
+        fortran-compiler:  [gfortran-9, gfortran-10]
+        rte-kernels: [default, openacc]
+    env:
+      FC: ${{ matrix.fortran-compiler }}
+      FCFLAGS: "-ffree-line-length-none -m64 -std=f2008 -march=native -fbounds-check -finit-real=nan -g -DUSE_CBOOL"
+      CC: gcc
+      NCHOME: /home/runner/netcdf-c
+      NFHOME: /home/runner/netcdf-fortran
+      RFMIP_DIR: /home/runner/rfmip-files
+    steps:
+    - name: Update system packages
+      run: sudo apt-get update
+    ############################################################################
+    #
+    # Compilers....
+    #
+    # Gfortran 10 not available in Github CI stack, so install
+    #
+    - name: gfortran-10 setup compiler
+      if: contains(matrix.fortran-compiler, 'gfortran-10')
+      run: |
+        sudo apt-get install gfortran-10 gcc-10
+        echo "CC=gcc-10" >> $GITHUB_ENV
+
+    ############################################################################
+    #
+    # Netcdf C and Fortran
+    #
+    - name: Install HDF5 library
+      run: |
+        sudo apt-get install libhdf5-dev libcurl4-gnutls-dev hdf5-helpers
+        dpkg -L libhdf5-dev
+
+    # Skipping this for now - netCDF configure doesn't see the HDF libararies
+    - name: cache-netcdf-c
+      id: cache-netcdf-c
+      uses: actions/cache@v2
+      with:
+        path: /home/runner/netcdf-c
+        key: netcdf-c-4.7.4a-${{ runner.os }}-${{ matrix.fortran-compiler }}
+
+    - name: Install netcdf C library from source
+      if: steps.cache-netcdf-c.outputs.cache-hit != 'true'
+      env:
+        CPPFLAGS: -I/usr/include/hdf5/serial
+        LDFLAGS: -L/usr/lib/x86_64-linux-gnu/hdf5/serial/
+      run: |
+        ${CC} --version
+        git clone https://github.com/Unidata/netcdf-c.git --branch v4.7.4
+        cd netcdf-c
+        ls /usr/include
+        ./configure --prefix=${NCHOME}
+        make -j
+        sudo make install
+
+    # Would be great to encode version info
+    - name: cache-netcdf-fortran
+      id: cache-netcdf-fortran
+      uses: actions/cache@v2
+      with:
+        path: /home/runner/netcdf-fortran
+        key: netcdf-fortran-4.5.3-${{ runner.os }}-${{ matrix.fortran-compiler }}
+
+    - name: Build NetCDF Fortran library
+      # Here too it would be nice to use the environment to specify netcdf-c location
+      env:
+        CPPFLAGS: -I/home/runner/netcdf-c/include
+        LDFLAGS: -L/home/runner/netcdf-c/lib
+        LD_LIBRARY_PATH: /home/runner/netcdf-c/lib
+        FCFLAGS: -fPIC
+      if: steps.cache-netcdf-fortran.outputs.cache-hit != 'true'
+      run: |
+        echo ${TEST}
+        ${FC} --version
+        git clone https://github.com/Unidata/netcdf-fortran.git --branch v4.5.3
+        cd netcdf-fortran
+        echo ${CPPFLAGS}
+        ./configure --prefix=${NFHOME}
+        make -j
+        sudo make install
+    ############################################################################
+    # Checks out repository under $GITHUB_WORKSPACE
+    - name: Check out code
+      uses: actions/checkout@v2
+
+    - name: Environmental variables
+      run: echo "RRTMGP_ROOT=${GITHUB_WORKSPACE}"        >> $GITHUB_ENV
+
+    - name: Make library, examples, tests
+      env:
+        RTE_KERNELS: ${{ matrix.rte-kernels }}
+      run: |
+        cd ${RRTMGP_ROOT}
+        ${FC} --version
+        make libs
+
+    ############################################################################
+    # Set up Python and packages
+    #
+    - name: Set up Python
+      uses: actions/setup-python@v2
+      with:
+        python-version: 3.7
+    - name: Setup conda
+      uses: s-weigand/setup-conda@v1
+      with:
+        python-version: 3.7
+    - name: Install python packages
+      run: conda install --yes urllib3 netcdf4 xarray dask scipy matplotlib seaborn colorcet
+    ############################################################################
+    - name: Cache RFMIP files
+      id: cache-rfmip-files
+      uses: actions/cache@v2
+      with:
+        path: /home/runner/rfmip-files # Same as #{RFMIP_DIR}
+        key: rfmip-files
+
+    - name: Stage RFMIP files
+      if: steps.cache-rfmip-files.outputs.cache-hit != 'true'
+      run: |
+        mkdir -p ${RFMIP_DIR}
+        cd ${RFMIP_DIR}
+        python ${RRTMGP_ROOT}/examples/rfmip-clear-sky/stage_files.py
+    ############################################################################
+    # Would be great to encode version info
+    - name: Run examples, tests
+      env:
+        LD_LIBRARY_PATH: /home/runner/netcdf-c/lib
+      run: |
+        export LD_LIBRARY_PATH=${NFHOME}/lib:${LD_LIBRARY_PATH}
+        make tests
+    - name: Comparison
+      run: |
+        make check

Compiler-flags.md

@@ -0,0 +1,30 @@
+# Compiler flag Examples
+
+Before using the Makefiles supplied with the `RTE+RRTMGP` repository, the environment variables `FC` and
+`FCFLAGS`, identifying the Fortran compiler and flags passed to it, need to be set. Here are some examples
+used during development and testing.
+
+To build any of the executables in `examples/` or `tests` the locations of the C and Fortran netCDF libraries
+need to be set via environment variables `NCHOME` and `NFHOME`, and the variable `RRTMGP_ROOT` must be set to the
+root of the RTE+RRTMGP installation.
+
+## Gnu Fortran
+`FC: gfortran-8` or `gfortran-9` or `gfortran-10`
+### Debugging flags
+`FCFLAGS: "-ffree-line-length-none -m64 -std=f2008 -march=native -fbounds-check -finit-real=nan -DUSE_CBOOL"`  
+### Even stricter debugging flags
+`FCFLAGS: "-ffree-line-length-none -m64 -std=f2008 -march=native -fbounds-check -fbacktrace -finit-real=nan -DUSE_CBOOL -pedantic -g -Wall"`  
+
+## Intel Fortran
+`FC: ifort`  
+### Debugging flags
+`FCFLAGS: "-m64 -g -traceback -heap-arrays -assume realloc_lhs -extend-source 132 -check bounds,uninit,pointers,stack -stand f08"`  
+### Optimization flags:  
+`FCFLAGS:"-m64 -O3 -g -traceback -heap-arrays -assume realloc_lhs -extend-source 132"`
+
+## PGI Fortran
+`FC: pgfortran` or `FC: nvfortran` (if using the Nvidia HPC SDK)
+### Debugging flags
+`FCFLAGS: "-g -Minfo -Mbounds -Mchkptr -Mstandard -Kieee -Mchkstk -Mallocatable=03  -Mpreprocess"`
+### Optimization flags:  
+`FCFLAGS: "-m64 -O3 -g -traceback -heap-arrays -assume realloc_lhs -extend-source 132"`

Makefile

@@ -0,0 +1,27 @@
+#
+# Top-level Makefile
+#
+.PHONY: libs tests check
+all: libs tests check
+
+libs:
+	make -C build -j
+	make -C tests -j 1
+	make -C examples/all-sky -j
+	make -C examples/rfmip-clear-sky -j
+
+tests:
+	make -C examples/rfmip-clear-sky tests
+	make -C examples/all-sky         tests
+	make -C tests                    tests
+
+check:
+	make -C examples/rfmip-clear-sky check
+	make -C examples/all-sky         check
+	make -C tests                    check
+
+clean:
+	make -C build clean
+	make -C examples/rfmip-clear-sky clean
+	make -C examples/all-sky         clean
+	make -C tests                    clean