Production code for Landsat ARD2 bimonthly gapfilling for v1.2

run.py and submit.sh are included, the rest of the workflow is standard.

Time-Series Iteration-free Reconstruction Framework (TSIRF) v1

Release associated to the manuscript:

A computational framework for processing time-series of Earth Observation data based on discrete convolution: global-scale historical Landsat cloud-free aggregates at 30 m spatial resolution
Consoli, Parente, Simoes, Şahin, Tian, Witjes, Sloat, Hengl DOI: https://doi.org/10.21203/rs.3.rs-4465582/v2

Processing large collections of Earth Observation (EO) time-series, often petabyte-sized, such as NASA's Landsat and ESA's Sentinel missions, can be computationally prohibitive and costly. Despite their name, even the Analysis Ready Data (ARD) versions of such collections can rarely be used as direct input for modeling because of cloud presence and/or prohibitive storage size. Existing solutions for readily using these data are not openly available, are poor in performance, or lack flexibility. Addressing this issue, we developed TSIRF (Time-Series Iteration-free Reconstruction Framework), a computational framework that can be used to apply diverse time-series processing tasks, such as temporal aggregation and time-series reconstruction by simply adjusting the convolution kernel. As the first large-scale application, TSIRF was employed to process the entire Global Land Analysis and Discovery (GLAD) ARD Landsat archive, producing a cloud-free bi-monthly aggregated product. This process, covering seven Landsat bands globally from 1997 to 2022, with more than two trillion pixels and for each one a time-series of 156 samples in the aggregated product, required approximately 28 hours of computation using 1248 Intel ® Xeon ® Gold 6248R CPUs. The quality of the result was assessed using a benchmark dataset derived from the aggregated product and comparing different imputation strategies. The resulting reconstructed images can be used as input for machine learning models or to map biophysical indices. To further limit the storage size the produced data was saved as 8-bit Cloud-Optimized GeoTIFFs (COG). With the hosting of about 20 TB per band/index for an entire 30 m resolution bi-monthly historical time-series distributed as open data, the product enables seamless, fast, and affordable access to the Landsat archive for environmental monitoring and analysis applications.

AI4SoilHealth SHDC v1 production code

production_ai4sh_shdc_v1

Add AI4SH production code for properties

Land Degradation Neutrality: SDG 15.3.1 land productivity sub indicator

ldn_land_productivity_sub_indicator

Modify warpTile for different resampling

GPW grassland production code

Derived from pasture_class_prod branch

Production code for Landsat ARD2 bimonthly aggregation in C++

To process different years you need to modify src/aggregation_ard2_production.cpp and compile again. The docker container used for production is 192.168.49.30:5000/pygeo-ide:v3.8.6-mkl-gdal343-ard2-prod and submit with

#!/bin/bash
#SBATCH --partition=cpu               						 # Partition name
#SBATCH --ntasks-per-node=1          						 # Run a single task
#SBATCH --ntasks=1                  						 # Run a single task
#SBATCH --job-name=ard_production						   # Job name
#SBATCH --array=1-10                 						 # Array range
#SBATCH --mail-type=ALL            						 # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=void   # Where to send mailo
#SBATCH --output=%x_%N.log  # Standard output and error log
#SBATCH --exclude=
# Slurm helper functions
source ogh_slurm.sh

# Total number of ids to split and process across the servers
N_IDS=18667

# Python docker container
TILE_FILEPATH="/mnt/slurm/jobs/ard_production/global_tiles_s1.csv"
DOCKER_IMAGE=192.168.49.30:5000/pygeo-ide:v3.8.6-mkl-gdal343-ard2-prod
LANG="C++"
CMD="/mnt/slurm/jobs/ard_production/aggregation_exec"
#CMD="realpath ."

execute

Compilation command example:

g++ -g -I/eigen/Eigen -o aggregation_exec src/aggregation_ard2_production.cpp -lgdal -fopenmp -lstdc++fs -lfftw3 -lfftw3f -fpermissive

Execution:
arg1 = tile_id_start
arg2 = tile_id_end
arg3 = tiles_list
arg4 = hostname
E.g.

./aggregation_exec 0 10 /path/to/tiles_list ripley

Dependencies installation (for Ubundu/Debian distros):

• FFTW:

sudo apt-get install libfftw3-dev libfftw3-doc

• Eigen: sudo apt install libeigen3-dev

cd /usr/include
sudo ln -sf eigen3/Eigen Eigen
sudo ln -sf eigen3/unsupported unsupported

• GDAL

sudo apt install libproj-dev libgeos-dev gdal-bin libgdal-dev postgis

• pybind11

sudo apt-get install python3-pybind11