There is data_prepare.py that acts as a first step to compile list of aux files:
- Copernicus 30m COG DEM tile names
- Orbit files (from text file list - poeorb.txt and resorb.txt)
Example call:
data_prepare.py --input-dir /mnt/vol/input_step_a --output-dir /mnt/vol/output
An example result file (stored in --output-dir)
The processing is done by processor.py script.
Example call:
processor.py --input-dir /mnt/vol/input_step_b --output-dir /mnt/vol/output --config-file /mnt/vol/input_step_b/arguments.txt
The arguments.txt file consists either both polarisation and AOI CLI arguments for alus-coh binary. Or simply the
polarisation (--polarisation)
An example directory contents for --input-dir that is sufficient for processing:
arguments.txt
Copernicus_DSM_COG_10_N49_00_E003_00_DEM.tif
Copernicus_DSM_COG_10_N49_00_E004_00_DEM.tif
Copernicus_DSM_COG_10_N49_00_E005_00_DEM.tif
Copernicus_DSM_COG_10_N49_00_E006_00_DEM.tif
Copernicus_DSM_COG_10_N49_00_E007_00_DEM.tif
Copernicus_DSM_COG_10_N50_00_E003_00_DEM.tif
Copernicus_DSM_COG_10_N50_00_E004_00_DEM.tif
Copernicus_DSM_COG_10_N50_00_E005_00_DEM.tif
Copernicus_DSM_COG_10_N50_00_E006_00_DEM.tif
Copernicus_DSM_COG_10_N50_00_E007_00_DEM.tif
Copernicus_DSM_COG_10_N51_00_E003_00_DEM.tif
Copernicus_DSM_COG_10_N51_00_E004_00_DEM.tif
Copernicus_DSM_COG_10_N51_00_E005_00_DEM.tif
Copernicus_DSM_COG_10_N51_00_E006_00_DEM.tif
Copernicus_DSM_COG_10_N51_00_E007_00_DEM.tif
S1A_IW_SLC__1SDV_20210703T055050_20210703T055117_038609_048E45_35F7.SAFE
S1A_OPER_AUX_POEORB_OPOD_20210723T121923_V20210702T225942_20210704T005942.EOF
S1B_IW_SLC__1SDV_20210615T054959_20210615T055026_027363_0344A0_83FE.SAFE
S1B_OPER_AUX_POEORB_OPOD_20210705T111814_V20210614T225942_20210616T005942.EOF
alus-coh also processes scenes when there is no orbital information, hence if no data is available or not needed,
the EOF files could be left unsupplied. If only a single one is available, then it is discarded and both scenese are
processed with the orbital information contained in original SAFE packages.
This repository contains sample processor docker file, docker-compose which uses the processor image and sample k8s pod manifest which uses processor in a preferred way.
$cd example
$ls
data docker-compose.yaml Dockerfile k8s processor.py requirements.txt
$docker-compose up --remove-orphans --force-recreate --build
Building hello-world-processor
(...)
(skip...)
(...)
hello-world-processor_1 | result.data
example_hello-world-processor_1 exited with code 0
This example was tested with docker:
docker version
Client: Docker Engine - Community
Version: 20.10.14
API version: 1.41
Go version: go1.16.15
Git commit: a224086
Built: Thu Mar 24 01:48:02 2022
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.14
API version: 1.41 (minimum version 1.12)
Go version: go1.16.15
Git commit: 87a90dc
Built: Thu Mar 24 01:45:53 2022
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.5.11
GitCommit: 3df54a852345ae127d1fa3092b95168e4a88e2f8
runc:
Version: 1.0.3
GitCommit: v1.0.3-0-gf46b6ba
docker-init:
Version: 0.19.0
GitCommit: de40ad0
and docker compose:
docker-compose --version
docker-compose version 1.27.4, build 40524192
This is a very simple dockerfile which creates an images with a very simple python app. This python app simulates a processor. Take a look at sample's processor help message:
/example$ python processor.py --help
Usage: processor.py [OPTIONS]
Options:
--input PATH [required]
--output PATH [required]
--config-file PATH
--install-completion [bash|zsh|fish|powershell|pwsh]
Install completion for the specified shell.
--show-completion [bash|zsh|fish|powershell|pwsh]
Show completion for the specified shell, to
copy it or customize the installation.
--help Show this message and exit.
All of the places where the processor is writing/reading from have to be configurable. Writing anywhere else than mounted volume will be forbidden. This means that the place where processor can write to must be passed by the command line options. This includes temporary directories, processing results, partial results, etc...
As you can see:
- input path - this is the path where the product is. It has to be configurable.
- output path - this is the directory name where the processor has to put its results
- config path - optional, the system provides a method to pass a config file to a processor
Docker compose serves as an example of processor usage. Docker compose has a volume attachment. This is a folder with an input data. Data tree looks like:
example$ tree data/
data/
└── input_file_location
└── example.data
1 directory, 1 file
There is also a command for the hello-world-processor which starts the processing. Note that input file location and output file location is passed as a command line argument.
To run docker file and build a processor image type
example$ docker-compose up --force-recreate --build
Processor will be built and started. It will write logs to stdout or stderr. It will create a processing result in the output path:
example$ tree data
data
├── input_file_location
│ └── example.data
└── output
└── my_new_product_name
└── result.data
3 directories, 2 files
There is a new directory called "output" and a processing result in it.
Manifest file is a simple Pod and VolumeClaim definition. This is using the same processor image as docker file - called "registry.cloudferro.com/processing/processors/hello-world". Volume is used to provide input data and collect output data. The idea is the same as in case of docker-compose. Resources definition is required therefore processor specification must have requests and limits (https://cloud.google.com/blog/products/containers-kubernetes/kubernetes-best-practices-resource-requests-and-limits). Resources:
resources:
limits:
cpu: 1
memory: 1Gi
requests:
cpu: 1
memory: 1Gi
Hello world processor is a very simple python script which writes a sample output to the directory from the command line options.
python3.10 -m venv venv310
source venv310/bin/activate
pip install -r requirements.txt
Processor internet connectivity is not guaranteed and it should work offline. Auxiliary files (if needed) should be passed separately.