Sen2Like @ EODC - Megathread

[LukeWeidenwalker]

Hi - let's use this Github discussion for our back-and-forth on the Sen2like integration at EODC.

Quick updates on the issues identified in our call earlier today:

• The SENTINEL2_L1C and SENTINEL1_GRD collections were indeed missing from the production deployment due to a misconfigured whitelist. We've fixed this and the collections are now available as intended.
• Sending jobs from the aggregator did not work, because the URL hadn't been changed in the aggregator yet. Stefaan has kindly done this in the afternoon, and this is now working as expected.
• The notebook that Valentina showed is here: https://github.com/eodcgmbh/sen2like/blob/master/openeo/openeo-sen2like.ipynb

[LukeWeidenwalker]

Just wanted to note again that stability will be an issue with the new backend until we've had a bit more hands-on usage. Please excuse any inconveniences and let us know about any weirdness you might experience, so that we can iron out these bugs as we go along!

[ValentinaHutter]

We are wondering, if there are specific combinations of options set to False, which we do not want to allow or which might not make sense for the L2H or L2F product? senbox-org#6

[Patrick1G]

@LukeWeidenwalker @ValentinaHutter

I tried the process out just now, and after 50min I get an error.
Below is the OPG. There are long logs in the Editor, but I did not see anything that explains what happened

{
"id": "FA17834AD13A413F",
"process_graph": {
"save2": {
"arguments": {
"data": {
"from_node": "sen1"
},
"format": "GTIFF"
},
"process_id": "save_result",
"result": true
},
"sen1": {
"arguments": {
"spatial_extent": {
"east": 12.413193325661679,
"north": 42.12036818673138,
"south": 41.99847423049263,
"west": 12.271135180368722
},
"temporal_extent": [
"2023-06-01T00:00:00Z",
"2023-06-30T00:00:00Z"
]
},
"process_id": "sen2like"
}
}
}

[LukeWeidenwalker]

Thanks for reporting - I'll look into this later today and report back when I've resolved the issue!

[LukeWeidenwalker]

Hey, here's a quick update on my findings:
The sen2like processor in this job ran through, but save_result back in the openeo execution hangs and times out at some point.
I'm currently investigating why that is and will keep you updated.

[LukeWeidenwalker]

I've spent another afternoon debugging this - we seem to have found the issue. I've now pushed a fix to dev and am rerunning your job there, as soon as that completes successfully, I'll push the fix to prod and let you know!

[LukeWeidenwalker]

@patrick-griffiths I've been able to successfully rerun your job on dev, so I've pushed just the fix to production - please feel free to try for yourself again and let me know if the issue has been resolved! :)

[Patrick1G]

@ValentinaHutter
we discussed today if we should maybe add an optional paramter, that is not part of sen2like SW:
the option to reproject to a target CRS such as UTM. A lot of users probably would like to stay in UTM.
an optional paramter in the sen2like process call would be nice. Alternatively the result of the sen2like process could be reprojected by the user of course..

[LukeWeidenwalker]

Yes, that would be possible, I'll put this on the roadmap!

[LukeWeidenwalker]

Short update on this after a quick investigation:

I initially thought this was an easy feature to add, but didn't consider the full picture. It is indeed quite easy to add a reprojection to an arbitrary CRS, however, in save_result the tiling that's applied is currently always EQUI7. There's currently no way of turning this off. To properly enable this target CRS parameter, we'd also need to address this inflexibility of save_result and let tiling behaviour be configured through /file_formats.

I'll ask for this to be put into the upcoming sprint tomorrow, but just wanted to keep you posted on why this will take more effort than my initial comment suggested.

[Patrick1G]

How about adding some optional convenience functionality for downloading of the resulting sen2like products?
See example below. Currently potentially hundreds of files need to be downloaded, which is not very feasible:
Open-EO/openeo-web-editor#308

A user might simply want all B12/SWIR bands stacked in one file and discard the rest.
This could be added as option for the sen2like process (i dont thing the standard sen2like provides that option, only processing one band), or it could be done with openEO functionality based on the sen2like result cube..

[Selimgit]

Hey @LukeWeidenwalker, thank you for your answer it totally makes sense !

However I have another concern.

In this case I am trying to export a band that should be present.
So why do I get an error ?
The job ID is eodc-3cd31eb3-07dd-480c-b363-8007bdb92c50.

I tried another time specifying GTIFF instead of NetCDF for the output result and I get no error but a hundred of empty tiffs.
The job ID is eodc-9d6756d0-b641-403e-8891-b81e9d633fec.

Thanks for your feedback

[LukeWeidenwalker]

cc @ValentinaHutter

[ValentinaHutter]

We are retrieving the required Landsat data from usgs. In this case, it seems like there was an issue with the usgs m2m-api and we received an HTTPError from usgs. Maybe, you can just retry it. We are working on returning proper logs for this.

[Selimgit]

Thanks ! I relaunched it and it works now. The spatial extents of the images don't really respect the bounding box but it s not a problem to me. I just wanted to know :
Why do I get so many files (102) ? Is there any documentation for that ?

[ValentinaHutter]

We already worked on the code for clipping the images to the bounding box extent, but we only pushed it to our development backend. This will be updated with the next production release. We split our results based on the Equi7 tiles and we normally create one file per datetime. This is probably the reason, why you get 102 files. :)

[Selimgit]

Hello everyone,

I am trying to use this notebook : "https://github.com/eodcgmbh/sen2like/blob/master/openeo/openeo-sen2like.ipynb"
BUT I get this error :

How to fix it ?
Thanks for answering.

[bschumac]

@Selimgit can you please provide me with the e-mail address you are using for your account. I checked the EGI database and it seems that you don't have an account in the virutal organization. You can also use the following link to create a petition which I will be approving: https://aai.egi.eu/registry/co_petitions/start/coef:327

[ValentinaHutter]

We also tried to reproduce the error now, but for both me and my colleague @GeraldIr the code works. As suggested by Benjamin, this could be related to your user account. Also, as the error comes up in the openeo python client (which is not related to the eodc backend), you could check which version of openeo in python you are using and if a new update is already available. An update could also possibly resolve the issue.

[Selimgit]

Hey @bschumac. The email I use is selim.behloul@outlook.com.

I upgraded my openeo version from 0.22 to 0.23 @ValentinaHutter but I still get the error

[bschumac]

@Selimgit you have a 30-day trial account that is expired. Please consider to request a NoR Sponsoring: https://openeo.cloud/#plans

[Patrick1G]

@valentina @LukeWeidenwalker
could you please look into this issue reported by Selim?
many thanks!

[Patrick1G]

@ValentinaHutter @LukeWeidenwalker
one more observation:
when processing a time seriers, the result consists of potentially hundreds of GeoTiff files, each currently has to be clicked individually. Could this be simplified ? Some download utility functionality? I think in the main openEO editor this is already possible...

[ValentinaHutter]

I am not sure, where you would do this in the main editor, as this screenshot looks like it was taken from the main editor - maybe you can check again :)
But I know that it is possible to download all files at once using python, the code to do this would look like this:

import openeo
backend = "https://openeo.cloud" # this could be a different backend url as well
conn = openeo.connect(backend) 
conn = conn.authenticate_oidc() # follow steps to authenticate here

job = conn.job( <JOB ID> )
results = job.get_results()
metadata = results.get_metadata()
results.download_files("./data/")

using .download_files() should start downloading all the available files then :)

[ValentinaHutter]

Downloading multiple files is a front end feature, which is available in the openeo python client as shown above.
This is not supported in the webeditor, yet, but we can bring this up in our next devtelco as a feature request.
This feature request is not directly related to sen2like, as having a large number of files can happen easily for any openeo job that includes a time series. I will therefore create a general issue.

[ValentinaHutter]

https://github.com/openEOPlatform/architecture-docs/issues/373

[Patrick1G]

@ValentinaHutter yes please bring this up in the dev telco..
exposing such functionality in the editor would be valuable.
please keep us posted

[Patrick1G]

added this comment here:
Open-EO/openeo-web-editor#268 (comment)

[Patrick1G]

@LukeWeidenwalker @ValentinaHutter
currently, when defining a sen2like process in the web editor, the sen2like parameters do not show (see below) , but there should be a predefined list

For example here, the geoTiff output options:

[christophreimer]

@ValentinaHutter please link related issue from sen2like repo

[ValentinaHutter]

senbox-org#6

we are working on that!

[m-mohr]

Looks like an incomplete schema. The Web Editor should pick up the schema once it's there and complete and offer a list of checkbox options.

[Patrick1G]

@LukeWeidenwalker @ValentinaHutter
Another issue is that for failed jobs the logs are not available:

For the example below the ID: eodc-793b8850-2860-4535-84af-ae30c3dfd0ba

[bschumac]

@patrick-griffiths can you send the process graph?

[Patrick1G]

Here it is:
sen2like_opg.txt

[ValentinaHutter]

We prepared a bug fix for this, which we are currently testing. We will let you know, once this is resolved.

[Patrick1G]

@ValentinaHutter @LukeWeidenwalker
I think some convenience functionality that prevents having to download 100s of files would be very important. Could we add optional parameters like 'stack_by_band' that creates Band specific geotiffs with one band per date?

[Patrick1G]

another issue:

if the sen2like process is integrated into a openEO workflow, i.e., the image collection is defined prior to calling sen2like, then the data cube parameters are currently not passed on to the sen2like process

[Patrick1G]

@LukeWeidenwalker @ValentinaHutter

when downloading the files for the job below, all GeoTiff files only have 7KB and do not open,
this is true for the Editor and the JupyterLab environment...

PS: NetCDF download works!
PPS: if no bands are specified as input, then bands 02, 03, 04 are transferred into the results NetCDF...

ID: eodc-e2b31155-dca5-424b-b5bf-9e0f2c2d8e07
Submitted: 10/13/2023, 1:09:11 PM UTC
Status: finished

[Patrick1G]

@LukeWeidenwalker @ValentinaHutter @christophreimer

we need to support users that do not want a Equi7 projected result files...This could be done as an optional parameter in the sen2like process.

Alternatively this could be done by the user using the resample_spatial() process.
We tried this right now and this fails.. Could you please look into this?

In general users should be able to work with the outputs of the sen2like process in larger openEO applications,
e.g. running a reducer on the outputs of the sen2like results. Could you please check and ensure that this is possible?

ID: eodc-dd7580ee-40e8-442c-93fe-9839414a31f2
Submitted: 10/17/2023, 12:39:51 PM UTC
Status: error

[Selimgit]

Hi everyone,

I wonder why my Sen2like process is correctly running for a temporal extent corresponding to June 2023 but not for January 2023.
(same spatial extent)

June : eodc-549e399d-cb20-4e2e-be42-7b0c870bb53c
January : eodc-c5ec50db-3c2e-4e32-8e57-b4a759306c0e

[Patrick1G]

another thing I noticed, @christophreimer @ValentinaHutter
is that EODC jobs, as seen in the Editor, conclude with a random job name, ignoring the job name that is given by the user

see below, example, the running job (EODC backend) has a user provided name, but will be renamed to a random name similar to #2 when the process competes...

[Patrick1G]

For NetCDF outputs, there is an option in the Editor that can be set to True to produce one netCDF per band, over time (see screenshot below). I just completed to the below process with that option selected, but the netCDFs are one per date, not as expected:
eodc-b0e496d7-189c-4b63-82bb-1c728daa45f1
This feature would be very valuable for the sen2like process...