Replies: 4 comments 3 replies
-
pipeline
titler
GET database
analytical tool in sveltekit
|
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
when we want to process a week of data, push 7 jobs for each day to queue. |
Beta Was this translation helpful? Give feedback.
-
|
This approach of container app job triggered by service bus queue event might be a good solution for this. |
Beta Was this translation helpful? Give feedback.
-
|
Let's close this discussion since it was implemented |
Beta Was this translation helpful? Give feedback.
-
Context
GeoHub RCA is a tool capable of generating assessments of change in human activity from VIIRS DNB nighttime lights imagery. The main source of nightime lights imagery is the Payne Institute for Public Policy's Earth Observation Group(COLEOG) part of Colorado School of mines and R1 reserach university in USA. The Suomi NPP VIIRS DNB imagery collect reflectance in 0.5-0.9 µm bandwidth and features a spatial resolution of 375m at the subsatellite point. The VIIRS DNB scans the Earth globe twice a day and the imagery is processed by the COLEOG into multiple products, respectively, nightly/daily, monthly and yearly mosaics. The mosaics cover the globe between latitudes of 75N and 65S using later on top logic in overlapping areas. At the same time COLEOG produces a cloud mask that can be used to filter out pixels that are covered by clouds.
how are the mosaic really created???? This can be detected by trying to fetch the data on a hourly basis and check how often it is uploaded/overwritten. It is possible the data is updated in close to real time which woud mean the modsaic is growing during the scan cycle. Another option is tghe mosais is created nbext day when all imagery for previous night is already collected.
Core idea
Put in a simplistic manner, the nighttime lights based change detection algorithm for a pixel entails evaluating the sign and magnitude of the difference between the reflectances at two distinct calendar dates. An important aspect is to consider also the cloudiness of the imagery because clouds can affect the precision of computations in a negative way.
Technical implementatiion
The implementation of the tools entails work in three distinct areas: data, backend and frontend.
Data
a processing pipeline is being developed at https://github.com/UNDP-Data/geo-nightlights
The data related functionality consists of a python pipeline deployed as a CRON job with Azure Container Instances or a similar resource. In essence, the pipeline is responsible for ensuring that suitable geospatial data layers are available(COGs) so an algorithm can be develped in cogserver to perform the change detection. As the VIIRS DNB mosaic feature a time axis (daily) it is only natural to organize them in a STAC datalog. For this reason we suggest the nighttime lights dataset should be hosted as a STAC collection inside a folder located in an Azure blob container. The root STAC catalog would then be registered and concequently all the collection could be available in GeoHub
Blob storage account folder
collection.json
graph TD A1(date) --> N(Interpolate) A(NTL file <br>pattern) --> N(Interpolate) N --> B{Azure blob exists?} B --> |No| D(Download) B --> |Yes| E(Fetch file size from EOG) E --> |Compare sizes| F{Sizes are <br>different?} F --> |No| G(Finish) F --> |Yes| D D --> |gdalwarp| I(COG) --> U(Upload) I --> CS{Item exists in STAC?}Backend
With current GeoHub setup the rapid change assessment functionality will be implemented as a titiler algorithm. While the cog endpoint supports creating WMTS from remote COGs it can consume only one url as string. This is a limitation in case a given solution needs two or more urls. The first approach is to build a custom FasAPI endpoint that accepts any number or input urls pointing to COG's and returns a GDAL VRT dataset where evry inout is a band assigned in order or input starting at 1.
a) VRT endpoint + existing titiler endpoint.
In this scenario the existing default cog endpoint cog/tiles/{z}/{x}/{y}/ is is used. This router's endpoints accept a url parameter as a string. As a result it is not possible to pass two datasets through url. There are two solutions here
create a custom endpoint to accept any numbers of datatasets and create on the fly a VRT where every dataset is mapped to a band starting at index 1. The url parameetr would actualy be a call to this endpoint: /api/cog/tiles?url=https://titiler.undpgeohub.org/api/vrt?url={cog1}&url={cog2}&algorithm={xxx}
advantages
1. use existing endpoint
2. we have VRT logic
pass the second dataset as a parameter to the algorithm
vrt spec
b) Custom titiler router whose endpoints accepting multiple instances of url parameters
A second option is to deploy a new COG titler endpoint that accepts a list of urls through url argument. This endpoint needs to implement alignment logic to compute the bounds and possible other properties that will have a direct impact on the computation. The advantage of this aproach is the new endpoint will bring more simplicity and elegance in the overall architecture.
This solution is more elegant and would result in developing a titiler endpoint capable to consume one or more dataset in a standardized way and in combination with the algorithm parameters would enable building complex raster based spatial analytics solutions in a standardized manner.
c) Algorithm
where
cog1 = nighttile lights mosaic at time 0
cog2 = nighttime lights mosaic at time 1
zs = z score transform
mask = cloud mask from colorado
Frontend
Similarly to the backend, the fronted could be implemented in three styles or ways.
This will ensure we have a consistent user interface (same UI enerywhere) and will simplify usage of analytical tools
Beta Was this translation helpful? Give feedback.
All reactions