This a repository of the application I did to the job of "Data Analyst" at SpaceAg.co. The purpose of the assignment is to unlock the image data from a Sentinel Satellite and to be able to depict the behavior of a plant of Avocado in Ica, Peru. The idea is to be able to analyze yield in terms of an indicator called NDVI derived from satellite imagery.
The repository contains the code used to process the data and to visualize main findings in terms of space and time approaches. In total there are 33 plots measured along 89 days from mid 2016 to early 2020. The preprocessing step contain a bunch of shell scripts that used the Unix tools in order to create tabular grids which represent each plot per day as a unit of analysis. The visualization step contain a bunch of Python scripts that use geopandas to change the tabular structures into several cubes of data and to create visualizations for the exploratory analysis.
Each data processing step were executed by a set of scripts following an ETL like approach. Each step takes files from a folder and creates new files in another folder and so on. As many steps contain more than 3000 files, the files were replaced for one text file representing the file lists in each step/folder.
Challenge description is in the following link:
https://bitbucket.org/adoval4/space-ag-data-analyst-challenge/src/master/
Are located in the following folders along with how many of them contains each.
- images_ndvi : 408 files.
- rename_ndvi : 204 files.
- trunc_n001_ndvi : 89 files.
- mask_n001_ndvi : 89 files.
- parce_vector : 33 files.
- raster_n001_ndvi : 2937 files.
- info_n001_ndvi : 2937 files.
- csv_n001_ndvi : 2937 files.
- stats_n001_ndvi : 2937 files.
The following workflow represents the pipeline created to process and visualize data.
- Descargar: In this step some Jupyter notebooks were used to setup SentinelHub API configurations and to test some initial queries.
- Editar: This step contains in one side renaming of the downloaded images, because they are downloaded with a filenames that do not represent the original date. In the other side, as we generated copies of the original images, the resulting ones were edited so as to generate statistics immediately and to add compression to them.
- Truncar: It consists of making every pixel below a NDVI cut-off of -0.01 to be No Data Value. The purpose is that those pixels should not be included in any subsequent calculation.
- Intersectar: This step is required for the images to take only the margins that correspond exactly to the original shapefile provided by spaceag team.
- Dividir: This step consists in mixing each 89 image with 1 shapefile which has been also recoded so to represent correctly each plot within the orchard.
- Generar Info: This step is useful because allows to check if all images have sufficient sample pixels to calculate common statistics. Also because querying text files is faster than querying each image with the GeoTiff driver.
- Exportar Csv: Csv files were created from GeoTiff images so as to calculate fast common statistics to it.
- Calcular Metricas: This step contain common statistics but also median which is used, because of the NDVI cut-off chosen for this analysis which is under the 0.2 value. So as to not be very biased by lower values of NDVI.
- Extra steps: Rest consisted in aggregating dataframes up to a desired time or spatial resolution to create a visualization for analysis. Python scripts were used here.
Either tabular or dataframe (cube) data were located under the csv_datos folder. Tabular data were originally in the main path.
The resulting document are below the respuesta folder.
- The main script needs to be further edited to reflect a more clean command line pipeline. Current one is Procesos_temp.sh which contains the pipeline, but also contains many temporary commands other than the pipeline itself.
- The path of original input and shapefiles were relocated under the insumos folder, so it will be necessary to edit their paths in the corresponding scripts in order to process everything with new NDVI cut-offs and so on. Or simply to place everything again in the main path in which they were called originally.
- Adapt this pipeline to be used in an environment such as Galaxy.