Skip to content

Update template and schema #5

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pont-us merged 14 commits into from
Feb 16, 2023
Merged

Update template and schema #5

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
3471272
Add a per-variable processing_steps to the schema
pont-us Nov 3, 2022
bd16fd2
Merge branch 'main' into pont-4-update-schema
pont-us Feb 14, 2023
0f7dbc9
Update template to match current specification
pont-us Feb 15, 2023
1e9d60c
Fix some typos in the specification template
pont-us Feb 15, 2023
4a9838f
Update specification schema to match current spec
pont-us Feb 15, 2023
5c4b002
Remove per-variable "sources" list
pont-us Feb 15, 2023
c36dd8e
Schema and template: move colour props to metadata
pont-us Feb 15, 2023
ddba884
Schema: remove processing_steps from variable
pont-us Feb 15, 2023
aab7404
Improve schema for data sources
pont-us Feb 15, 2023
c5607ff
Schema: require minimum spatial_size of 2
pont-us Feb 16, 2023
c597dce
Spec and schema: update some property names
pont-us Feb 16, 2023
29cf2dc
Spec and schema: make units optional
pont-us Feb 16, 2023
72adf52
Schema: match exactly one of the two source types
pont-us Feb 16, 2023
18110f4
Minor updates to spec and schema
pont-us Feb 16, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 19 additions & 12 deletions dataset-spec/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,11 +158,11 @@ data cube:

| Property | Type | Req.? | Description | Example |
|------------|--------------------|:-----:|---------------------------------------------------------|--------------------------------|
| name | string | Y | Array of variable definitions. | `"CHL"` |
| long_name | string | Y | Array of variable definitions. | `"Chlorophyll Concentration"` |
| name | string | Y | Name of the variable in the dataset. | `"CHL"` |
| long_name | string | Y | Descriptive name of variable, per CF conventions. | `"Chlorophyll Concentration"` |
| dtype | string | Y | Numpy-compatible data type name. | `float32"` |
| dims | string[] | Y | Array of dimension names. | `["time", "lat", "lon"]"` |
| units | string | null | Y | Physical unit. | `"mg/m^3"`, `"n.a."` |
| units | string | null | Y | Physical unit (null if not applicable). | `"mg/m^3"`, `"n.a."` |
| fill_value | number | null | N | Unscaled values equal to `fill_value` are undefined. | 1.0 |
| valid_min | number | N | Values below that number are undefined. | 0.0 |
| valid_max | number | N | Values above that number are undefined. | 1.0 |
Expand All @@ -174,11 +174,18 @@ letter and should be continued either by letters, digits, or the underscore
(`_`) character. Spaces (' '), hyphens (`-`), and other characters should be
avoided.

The `long_name` corresponds to the [`long_name` attribute](https://cfconventions.org/Data/cf-conventions/cf-conventions-1.10/cf-conventions.html#long-name)
defined in the Climate and Forecast (CF) Metadata Conventions, where it is
described as ‘a long descriptive name which may, for example, be used for
labeling plots.’

If `fill_value` is missing or `null` and `dtype` is a floating point data,
`NaN` is assumed.

`units` should not be specified or set to set to `null` for data that has
no units, e.g., categorical data.
`units` is required despite not always being applicable, in order to reduce the
risk of its omission in cases where is *is* applicable.
For data that has no units (e.g. categorical data), `units` should be set to
`null`.

`time_range` defaults to the dataset's `time_range`. It should not be
specified for variables that have no "time" dimension.
Expand Down Expand Up @@ -222,13 +229,13 @@ data files in a described source, usually by some URL or set of URLs.

The following properties describe access to a xcube data store:

| Property | Type | Req.? | Description | Example |
|----------------------|--------|:-----:|------------------------------------------------------|------------------------|
| name | string | Y | Human-readable title of this data source. | `"Sentinel Hub S2L2A"` |
| variable_names | string | Y | Array of variable names provided by this data store. | `["B06", "SCL"]` |
| store_id | string | Y | xcube data store identifier. | `"sentinelhub"` |
| store_params | object | N | xcube data store parameters. | `{"num_retries": 80}` |
| variable_open_params | object | N | xcube data store open parameters. | |
| Property | Type | Req.? | Description | Example |
|----------------|--------|:-----:|------------------------------------------------------|------------------------|
| name | string | Y | Human-readable title of this data source. | `"Sentinel Hub S2L2A"` |
| variable_names | string | Y | Array of variable names provided by this data store. | `["B06", "SCL"]` |
| store_id | string | Y | xcube data store identifier. | `"sentinelhub"` |
| store_params | object | N | xcube data store parameters. | `{"num_retries": 80}` |
| data_params | object | N | xcube data store open parameters. | |

`variable_open_params` is an object that maps variable names to specific
open parameters passed to the given data store.
Expand Down
85 changes: 57 additions & 28 deletions dataset-spec/template.geojson
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,49 +1,78 @@
{
"type": "Feature",
"properties": {
"title": "Ocean Color Products in the Mediterranean Sea",
"data_id": "OC-CMEMS-MED-005-2005-2022-1d-v1",
"title": "Ocean Color Products in the Mediterranean Sea",
"description": "CMEMS Ocean Colour for Mediterranean Sea, 2005-2022",
"time_range": ["2005-05-01", "2022-05-01"],
"time_period": "1D",
"spatial_ref": "CRS84",
"spatial_bbox": [-6.75, 30, 36.5, 46],
"spatial_res": 0.05,
"metadata": {
"published_on": "2022-10-26",
"keywords": ["mediterranean", "ocean colour", "chlorophyll"]
},
"version": "1.0.3",
"recipe": "https://example.com/recipe/",
"changes": "https://example.com/recipe/CHANGES.md",
"license": "Creative Commons Attribution 4.0 International (CC BY 4.0)",
"variables": [
{
"name": "CHL",
"units": "mg/m^3",
"long_name": "Chlorophyll Concentration",
"dtype": "float32",
"dims": ["time", "lat", "lon"],
"units": "mg/m^3",
"fill_value": -1.0,
"valid_min": 0.0,
"valid_max": 1000.0,
"color_bar_name": "YlGn",
"time_range": ["2005-05-01", "2022-05-01"],
"dims": ["time", "lat", "lon"],
"sources": [
{
"home_url": "https://resources.marine.copernicus.eu/",
"data_url": "https://resources.marine.copernicus.eu/",
"license_url": "https://marine.copernicus.eu/user-corner/service-commitments-and-licence",
"attributions": [],
"processing_steps": [
"Masking out low quality pixels",
"Temporal NaN-mean aggregation",
"Spatial bi-cubic interpolation"
],
"remarks": ""
}
],
"metadata": {
"color_bar_name": "YlGn",
"color_value_min": 0.5,
"color_value_max": 150,
"keywords": ["colour", "marine", "chlorophyll"],
"processing-level": "L4"
}
}
]
],
"sources": [
{
"name": "Sentinel Hub S2L2A",
"variable_names": ["CHL"],
"store_id": "sentinelhub",
"store_params": {
"num_retries": 80
},
"data_params": {
"CHL": ["some_parameter"]
}
}
],
"metadata": {
"Conventions": "CF-1.9",
"acknowledgment": "EO4SIBS, CMEMS, DeepESDL project",
"contributor_name": "Brockmann Geomatics Sweden AB",
"contributor_url": "www.brockmann-geomatics.se",
"creator_email": "info@brockmann-consult.de",
"creator_name": "Brockmann Consult GmbH",
"creator_url": "www.brockmann-consult.de",
"institution": "Brockmann Consult GmbH",
"project": "DeepESDL",
"publisher_email": "info@brockmann-consult.de",
"publisher_name": "Brockmann Consult GmbH",
"license_url": "https://creativecommons.org/licenses/by/4.0/",
"source": "EO4SIBS, CMEMS",
"date_modified": "2022-08-19 16:19:15.359970",
"geospatial_lon_min": -6.75,
"geospatial_lon_max": 36.5,
"geospatial_lat_min": 30.0,
"geospatial_lat_max": 46.0,
"geospatial_lon_resolution": 0.05,
"geospatial_lat_resolution": 0.05,
"temporal_coverage_start": "2005-05-01 00:00:00",
"temporal_coverage_end": "2022-05-01 23:59:59",
"temporal_resolution": "1D"
},
"time_range": ["2005-05-01", "2022-05-01"],
"time_period": "1D",
"spatial_ref": "CRS84",
"spatial_bbox": [-6.75, 30, 36.5, 46],
"spatial_size": [865, 320],
"spatial_offset": [0, 0],
"spatial_res": [0.05, 0.05]
},
"geometry": {
"type": "Polygon",
Expand Down
Loading