Observation
The main standard about Environmental Data is the ISO-19156 "observation and measurement" standard :
"This International Standard defines a conceptual schema for observations, and for features involved in sampling when making observations. These provide models for the exchange of information describing observation acts and their results, both within and between different scientific and technical communities."
But this standard specifies that:
"ISO 19156 concerns only interfaces visible from the outside and does not impose any restrictions on the underlying implementations"
The other standards concern more restricted areas and are sometimes incompatible. They often remain at a syntactic and non-semantic level. The proposed data structure is based on existing standards that it complements by ensuring convergence:
In the ISO-19156 standard, an Observation is characterized by:
- "observed property": the observed property,
- "feature of interest": the object (most often a location) of the observation,
- "procedure": the information acquisition mode (sensor, model, etc.)
- "result": result of the observation or the measurement
The result is a set of values or objects referenced according to the 3 dimensions:
- temporal,
- spatial,
- physical (observed property)
In the ES project, the three dimensions are respected but it is possible to add other dimensions.
An Observation object consists of five components :
- a set of Features (Datation Object, Location Object, Property Object, Variable Object)
- a Result Object
- an Index object (link between Result and Axes)
- a Parameter object
It can be converted into a multi-dimensional matrix (one dimension for each Feature), each result being indexed by Features Values (e.g. temporal, spatial, physical values).
Common properties (indicators) are associated with each Observation. They allow processing to be performed on Observations without having to know their composition (e.g. bounding boxes, type of observation, volume, etc.).
The implemented data structure respects the principles of the standard.
An Observation object groups common data (name, options, parameters) and associates each of the Features
- spatial (LocationValue objects)
- temporal (DatingValue objects)
- properties (PropertyValue objects)
- others (NamedValue, ExternValue objects)
to results (any objects).
An Observation can represent both a single measurement and a large historical data set.
In the usual tabular representations (like Excel or csv) there is one row for each Result and a lot of columns for Datation, Location and Property. This representation is simple and readable, but it duplicates the information and is not suitable for updates.
In the ES project, we choose the indexed representation suitable for computer processing. Thus, the Result object is made up of its own attributes as well as an index to the Features objects.
A result is associated with a property, a location and a date. The Result Object is therefore indexed with three axes (dimension = 3). But there are two cases where the dimension is reduced:
- if an axis has only one value
- if two axes are coupled
For example, if on a path we measure a property, the dimension is 1 (Location and Datation are coupled, Property has one value).
This notion is important because it conditions the modes of representation (e.g., plot).
Parameters and additional information may be added at different levels:
- global : At the Observation level, we may add specific informations with specific
key / value e.g.:
- { 'type' : 'observation' ,'test campaign' : 'first', 'datation' : 'morning' , 'location' : 'paris' , 'property' : ' Temp' , 'result' : 'high' }
- local : We may customize the 'name' attribute e.g.:
- *'datation' : [ { 'first campaign' : '2021-01-01' } , { 'second campaign' : ''2021-03-01' } ]
- 'property' : [ { 'inside' : 'Temp' } , { 'outside' : 'Temp' } ]*
- individual : Each Result value may be annotate e.g.:
- *'result' : { 'first result' : 'high' }
- 'result' : {'https://loco-philippe.github.io/ES.html' : 'high' }*
The PropertyValues are defined in a catalogue which may be specific.
Some operations are available for Observation objects
-
between two Observation :
- Addition function : The Result values are added
- Extend function : The axes are added (e.g. template Observation)
- Append function : An Observation is included as a Result value (e.g. tree structure)
-
for one Observation :
- Sort function : changing the order of data according to the axes order
- Find function : selecting some data
- Full function : add empty Result value to complete an Observation
These operations make it possible to cut, group, restructure and classify the Observations.
The JSON format is used for Observation interchange. The ObsJSON format support the Observation data model. This means that an Observation generated from a JSON format from another Observation is identical to this one.
This format is defined in the ObsJSON document.
This format is extended to JSON binary format which allows :
- storage of observation as a file
- natural representation of dates (datetime)
- taking into account all types of objects in binary form
Two binary format are available :
- BSON format (used in NoSQL databases like Mongodb)
- CBOR format (dedicated to small code size, small message size)
The binary payload is necessary for exchanges with LPWAN networks (e.g. LoRaWAN). The payload should be as compact as possible to minimize the Time-on-Air and reduce power consumption (for battery operated) devices. For example, the maximum lenght of the payload is between 51 bytes and 222 bytes for LoRaWAN.
To obtain this maximum length, limitations are imposed.
Two formats are defined :
- CBOR format with optimization to insure low payload,
- internal optimized format for the most simple usages.
The diagram above shows the structure of the payload.
Note : The right side of the diagram explains the coding of the values. This coding is the same as that used by Bluetooth in the Environmental Sensing Service.
To obtain low payload, a specific process can be used (see below). It allows data to be sent in two stages: first send metadata, second (in operation use) send data.
The Environmental Sensing Service is a Bluetooth protocol for sensors. The data exposed in this protocol are compatible and consistent with the Observation data model. Thus, Bluetooth data is automatically converted into Observation data.
The diagram below shows the mapping of the two structures.
Xarray is very powerful to analyze and process multi-dimensional data. Xarray share the same principle as Observation: indexed multi-dimensional data. Thus, it's natural du use Xarray if you want to analyze Observation data.
The difference between Observation and Xarray is that Xarray uses matrix data and Observation uses only indexed data. Therefore, to transfer Observation data to Xarray we must complete data with 'nan' value to obtain a complete matrix with the right dimension (1, 2 or 3).
The diagram below shows the mapping of the two structures.
