Implementation in HDF5
In this section we want to describe how the previously described model for data and metadata can be stored in a HDF5 file. HDF5 provides three elements that define the structure of the format. A group can contain zero or more group, dataset or attribute objects. An attribute in HDF5 is represented by a name and one ore more values of the same datatype. The probably most important structuring element in HDF5 is the dataset: it can contain arbitrary data in a multidimensional array and stores information about its dimensionality, data type and a name. Like a group a dataset may also contain zero or more group, dataset or attribute objects. Since all elements defined in HDF5 have a name, they can be references by a path.
To visualize the hierarchical structure of HDF5 elements are annotated as nested list with multiple indention levels. Each element is represented with its element type followed by ':' and the element name. Attributes can have a value assigned. Literals with enclosed by '<>' are placeholders for the actual values:
<type>:<name> = [<type>|<value>]
A group named 'foo' with an attribute named 'greeting' and the value 'hello world' is shown like this:
group:foo
attribute:greeting = 'hello world'For datasets the following annotations are used:
{{{
dataset:<name> = <type>[] // dataset with one dimension
dataset:<name> = <type>[][] // dataset with two dimensions
dataset:<name> = <type>[] ... n // dataset with n dimensions
dataset:foo = double[1.0, 2.0] // dataset named 'foo' with one dimension of double
// values and the content 1.0 and 2.0 - All HDF5 elements representing an entity of the model for data or metadata are named with their id.
- All optional attributes that are empty or set to NULL have to be omitted.
The root of a HDF5 file implementing this standard contains a set of attributes describing the version and name of the schema as well as time stamps with the creation date an the last update. Further the root contains two groups named metadata and data.
root:
attribute:format = string
attribute:version = string
attribute:created_at = date
attribute:updated_at = date
group:metadata
...
group:data
...A third group named 'terminology' is reserved for future use.
All entities of the data model are stored under the path '/data' in the HDF5 file.
Entities of the type Block can only be stored at the first hierarchy level at the '/data' path. Blocks always have the following structure.
group:<block_id>
attribute:type = string
attribute:name = string
attribute:definition = string
attribute:date = date
attribute:metadata = string
group:sources
// all sources in this block
group:data_arrays
// all data arrays in this block
group:tags
// all tags in this block
group: representations
// all representations in this blockEntities of the type DataArray can only be defined in the group called 'data_arrays' of its respective Block. The path to a DataArray can therefore always be defined like this:
/data/<block_id>/data_arrays/<data_array_id>
In HDF5 DataArray objects always have the following structure:
group:<data_array_id>
attribute:type = string
attribute:name = string
attribute:definition = string
attribute:label = string
attribute:unit = string
attribute:metadata = string
dataset:data = <data_type>[]...n
dataset:sources = string[]
group:dimensions
// definition of all dimensions of the data arrayThe field 'data_type' that is defined in the model is defined becomes part of the dataspace definition of the hdf5 data dataset and thus is not represented in a distinct field. Native data types names will be mapped to the data types of HDF5 as defined in the following table:
| Type name | HDF5 data type |
|---|---|
| byte | H5T_STD_I8LE |
| uint16 | H5T_STD_U16LE |
| uint,uint32 | H5T_STD_U32LE |
| int16 | H5T_STD_I16LE |
| int, int32 | H5T_STD_I32LE |
| long, int64 | H5T_STD_I64LE |
| float | H5T_IEEE_F32LE |
| double | H5T_IEEE_F64LE |
| string | H5T_C_S1 |
Dimensions Set, Range, and Sample can only be defined inside the group called 'dimensions' of their parent DataArray. The 'order' attribute of each dimension type defines the sequential arrangement of the dimension. The dimension with the lowest order value applies to the first dimension etc. The structure of all dimension entities is defined as described below:
// Set
group:<dimension_id>
attribute:order: int
attribute:dimension_type = 'set'
dataset:labels = string[]
// Range
group:<dimension_id>
attribute:order: int
attribute:dimension_type = 'range'
attribute:label = string
attribute:unit = string
dataset:tics = double[]
// Sample
group:<dimension_id>
attribute:order: int
attribute:dimension_type = 'sample'
attribute:label = string
attribute:unit = string
attribute:sampling_interval = double
attribute:offset = double
Entities of the type Tag can only be defined inside first hierarchy level of the group 'tags' of the parent block. The structure of Tag objects in a HDF5 file can be defined as depicted below:
group:<tag_id>
attribute:type = string
attribute:name = string
attribute:definition = string
attribute:metadata = string
dataset:units = string[]
dataset:positions = double[][]
dataset:extents = double[][]
dataset:references = string[] // just for now as one dimensional array of ids
dataset:representations = string[] // just for now as one dimensional array of idsThe definition of Representation entities is strictly restricted to the group 'representations' of parent Tag entities. The schema below describes the structure:
group:<representation_id>
attribute:link_type = enum{tagged, untagged, indexed}
attribute:data = stringEntities of the type Source can only be defined inside the group 'sources' of the parent Block entity.
group:<source_id>
attribute:type = string
attribute:name = string
attribute:definition = string
attribute:metadata = string
attribute:parent_source = stringRepresentation of the metadata model (odML) in HDF5. Metadata objects can only be located in the Metadata group that is a direct child of the root node. Sections are stored in a flat way, there is no hierarchy in the sections. Properties and Values, on the other hand, are stored as children of their parent sections, respectively their parent Properties.
group:<section_id>
attribute:name = string
attribute:type = string
attribute:definition = string
attribute:link = string
attribute:include = string
attribute:repository = string
attribute:mapping = string
attribute:parent_section = string
group:properties group:<property_id>
attribute:name = string
attribute:definition = string
attribute:unit = string
attribute:data_type = string
attribute:mapping = string
group:valuesThe value actually stores the values of the properties.
group:<value_id> dataset:value = string attribute:uncertainty = double attribute:filename = string attribute:checksum = string attribute:encoder = string
Empty elements omitted.
root:
attribute:format = 'pandora'
attribute:version = '1.0'
attribute:created_at = 2013-02-01:..
attribute:updated_at = 2013-02-01:..
group:metadata
group:prefix_01
attribute:author = 'John Doe'
attribute:date = 2013-02-01:..
attribute:version = '1.0'
attribute:repository = 'http://www...'
group:prefix_02
attribute:name = 'First Experiment'
attribute:type = 'experiment'
attribute:definition = '...'
attribute:parent_section = 'prefix_01'
group:properties
group:<property_id>
attribute:name = 'Experimenter'
attribute:definition = '...'
attribute:data_type = 'string'
compound:values
value = 'John Doe'
uncertainty
// more sections with properties and values
group:data
...