Skip to content

Commit bce0c7a

Browse files
VisLabVisLab
authored
Merge pull request #385 from VisLab/develop
Added draft documentation for HED and NWB
2 parents 7117cf9 + 16e31ed commit bce0c7a

13 files changed

+3554
-3550
lines changed

‎docs/source/HedAnnotationInNWB.md

Lines changed: 198 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,198 @@
1+
# HED annotation in NWB (draft)
2+
3+
[**Neurodata Without Borders (NWB)**](https://www.nwb.org/) is a data standard for organizing neurophysiology data.
4+
NWB is used extensively as the data representation for single cell and animal recordings as well as
5+
human neuroimaging modalities such as IEEG. HED (Hierarchical Event Descriptors) is a system of
6+
standardized vocabularies and supporting tools that allows fine-grained annotation of data.
7+
HED annotations can now be used in NWB.
8+
9+
A standardized HED vocabulary is referred to as a HED schema.
10+
A single term in a HED vocabulary is called a HED tag.
11+
A HED string consists of one or more HED tags separated by commas and possibly grouped using parentheses.
12+
13+
The [**ndx-hed**](https://github.com/hed-standard/ndx-hed) extension consists of a `HedTags` class that extends
14+
the NWB [**VectorData**](https://hdmf-common-schema.readthedocs.io/en/stable/format.html#sec-dynamictable) class,
15+
allowing HED data to be added as a column to any NWB [**DynamicTable**](https://hdmf-common-schema.readthedocs.io/en/stable/format.html#sec-dynamictable).
16+
`VectorData` and `DynamicTable` are base classes for many NWB data structures.
17+
See the [**DynamicTable Tutorial**](https://hdmf.readthedocs.io/en/stable/tutorials/plot_dynamictable_tutorial.html#sphx-glr-tutorials-plot-dynamictable-tutorial-py)
18+
for a basic guide for usage in Python and
19+
[**DynamicTable Tutorial (MATLAB)**](https://neurodatawithoutborders.github.io/matnwb/tutorials/html/dynamic_tables.html)
20+
for introductory usage in MATLAB.
21+
The `ndx-hed` extension is not currently supported in MATLAB, although support is planned in the future.
22+
23+
## NWB ndx-hed installation
24+
25+
Should it be uploaded to PyPi?
26+
27+
## NWB ndx-hed examples
28+
29+
### HedTags as a standalone vector
30+
31+
The `HedTags` class has two required arguments (`hed_version` and `data`) and two optional arguments
32+
(`name` and `description`).
33+
The result of the following example is a `HedTags` object whose data vector includes 2 elements.
34+
Notice that the `data` argument value is a list with 2 values representing two distinct HED strings.
35+
These values are validated using HED schema version 8.3.0 when `tags` is created.
36+
If any of the tags had been invalid, the constructor would have raised a `ValueError`.
37+
The example uses the default column name (`HED`) and the default column description.
38+
39+
````{admonition} Create a HedTags object.
40+
:class: tip
41+
42+
```python
43+
tags = HedTags(hed_version='8.3.0', data=["Correct-action", "Incorrect-action"])
44+
```
45+
````
46+
47+
You must specify the version of the HED vocabulary to be used.
48+
We recommend that you use the latest version of HED (currently 8.3.0).
49+
A separate HED version is used for each instance of the `HedTags` column,
50+
so in theory you could use a different version for each column.
51+
This is not recommended, as annotations across columns and tables may be combined for analysis.
52+
See [**Understanding HED versions**](./UnderstandingHedVersions.md) for a more detailed explanation
53+
of HED versioning.
54+
55+
### Adding a row to HedTags
56+
57+
The following example assumings that a `HedTags` object `tags` as already been
58+
created as illustrated in the previous example.
59+
60+
````{admonition} Add a row to an existing HedTags object
61+
:class: tip
62+
63+
```python
64+
tags.add_row("Sensory-event, Visual-presentation")
65+
```
66+
````
67+
68+
After this `add_row` operation, `tags` has 3 elements. Notice that "Sensory-event, Visual-presentation"
69+
is a single HED string, not two HED strings.
70+
In contrast, ["Correct-action", "Incorrect-action"] is a list with two HED strings.
71+
72+
### HED in a table
73+
74+
The following color table uses HED tags to define the meanings of integer codes:
75+
| color_code | HED |
76+
|----- | --- |
77+
| 1 | `Red` |
78+
| 2 | `Green` |
79+
| 3 | `Blue` |
80+
81+
````{admonition} Create an NWB DynamicTable to represent the color table.
82+
:class: tip
83+
84+
```python
85+
86+
color_nums = VectorData(name="color_code", description="Internal color codes", data=[1,2,3])
87+
color_tags = HedTags(name="HED", hed_version="8.2.0", data=["Red", "Green", "Blue"])
88+
color_table = DynamicTable(
89+
name="colors", description="Experimental colors", columns=[color_num, color_tags])
90+
```
91+
````
92+
The example sets up a table with columns named `color_code` and `HED`.
93+
Table `colors` has 3 rows.
94+
95+
### Add a row to a `DynamicTable`
96+
Once a table has been required, you can add a row using the table's `add_row` method.
97+
98+
````{admonition} Get row 0 of color_table as a Pandas DataFrame:
99+
```python
100+
df = color_table[0]
101+
```
102+
Append a row to `color_table`:
103+
```python
104+
color_table.add_row(color_code=4, HED="Black")
105+
```
106+
````
107+
As mentioned above, the `DynamicTable` class is used as the base class for many table classes including the
108+
`TimeIntervals`, `Units`, and `PlaneSegmentation`.
109+
For example `icephys` classes that extend `DynamicTable` include `ExperimentalConditionsTable`, `IntracellularElectrodesTable`,
110+
`IntracellularResponsesTable`, `IntracellularStimuliTable`, `RepetitionsTable`, `SequentialRecordingsTable`,
111+
`SimultaneousRecordingsTable` and the `SweepTable`.
112+
This means that HED can be used to annotate a variety of NWB data.
113+
114+
HED tools recognize a column as containing HED annotations if it is an instance of `HedTags`.
115+
This is in contrast to BIDS ([**Brain Imaging Data Structure**](https://bids.neuroimaging.io/)),
116+
which identifies HED in tabular files by the presence of a `HED` column,
117+
or by an accompanying JSON sidecar, which associates HED annotations with tabular column names.
118+
119+
## HED and ndx-events
120+
121+
The NWB [**ndx-events**](https://github.com/rly/ndx-events) extension provides data structures for
122+
representing event information about data recordings.
123+
The following table lists elements of the *ndx-events* extension that inherit from
124+
`DynamicTable` and can accommodate HED annotations.
125+
126+
```{list-table} ndx-events tables that can use HED.
127+
:header-rows: 1
128+
:name: ndx-events-data-structures
129+
130+
* - Table
131+
- Purpose
132+
- Comments
133+
* - `EventsTypesTable`
134+
- Information about each event type<br/>One row per event type.
135+
- Analogous to BIDS events.json.
136+
* - `EventsTable`
137+
- Stores event instances<br/>One row per event instance.
138+
- Analogous to BIDS events.tsv.
139+
* - `TtlTypesTable`
140+
- Information about each TTL type.
141+
-
142+
* - `TtlTable`
143+
- Information about each TTL instance.
144+
-
145+
```
146+
147+
HED annotations that are common to a particular type of event can be added to e NWB `EventsTypesTable`,
148+
which is analogous to the `events.json` in BIDS.
149+
A `HED` column can be added to a BIDS `events.tsv` file to provide HED annotations specific
150+
to each event instance.
151+
Any number of `HedTags` columns can be added to the NWB `EventsTable` to provide different types
152+
of HED annotations for each event instance.
153+
154+
The HEDTools ecosystem currently supports assembling the annotations from all sources to construct
155+
complete annotations for event instances in BIDS. Similar support is planned for NWB files.
156+
157+
## HED in NWB files
158+
159+
A single NWB recording and its supporting data is stored in an `NWBFile` object.
160+
The NWB infrastructure efficiently handles reading, writing, and accessing large `NWBFile` objects and their components.
161+
The following example shows the creation of a simple `NWBFile` using only the required constructor arguments.
162+
163+
164+
````{admonition} Create an NWBFile object called my_nwb.
165+
```python
166+
from datetime import datetime
167+
from dateutil.tz import tzutc
168+
from pynwb import NWBFile
169+
170+
my_nwb = NWBFile(session_description='a test NWB File',
171+
identifier='TEST123',
172+
session_start_time=datetime(1970, 1, 1, 12, tzinfo=tzutc()))
173+
174+
```
175+
````
176+
177+
An `NWBFile` has many fields, which can be set using optional parameters to the constructor
178+
or set later using method calls.
179+
180+
````{admonition} Add a HED trial column to an NWB trial table and add trial information.
181+
```python
182+
my_nwb.add_trial_column(name="HED", hed_version="8.3.0", col_cls=HedTags, data=[], description="temp")
183+
my_nwb.add_trial(start_time=0.0, stop_time=1.0, HED="Correct-action")
184+
my_nwb.add_trial(start_time=2.0, stop_time=3.0, HED="Incorrect-action")
185+
```
186+
````
187+
The optional parameters for the `NWBFile` constructor whose values can inherit from `DynamicTable`
188+
include `epochs`, `trials`, `invalid_times`, `units`, `electrodes`, `sweep_table`,
189+
`intracellular_recordings`, `icephys_simultaneous_recordings`, `icephys_repetitions`, and
190+
`icephys_experimental_conditions`.
191+
The `NWBFile` class has methods of the form `add_xxx_column` for the
192+
`epochs`, `electrodes`, `trials`, `units`,and `invalid_times` tables.
193+
The other tables also allow a HED column to be added by constructing the appropriate table
194+
prior to passing it to the `NWBFile` constructor.
195+
196+
In addition, the `stimulus` input is a list or tuple of objects that could include `DynamicTable` objects.
197+
198+
The NWB infrastructure provides IO functions to serialize these HED-augmented tables.

‎docs/source/HedMatlabTools.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -722,8 +722,8 @@ For the remodeling operations, first and second operation must be the dataset ro
722722
directory and the remodeling file name, respectively.
723723
In this example, dataset `ds003645` has been downloaded from [**openNeuro**](https://openneuro.org) to the `G:\` drive.
724724
The remodeling file used in this example can be found at
725-
See [**File remodeling quickstart**](FileRemodelingQuickstart.md)
726-
and [**File remodeling tools**](FileRemodelingTools.md) for
725+
See [**HED remodeling quickstart**](HedRemodelingQuickstart)
726+
and [**HED remodeling tools**](HedRemodelingTools) for
727727
additional information.
728728

729729
(web-service-matlab-demos-anchor)=

‎docs/source/HedOnlineTools.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -151,8 +151,8 @@ to generate a JSON sidecar based on all the events files in a BIDS dataset.
151151
The HED remodeling tools provide an interface to nearly all the HED tools functionality without programming. To use the tools, create a JSON file containing the commands that you
152152
wish to execute on the events file. Command are available to do various transformations
153153
and summaries of events files as explained in
154-
the [**File remodeling quickstart**](https://www.hed-resources.org/en/latest/FileRemodelingQuickstart.html) and the
155-
[**File remodeling tools**](https://www.hed-resources.org/en/latest/FileRemodelingTools.html).
154+
the [**HED remodeling quickstart**](https://www.hed-resources.org/en/latest/HedRemodelingQuickstart.html) and the
155+
[**HED remodeling tools**](https://www.hed-resources.org/en/latest/HedRemodelingTools.html).
156156

157157
``````{admonition} Execute a remodel script.
158158

0 commit comments

Comments
 (0)