-
Notifications
You must be signed in to change notification settings - Fork 2
/
Copy pathdatamodel.html
303 lines (290 loc) · 23.7 KB
/
datamodel.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
<!DOCTYPE html>
<html lang="en" data-content_root="./">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.18.1: http://docutils.sourceforge.net/" />
<title>Datamodel — SpacePy v0.7.0 Manual</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b76e3c8a" />
<link rel="stylesheet" type="text/css" href="_static/sphinxdoc.css?v=92e3d466" />
<link rel="stylesheet" type="text/css" href="_static/graphviz.css?v=fd3f3429" />
<link rel="stylesheet" type="text/css" href="_static/plot_directive.css" />
<script src="_static/documentation_options.js?v=fe7df9b0"></script>
<script src="_static/doctools.js?v=9a2dae69"></script>
<script src="_static/sphinx_highlight.js?v=dc90522c"></script>
<script type="text/javascript" src="_static/copybutton.js"></script>
<link rel="icon" href="_static/spacepy_favicon.ico"/>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="OMNI" href="omni.html" />
<link rel="prev" title="Coordinates - Implementation of Coords class functions for coordinate transformations" href="coordinates.html" />
</head><body>
<div style="background-color: white; text-align: left; padding: 10px 10px 15px 15px">
<a href="index.html"><img src="_static/spacepy_logo.jpg" border="0" alt="spacepy_logo"/></a>
</div>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="genindex.html" title="General Index"
accesskey="I">index</a></li>
<li class="right" >
<a href="py-modindex.html" title="Python Module Index"
>modules</a> |</li>
<li class="right" >
<a href="omni.html" title="OMNI"
accesskey="N">next</a> |</li>
<li class="right" >
<a href="coordinates.html" title="Coordinates - Implementation of Coords class functions for coordinate transformations"
accesskey="P">previous</a> |</li>
<li><a href="https://spacepy.github.io/"">homepage</a>| </li>
<li><a href="https://github.com/spacepy/spacepy">development</a>| </li>
<li><a href="search.html">search</a>| </li>
<li><a href="index.html">documentation </a> »</li>
<li class="nav-item nav-item-this"><a href="">Datamodel</a></li>
</ul>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body" role="main">
<section id="datamodel">
<h1>Datamodel<a class="headerlink" href="#datamodel" title="Link to this heading">¶</a></h1>
<p>The SpacePy datamodel module enables easy implementation of a standard data model, using concepts similar to those of HDF5 and NASA CDF.</p>
<nav class="contents local" id="table-of-contents">
<p class="topic-title">Table of Contents</p>
<ul class="simple">
<li><p><a class="reference internal" href="#documentation" id="id1">Documentation</a></p>
<ul>
<li><p><a class="reference internal" href="#about-datamodel" id="id2">About datamodel</a></p></li>
<li><p><a class="reference internal" href="#examples" id="id3">Examples</a></p></li>
<li><p><a class="reference internal" href="#guide-for-nasa-cdf-users" id="id4">Guide for NASA CDF users</a></p></li>
<li><p><a class="reference internal" href="#a-quick-guide-to-json-headed-ascii" id="id5">A quick guide to JSON-headed ASCII</a></p></li>
</ul>
</li>
</ul>
</nav>
<p>See also the <a class="reference internal" href="autosummary/spacepy.datamodel.html#module-spacepy.datamodel" title="spacepy.datamodel"><code class="xref py py-obj docutils literal notranslate"><span class="pre">full</span> <span class="pre">API</span> <span class="pre">documentation</span></code></a>.</p>
<section id="documentation">
<h2><a class="toc-backref" href="#id1" role="doc-backlink">Documentation</a><a class="headerlink" href="#documentation" title="Link to this heading">¶</a></h2>
<dl class="simple">
<dt>This contains the following classes:</dt><dd><ul class="simple">
<li><p><a class="reference internal" href="autosummary/spacepy.datamodel.dmarray.html#spacepy.datamodel.dmarray" title="spacepy.datamodel.dmarray"><code class="xref py py-class docutils literal notranslate"><span class="pre">dmarray</span></code></a> - numpy arrays that support .attrs for information about the data</p></li>
<li><p><a class="reference internal" href="autosummary/spacepy.datamodel.SpaceData.html#spacepy.datamodel.SpaceData" title="spacepy.datamodel.SpaceData"><code class="xref py py-class docutils literal notranslate"><span class="pre">SpaceData</span></code></a> - base class that extends dict, to be extended by others</p></li>
</ul>
</dd>
</dl>
<p>Authors: Steve Morley and Brian Larsen</p>
<p>Additional Contributors: Charles Kiyanda and Miles Engel</p>
<p>Institution: Los Alamos National Laboratory</p>
<p>Contact: <a class="reference external" href="mailto:smorley%40lanl.gov">smorley<span>@</span>lanl<span>.</span>gov</a>; <a class="reference external" href="mailto:balarsen%40lanl.gov">balarsen<span>@</span>lanl<span>.</span>gov</a></p>
<p>Copyright 2010-2016 Los Alamos National Security, LLC.</p>
<section id="about-datamodel">
<h3><a class="toc-backref" href="#id2" role="doc-backlink">About datamodel</a><a class="headerlink" href="#about-datamodel" title="Link to this heading">¶</a></h3>
<p>The SpacePy datamodel module implements classes that are designed to make implementing a standard
data model easy. The concepts are very similar to those used in standards like HDF5, netCDF and
NASA CDF.</p>
<p>The basic container type is analogous to a folder (on a filesystem; HDF5 calls this a
group): Here we implement this as a dictionary-like object, a datamodel.SpaceData object, which
also carries attributes. These attributes can be considered to be global, i.e. relevant for the
entire folder. The next container type is for storing data and is based on a numpy array, this
class is datamodel.dmarray and also carries attributes. The dmarray class is analogous to an
HDF5 dataset.</p>
<p>In fact, HDF5 can be loaded directly into a SpacePy datamodel, carrying across all attributes,
using the function fromHDF5:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">spacepy.datamodel</span> <span class="k">as</span> <span class="nn">dm</span>
<span class="gp">>>> </span><span class="n">data</span> <span class="o">=</span> <span class="n">dm</span><span class="o">.</span><span class="n">fromHDF5</span><span class="p">(</span><span class="s1">'test.h5'</span><span class="p">)</span>
</pre></div>
</div>
<p>Functions are also available to directly load data and metadata into a
SpacePy datamodel from NASA CDF as well as JSON-headed ASCII. Writers also
exist to output a SpacePy datamodel directly to HDF5 or JSON-headed ASCII.
See <a class="reference internal" href="autosummary/spacepy.datamodel.html#spacepy.datamodel.fromCDF" title="spacepy.datamodel.fromCDF"><code class="xref py py-func docutils literal notranslate"><span class="pre">fromCDF()</span></code></a>, <a class="reference internal" href="autosummary/spacepy.datamodel.html#spacepy.datamodel.readJSONheadedASCII" title="spacepy.datamodel.readJSONheadedASCII"><code class="xref py py-func docutils literal notranslate"><span class="pre">readJSONheadedASCII()</span></code></a>,
<a class="reference internal" href="autosummary/spacepy.datamodel.html#spacepy.datamodel.toHDF5" title="spacepy.datamodel.toHDF5"><code class="xref py py-func docutils literal notranslate"><span class="pre">toHDF5()</span></code></a>, and <a class="reference internal" href="autosummary/spacepy.datamodel.html#spacepy.datamodel.toJSONheadedASCII" title="spacepy.datamodel.toJSONheadedASCII"><code class="xref py py-func docutils literal notranslate"><span class="pre">toJSONheadedASCII()</span></code></a> for more details.</p>
</section>
<section id="examples">
<h3><a class="toc-backref" href="#id3" role="doc-backlink">Examples</a><a class="headerlink" href="#examples" title="Link to this heading">¶</a></h3>
<p>Imagine representing some satellite data within the global attributes might be
the mission name and the instrument PI, the variables might be the
instrument counts [n-dimensional array], timestamps[1-dimensional array and an orbit number [scalar].
Each variable will have one attribute (for this example).</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">spacepy.datamodel</span> <span class="k">as</span> <span class="nn">dm</span>
<span class="gp">>>> </span><span class="n">mydata</span> <span class="o">=</span> <span class="n">dm</span><span class="o">.</span><span class="n">SpaceData</span><span class="p">(</span><span class="n">attrs</span><span class="o">=</span><span class="p">{</span><span class="s1">'MissionName'</span><span class="p">:</span> <span class="s1">'BigSat1'</span><span class="p">})</span>
<span class="gp">>>> </span><span class="n">mydata</span><span class="p">[</span><span class="s1">'Counts'</span><span class="p">]</span> <span class="o">=</span> <span class="n">dm</span><span class="o">.</span><span class="n">dmarray</span><span class="p">([[</span><span class="mi">42</span><span class="p">,</span> <span class="mi">69</span><span class="p">,</span> <span class="mi">77</span><span class="p">],</span> <span class="p">[</span><span class="mi">100</span><span class="p">,</span> <span class="mi">200</span><span class="p">,</span> <span class="mi">250</span><span class="p">]],</span> <span class="n">attrs</span><span class="o">=</span><span class="p">{</span><span class="s1">'Units'</span><span class="p">:</span> <span class="s1">'cnts/s'</span><span class="p">})</span>
<span class="gp">>>> </span><span class="n">mydata</span><span class="p">[</span><span class="s1">'Epoch'</span><span class="p">]</span> <span class="o">=</span> <span class="n">dm</span><span class="o">.</span><span class="n">dmarray</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">],</span> <span class="n">attrs</span><span class="o">=</span><span class="p">{</span><span class="s1">'units'</span><span class="p">:</span> <span class="s1">'minutes'</span><span class="p">})</span>
<span class="gp">>>> </span><span class="n">mydata</span><span class="p">[</span><span class="s1">'OrbitNumber'</span><span class="p">]</span> <span class="o">=</span> <span class="n">dm</span><span class="o">.</span><span class="n">dmarray</span><span class="p">(</span><span class="mi">16</span><span class="p">,</span> <span class="n">attrs</span><span class="o">=</span><span class="p">{</span><span class="s1">'StartsFrom'</span><span class="p">:</span> <span class="mi">1</span><span class="p">})</span>
<span class="gp">>>> </span><span class="n">mydata</span><span class="o">.</span><span class="n">attrs</span><span class="p">[</span><span class="s1">'PI'</span><span class="p">]</span> <span class="s1">'Prof. Big Shot'</span>
</pre></div>
</div>
<p>This has now populated a structure that can map directly to a NASA CDF, HDF5 or JSON-headed ASCII file.
To visualize our datamodel, we can use tree method (which can be applied to any dictionary-like object
using <a class="reference internal" href="autosummary/spacepy.toolbox.html#spacepy.toolbox.dictree" title="spacepy.toolbox.dictree"><code class="xref py py-func docutils literal notranslate"><span class="pre">dictree()</span></code></a>).</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">mydata</span><span class="o">.</span><span class="n">tree</span><span class="p">(</span><span class="n">attrs</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</pre></div>
</div>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="o">+</span>
<span class="p">:</span><span class="o">|</span><span class="n">____MissionName</span>
<span class="p">:</span><span class="o">|</span><span class="n">____PI</span>
<span class="o">|</span><span class="n">____Counts</span>
<span class="p">:</span><span class="o">|</span><span class="n">____Units</span>
<span class="o">|</span><span class="n">____Epoch</span>
<span class="p">:</span><span class="o">|</span><span class="n">____units</span>
<span class="o">|</span><span class="n">____OrbitNumber</span>
<span class="p">:</span><span class="o">|</span><span class="n">____StartsFrom</span>
</pre></div>
</div>
</section>
<section id="guide-for-nasa-cdf-users">
<h3><a class="toc-backref" href="#id4" role="doc-backlink">Guide for NASA CDF users</a><a class="headerlink" href="#guide-for-nasa-cdf-users" title="Link to this heading">¶</a></h3>
<p>By definition, a NASA CDF only has a single ‘layer’. That is, a CDF contains a series of records
(stored variables of various types) and a set of attributes that are either global or local in
scope. Thus to use SpacePy’s datamodel to capture the functionality of CDF the two basic data types
are all that is required, and the main constraint is that datamodel.SpaceData objects cannot be
nested (more on this later, if conversion from a nested datamodel to a flat datamodel is required).</p>
<p>Opening a CDF and working directly with the contents can be easily done using the PyCDF module, however,
if you wish to load the entire contents of a CDF directly into a datamodel (complete with attributes)
the following will make life easier:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">spacepy.datamodel</span> <span class="k">as</span> <span class="nn">dm</span>
<span class="gp">>>> </span><span class="n">data</span> <span class="o">=</span> <span class="n">dm</span><span class="o">.</span><span class="n">fromCDF</span><span class="p">(</span><span class="s1">'inFile.cdf'</span><span class="p">)</span>
</pre></div>
</div>
</section>
<section id="a-quick-guide-to-json-headed-ascii">
<h3><a class="toc-backref" href="#id5" role="doc-backlink">A quick guide to JSON-headed ASCII</a><a class="headerlink" href="#a-quick-guide-to-json-headed-ascii" title="Link to this heading">¶</a></h3>
<p>In many cases it is preferred to have a human-readable ASCII file, rather than a binary file like CDF
or HDF5. To make it easier to carry all the same metadata that is available in HDF5 or CDF we have
developed an ASCII data storage format that encodes the metadata using JSON (JavaScript Object Notation).
This notation supports two basic datatypes: key/value collections (like a SpaceData) and ordered lists
(which can represent arrays). JSON is human-readable, but if large arrays are stored in metadata is quickly
becomes difficult to read. For this reason we use JSON to encode the metadata (usually smaller datasets)
and store the data in a standard flat-ASCII format. The metadata is provided as a header that describes
the contents of the file.</p>
<p>To use JSON for storing only metadata associated with the data to be written to an ASCII file a minimal
metadata standard must be implemented. We use the following attribute names: DIMENSION and START_COLUMN.
We also recommend using the NASA ISTP metadata standard to assign attribute names. The biggest limitation
of flat ASCII is that sensibly formatting datasets of more than 2-dimensions (i.e. ranks greater than 2)
is not possible. For this reason if you have datasets of rank 3 or greater then we recommend using HDF5.
If text is absolutely required then it is possible to encode multi-dimensional arrays in the JSON metadata,
but this is not recommended.</p>
<p>This format is best understood by illustration. The following example builds a toy SpacePy datamodel and
writes it to a JSON-headed ASCII file. The contents of the file are then shown.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">spacepy.datamodel</span> <span class="k">as</span> <span class="nn">dm</span>
<span class="gp">>>> </span><span class="n">data</span> <span class="o">=</span> <span class="n">dm</span><span class="o">.</span><span class="n">SpaceData</span><span class="p">()</span>
<span class="gp">>>> </span><span class="n">data</span><span class="o">.</span><span class="n">attrs</span><span class="p">[</span><span class="s1">'Global'</span><span class="p">]</span> <span class="o">=</span> <span class="s1">'A global attribute'</span>
<span class="gp">>>> </span><span class="n">data</span><span class="p">[</span><span class="s1">'Var1'</span><span class="p">]</span> <span class="o">=</span> <span class="n">dm</span><span class="o">.</span><span class="n">dmarray</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">4</span><span class="p">,</span><span class="mi">5</span><span class="p">],</span> <span class="n">attrs</span><span class="o">=</span><span class="p">{</span><span class="s1">'Local1'</span><span class="p">:</span> <span class="s1">'A local attribute'</span><span class="p">})</span>
<span class="gp">>>> </span><span class="n">data</span><span class="p">[</span><span class="s1">'Var2'</span><span class="p">]</span> <span class="o">=</span> <span class="n">dm</span><span class="o">.</span><span class="n">dmarray</span><span class="p">([[</span><span class="mi">8</span><span class="p">,</span><span class="mi">9</span><span class="p">],[</span><span class="mi">9</span><span class="p">,</span><span class="mi">1</span><span class="p">],[</span><span class="mi">3</span><span class="p">,</span><span class="mi">4</span><span class="p">],[</span><span class="mi">8</span><span class="p">,</span><span class="mi">9</span><span class="p">],[</span><span class="mi">7</span><span class="p">,</span><span class="mi">8</span><span class="p">]])</span>
<span class="gp">>>> </span><span class="n">data</span><span class="p">[</span><span class="s1">'MVar'</span><span class="p">]</span> <span class="o">=</span> <span class="n">dm</span><span class="o">.</span><span class="n">dmarray</span><span class="p">([</span><span class="mf">7.8</span><span class="p">],</span> <span class="n">attrs</span><span class="o">=</span><span class="p">{</span><span class="s1">'Note'</span><span class="p">:</span> <span class="s1">'Metadata'</span><span class="p">})</span>
<span class="gp">>>> </span><span class="n">dm</span><span class="o">.</span><span class="n">toJSONheadedASCII</span><span class="p">(</span><span class="s1">'outFile.txt'</span><span class="p">,</span> <span class="n">data</span><span class="p">,</span> <span class="n">depend0</span><span class="o">=</span><span class="s1">'Var1'</span><span class="p">,</span> <span class="n">order</span><span class="o">=</span><span class="p">[</span><span class="s1">'Var1'</span><span class="p">])</span>
<span class="go">#Note that not all field names are required, those not given will be listed</span>
<span class="go">#alphabetically after those that are specified</span>
</pre></div>
</div>
<p>The file looks like:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>#{
# "MVar": {
# "Note": "Metadata",
# "VALUES": [7.8]
# },
# "Global": "A global attribute",
# "Var1": {
# "Local1": "A local attribute",
# "DIMENSION": [1],
# "START_COLUMN": 0
# },
# "Var2": {
# "DIMENSION": [2],
# "START_COLUMN": 2
# }
#}
1 8 9
2 9 1
3 3 4
4 8 9
5 7 8
</pre></div>
</div>
</section>
</section>
</section>
<div class="clearer"></div>
</div>
</div>
</div>
<div class="sphinxsidebar" role="navigation" aria-label="main navigation">
<div class="sphinxsidebarwrapper">
<p class="logo"><a href="index.html">
<img class="logo" src="_static/logo.png" alt="Logo"/>
</a></p>
<div>
<h3><a href="index.html">Table of Contents</a></h3>
<ul>
<li><a class="reference internal" href="#">Datamodel</a><ul>
<li><a class="reference internal" href="#documentation">Documentation</a><ul>
<li><a class="reference internal" href="#about-datamodel">About datamodel</a></li>
<li><a class="reference internal" href="#examples">Examples</a></li>
<li><a class="reference internal" href="#guide-for-nasa-cdf-users">Guide for NASA CDF users</a></li>
<li><a class="reference internal" href="#a-quick-guide-to-json-headed-ascii">A quick guide to JSON-headed ASCII</a></li>
</ul>
</li>
</ul>
</li>
</ul>
</div>
<div>
<h4>Previous topic</h4>
<p class="topless"><a href="coordinates.html"
title="previous chapter">Coordinates - Implementation of Coords class functions for coordinate transformations</a></p>
</div>
<div>
<h4>Next topic</h4>
<p class="topless"><a href="omni.html"
title="next chapter">OMNI</a></p>
</div>
<div role="note" aria-label="source link">
<h3>This Page</h3>
<ul class="this-page-menu">
<li><a href="_sources/datamodel.rst.txt"
rel="nofollow">Show Source</a></li>
</ul>
</div>
<search id="searchbox" style="display: none" role="search">
<h3 id="searchlabel">Quick search</h3>
<div class="searchformwrapper">
<form class="search" action="search.html" method="get">
<input type="text" name="q" aria-labelledby="searchlabel" autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false"/>
<input type="submit" value="Go" />
</form>
</div>
</search>
<script>document.getElementById('searchbox').style.display = "block"</script>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="genindex.html" title="General Index"
>index</a></li>
<li class="right" >
<a href="py-modindex.html" title="Python Module Index"
>modules</a> |</li>
<li class="right" >
<a href="omni.html" title="OMNI"
>next</a> |</li>
<li class="right" >
<a href="coordinates.html" title="Coordinates - Implementation of Coords class functions for coordinate transformations"
>previous</a> |</li>
<li><a href="https://spacepy.github.io/"">homepage</a>| </li>
<li><a href="https://github.com/spacepy/spacepy">development</a>| </li>
<li><a href="search.html">search</a>| </li>
<li><a href="index.html">documentation </a> »</li>
<li class="nav-item nav-item-this"><a href="">Datamodel</a></li>
</ul>
</div>
<div class="footer" role="contentinfo">
© Copyright 2011-2024, The SpacePy Team.
Created using <a href="https://www.sphinx-doc.org/">Sphinx</a> 7.3.7.
</div>
</body>
</html>