You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Create database tables (and schema) from oemetadata json file(s)
3
+
Create database tables (and schema) from oemetadata json file(s). This tool is part of the open-energy-metadata (OEM) integration into the [OEP](https://openenergyplatform.org/).
4
4
5
-
## Installation:
5
+
## Installation
6
6
7
7
You can install pacakge using standard python installation:
8
8
`
@@ -15,94 +15,78 @@ pipx install oem2orm
15
15
`
16
16
see [Pipx-Documentation](https://pypa.github.io/pipx/) for further information.
17
17
18
+
## Usage
18
19
19
-
## Usage:
20
+
Read the Restrictions section and have a look at our [tutorial](./tutorial/USAGE.md) section to get more information about the usage of oem2orm either as code module or CLI tool. The tutorials also provide information how to validate your oemetadata files.
20
21
21
-
This tool is part of the open-energy-metadata (OEM) integration into the [OEP](https://openenergyplatform.org/).
22
-
To use this tool with the OEP API you need to be signed up to the OEP since
23
-
you need to provide an API-Token.
22
+
### Restrictions
24
23
25
-
If you want to upload OEM that was officially reviewed you must clone the
26
-
OEP data-preprocessing repository on [GitHub](https://github.com/OpenEnergyPlatform/data-preprocessing).
27
-
The data-review folder contains all of the successfully reviewed OEM files.
24
+
To use this tool with the OEP API you need to be signed up to the OEP since
25
+
you need to provide an API-Token.
28
26
29
-
For security reasons, tables can only be created in existing
27
+
For security reasons, tables can only be created in existing
30
28
schemas and just in the schemas "model_draft" and "sandbox".
31
29
32
-
Keep in mind the current state is not fully tested. The code is
33
-
still quit error prone f.e. the postgres types (column datatype) are not fully
30
+
Keep in mind that f.e. the postgres types (column datatype) are not fully
34
31
supported by the [oedialct](https://pypi.org/project/oedialect/) - work in progress.
35
32
36
-
### Terminal/CLI-Application
37
-
Step-by-Step:
38
-
0. pip and python have to be installed and setup on your machine
39
-
1. Create env from requirements.txt, and activate
40
-
2. Put the metadata file in the folder metadata or put your own folder in this
41
-
directory
42
-
3. execute the following in a terminal:
43
-
```
44
-
pipx install oem2orm
45
-
oem2orm
46
-
Enter metadata folder name:
47
-
...
48
-
```
49
-
4. Provide credentials and folder name in prompt
50
-
5. The table will be created
33
+
## Docs
51
34
52
-
### Import as Module
35
+
### Database connection
53
36
54
-
You can simply import this module in your Python script.py like this:
37
+
We use a global namedtuple called "DB" To store the sqlalchemy connection objects engine and metadata.
38
+
The namedtuple is available wen import oem2orm in a script. To establish the namedtuple use the function
39
+
setup_db_connection(). Now you can use DB.engine or DB.metadata. In the background the connection is established
40
+
using oedialect and the http API of the oeplatform website.
55
41
56
-
```python
57
-
from oem2orm import oep_oedialect_oem2orm as oem2orm
58
-
```
42
+
### oem2orm generator
59
43
60
-
Now just call the functions provided in oem2orm like this:
44
+
The table objects (ORM) are generated on the fly from an oemetadata.json file. [oemetadata](https://github.com/OpenEnergyPlatform/oemetadata) is a metadata specification of the Open Energy Family. It includes about 50 fields that can be used to provide metadata for tabular data resources.
45
+
A subset of these fields are grouped in the key "resources" ([see out example](https://github.com/OpenEnergyPlatform/oemetadata/blob/develop/metadata/v160/example.json#L237-L388)) in the metadata. These fields describe the schema of
46
+
the data table (like table name, columns, data types & table relations).
61
47
62
-
Recommended execution order:
63
-
- Setup the logger
64
-
```python
65
-
oem2orm.setup_logger()
66
-
```
48
+
The method oem2orm provides to create data tables on the OEP. It is especially useful if you attempt to automate the table creation and already use python or already have a oemetadata file available. The alternatives are:
67
49
68
-
- Setup the Database API connection as Namedtuple storing the SQLAlchemy engine and metadata:
69
-
```python
70
-
db = oem2orm.setup_db_connection()
71
-
```
50
+
1.[manually describing](https://openenergyplatform.github.io/academy/tutorials/01_api/02_api_upload/#create-table) the table object in JSON and then use the oep HTTP API directly to create a table.
51
+
2. Use the [User Interface of the oeplatform website](https://openenergyplatform.org/dataedit/wizard/) to create a table and upload data.
72
52
73
-
- Provide the oem files in a folder (in the current directory).
[Specification for the oemetadata](https://github.com/OpenEnergyPlatform/oemetadata)
83
56
84
-
- Create the tables on the Database:
85
-
```python
86
-
oem2orm.create_tables(db, orm)
87
-
```
57
+
#### Oemetadata validation
88
58
89
-
- Delete all tables that have been created (all tables available in sa.metadata)
90
-
```python
91
-
oem2orm.delete_tables(db, orm)
92
-
```
59
+
The oemetadata specification is integrated into the open energy platform using a tool called [omi (metadata integration)](https://github.com/OpenEnergyPlatform/omi). OMI provides functionality to run validation checks on the metadata up to the oemetadata version 1.6.0. oem2orm also provides a minimal oep compliance check that mocks the checks that are run on the oep website once the metadata is uploaded to a table.
93
60
94
-
##Docs:
61
+
#### Supported column data types
95
62
96
-
### Database connection
97
-
We use a global namedtuple called "DB" To store the sqlalchemy connection objects engine and metadata.
98
-
The namedtuple is available wen import oem2orm in a script. To establish the namedtuple use the function
99
-
setup_db_connection(). Now you can use DB.engine or DB.metadata.
100
-
101
-
### oem2orm generator
63
+
Currently oem2orm supports
102
64
103
-
#### Supported datatypes
65
+
"bigint"
66
+
"int":
67
+
"integer"
68
+
"varchar"
69
+
"json"
70
+
"text"
71
+
"timestamp"
72
+
"interval"
73
+
"string"
74
+
"float"
75
+
"boolean"
76
+
"date"
77
+
"hstore"
78
+
"decimal"
79
+
"numeric"
80
+
"double precision"
104
81
105
82
#### Spatial Types
106
-
We create columns with spatial datatypes using Geoalchemy2.
0 commit comments