Skip to content

Commit 247bbc2

Browse files
author
philblackwood
committed
add README and yaml files
1 parent 7b99e61 commit 247bbc2

File tree

3 files changed

+371
-0
lines changed

3 files changed

+371
-0
lines changed

‎migration/v13.0/README.md

Lines changed: 143 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,143 @@
1+
# Migrating to gist v13.0 from v12.0
2+
3+
This file gives a quick overview of using the migration utilities provided
4+
by the gist team to help migrate from the previous version of gist.
5+
6+
> See also additional documentation in the file `docs/MajorVersionMigration.md`.
7+
8+
## Important Remarks
9+
10+
- **These scripts are not cumulative; you must run the migrations for each version of gist
11+
in sequence.**
12+
13+
- **These utilities are a starting point.** You should review them before running
14+
them to make sure you approve of the actions prior to using them.
15+
16+
- **These utilities do not handle everything**, in some cases we can only warn you
17+
about changes that were made in gist and you will need to decide how you want
18+
to handle the changes.
19+
20+
## Migration Queries
21+
22+
All of our migration tools are SPARQL queries.
23+
24+
The following directory structure holds the migration scripts:
25+
26+
```
27+
.
28+
└── queries/
29+
│
30+
├── action/
31+
│ ├── default/
32+
│ │ └── *.rq
33+
│ └── ngraphs/
34+
│ └── *.rq
35+
│
36+
├── report/
37+
│ ├── default/
38+
│ │ └── *.rq
39+
│ └── ngraphs/
40+
│ └── *.rq
41+
│
42+
└── uom_queries/
43+
└── *.rq
44+
45+
```
46+
47+
The `./queries/` directory contains the following directories:
48+
49+
- `./queries/action/` : Update queries for migrating from Gist v12.0 to v13.0
50+
51+
- `./queries/report/` : Validation queries that return SHACL validation reports for data that does not comply with Gist v13.0
52+
53+
- `./queries/uom_queries/` : Queries for migrating to the new ontology for magnitudes, aspects, and units of measure
54+
55+
The action and report directories each contain the following two directories:
56+
57+
- `default/` : These queries only work on the default graph`*`
58+
59+
> `*` the handling of these queries will be dependent upon which Triplestore you are using and how it was configured. Some Triplestores will only use
60+
> triples in the default graph, some will use all triples in all named graphs.
61+
62+
- `ngraphs/` : These queries only work on named graphs
63+
64+
65+
## Requirements
66+
67+
These migration scripts are meant to be run using the open source `onto_tool`
68+
program. Information about onto_tool and how to install it, is available at
69+
[github](https://github.com/semanticarts/ontology-toolkit).
70+
71+
## Run on local files
72+
73+
> Note: A known limitation of `onto_tool` is that it does not support named graphs when run against local files. It does support named graphs when run against a SPARQL endpoint.
74+
75+
1. Starting in this directory, put source data files in the `./input/` directory (the sample data can be removed). See the onto_tool documentation for which file formats are supported.
76+
77+
> The `migrate_local.yaml` file currently looks in the `./input/` directory for `*.owl` and `*.ttl` files. If other formats are supported and you plan to use them, you will need to update the yaml.
78+
79+
2. Execute the following command:
80+
81+
```shell
82+
onto_tool bundle migrate_local.yaml
83+
```
84+
85+
3. `onto_tool` will output to STDOUT, you should see something like the following (this is the output from the sample data included in the `./input/` directory):
86+
87+
```
88+
INFO:root:Rename properties in default graph.
89+
INFO:root:Replace address classes in default graph.
90+
INFO:root:Replace properties with inverses in default graph.
91+
INFO:root:Use new gist:offers property in default graph (script 1).
92+
INFO:root:Use new gist:offers property in default graph (script 2).
93+
INFO:root:Rename properties in named graphs.
94+
INFO:root:Replace address classes in named graphs.
95+
INFO:root:Replace properties with inverses in named graphs.
96+
INFO:root:Use new gist:offers property in named graphs (script 1).
97+
INFO:root:Use new gist:offers property in named graphs (script 2).
98+
INFO:root:Check the default graph for other items that should be reviewed.
99+
INFO:root:Check named graphs for other items that should be reviewed.
100+
```
101+
102+
- Output files will be created in the `./output/` directory.
103+
104+
- Report files will be created in the `./reports/` directory.
105+
106+
## Run Against SPARQL Endpoint
107+
108+
You will need to tell `onto_tool` the URL of your SPARQL endpoint. You can do that
109+
by editing the `migration_endpoint.yaml` file, or you can put it into the command
110+
line execution like this:
111+
112+
> Note: Use your own values in place of `<...>`
113+
114+
```shell
115+
onto_tool bundle -v user <USER> -v password <PWD>
116+
-v endpoint <ENDPOINT-URI>
117+
[ -v update_endpoint <UPDATE-URI> ]
118+
-v report <REPORT-DIR> migrate_endpoint.yaml
119+
```
120+
121+
## Examples:
122+
123+
- renameProperties replaces `gist:isGeographicallyContainedIn` with `gist:isGeoContainedIn`
124+
- replaceAddressClasses replaces `gist:EmailAddress` with `gist:ElectronicAddress`
125+
- replaceAddressClasses replaces `gist:StreetAddress` with `gist:PhysicalAddress`
126+
127+
The different types and uses of addresses can be modeled as categories; see [the address migration documentation(AddressMigration.md) as well as the [full documentation of the new address model](/docs/model_documentation/AddressGuidance.md).
128+
129+
- replacePropertiesWithInverses replaces `gist:hasPart` with `gist:isPartOf`
130+
- useOffersProperty (scripts 1 and 2) replaces `gist:hasPart` with `gist:offers` in the context of an offer
131+
- useOffersProperty (scripts 3 and 4) replaces `gist:isPartOf` with `gist:offers` in the context of an offer
132+
133+
When converting data about offers, the offer and any sub-offer must belong to a the `gist:Offer` class or a subclass of it for the script to recognize the thing being offered.
134+
135+
The examples above are not a complete description of each script; read the scripts to see the complete list of changes they make to the data.
136+
137+
## Notes
138+
139+
- read the scripts to see the complete list of changes they make to the data (the examples above are not comprehensive).
140+
- renameProperties, replaceAddressClasses, and replacePropertiesWithInverses remove classes and properties that were deprecated in release 12, i.e. they perform data conversion necessary to conform to gist 13.
141+
- useOffersProperty is optional for conformance with gist 13
142+
- be sure to update all artifacts such as data ingestion, queries, forms, documentation, etc. along with converting the data
143+
- for data conversion of magnitudes and units of measure, see the README file in the uom_queries directory

‎migration/v13.0/migrate_endpoint.yaml

Lines changed: 140 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,140 @@
1+
jbundle: gist-v13.0-migration
2+
variables:
3+
action_queries_path_default: "./queries/action/default"
4+
action_queries_path_ngraphs: "./queries/action/ngraphs"
5+
reporting_queries_path_default: "./queries/report/default"
6+
reporting_queries_path_ngraphs: "./queries/report/ngraphs"
7+
endpoint: "<SPARQL_UPDATE_URI>"
8+
update_endpoint: "{endpoint}"
9+
user: "UPDATE_USER"
10+
password: "UPDATE_PASSWORD"
11+
report: "./reports"
12+
13+
actions:
14+
# default
15+
- action: "sparql"
16+
message: "Rename properties in default graph."
17+
query: "{action_queries_path_default}/renamePropertiesInDefaultGraph.rq"
18+
endpoint:
19+
query_uri: "{endpoint}"
20+
update_uri: "{update_endpoint}"
21+
user: "{user}"
22+
password: "{password}"
23+
24+
- action: "sparql"
25+
message: "Replace address classes in default graph."
26+
query: "{action_queries_path_default}/replaceAddressClassesInDefaultGraph.rq"
27+
endpoint:
28+
query_uri: "{endpoint}"
29+
update_uri: "{update_endpoint}"
30+
user: "{user}"
31+
password: "{password}"
32+
33+
- action: "sparql"
34+
message: "Replace properties with inverses in default graph."
35+
query: "{action_queries_path_default}/replacePropertiesWithInversesInDefaultGraph.rq"
36+
endpoint:
37+
query_uri: "{endpoint}"
38+
update_uri: "{update_endpoint}"
39+
user: "{user}"
40+
password: "{password}"
41+
42+
- action: "sparql"
43+
message: "Use new gist:offers property in default graph (script 1)."
44+
query: "{action_queries_path_default}/useOffersPropertyInDefaultGraph1.rq"
45+
endpoint:
46+
query_uri: "{endpoint}"
47+
update_uri: "{update_endpoint}"
48+
user: "{user}"
49+
password: "{password}"
50+
51+
- action: "sparql"
52+
message: "Use new gist:offers property in default graph (script 2)."
53+
query: "{action_queries_path_default}/useOffersPropertyInDefaultGraph2.rq"
54+
endpoint:
55+
query_uri: "{endpoint}"
56+
update_uri: "{update_endpoint}"
57+
user: "{user}"
58+
password: "{password}"
59+
60+
# ngraphs
61+
- action: "sparql"
62+
message: "Rename properties in named graphs."
63+
query: "{action_queries_path_ngraphs}/renamePropertiesInNamedGraphs.rq"
64+
endpoint:
65+
query_uri: "{endpoint}"
66+
update_uri: "{update_endpoint}"
67+
user: "{user}"
68+
password: "{password}"
69+
70+
- action: "sparql"
71+
message: "Replace address classes in named graphs."
72+
query: "{action_queries_path_ngraphs}/replaceAddressClassesInNamedGraphs.rq"
73+
endpoint:
74+
query_uri: "{endpoint}"
75+
update_uri: "{update_endpoint}"
76+
user: "{user}"
77+
password: "{password}"
78+
79+
- action: "sparql"
80+
message: "Replace properties with inverses in named graphs."
81+
query: "{action_queries_path_ngraphs}/replacePropertiesWithInversesInNamedGraphs.rq"
82+
endpoint:
83+
query_uri: "{endpoint}"
84+
update_uri: "{update_endpoint}"
85+
user: "{user}"
86+
password: "{password}"
87+
88+
- action: "sparql"
89+
message: "Use new gist:offers property in named graphs (script 1)."
90+
query: "{action_queries_path_ngraphs}/useOffersPropertyInNamedGraphs1.rq"
91+
endpoint:
92+
query_uri: "{endpoint}"
93+
update_uri: "{update_endpoint}"
94+
user: "{user}"
95+
password: "{password}"
96+
97+
- action: "sparql"
98+
message: "Use new gist:offers property in named graphs (script 2)."
99+
query: "{action_queries_path_ngraphs}/useOffersPropertyInNamedGraphs2.rq"
100+
endpoint:
101+
query_uri: "{endpoint}"
102+
update_uri: "{update_endpoint}"
103+
user: "{user}"
104+
password: "{password}"
105+
106+
# validation
107+
# default
108+
- action: "verify"
109+
message: "Check the default graph for other items that should be reviewed."
110+
type: "construct"
111+
inference: "none"
112+
stopOnFail: false
113+
target: "{report}"
114+
endpoint:
115+
query_uri: "{endpoint}"
116+
user: "{user}"
117+
password: "{password}"
118+
queries:
119+
source: "{reporting_queries_path_default}/"
120+
includes:
121+
- "detect_removed_default_graph.rq"
122+
- "detect_renamed_default_graph.rq"
123+
124+
# ngraphs
125+
- action: "verify"
126+
message: "Check named graphs for other items that should be reviewed."
127+
type: "construct"
128+
inference: "none"
129+
stopOnFail: false
130+
target: "{report}"
131+
endpoint:
132+
query_uri: "{endpoint}"
133+
user: "{user}"
134+
password: "{password}"
135+
queries:
136+
source: "{reporting_queries_path_ngraphs}/"
137+
includes:
138+
- "detect_removed_named_graphs.rq"
139+
- "detect_renamed_named_graphs.rq"
140+

‎migration/v13.0/migrate_local.yaml

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
bundle: gist-v13.0-migration
2+
variables:
3+
action_queries_path: "./queries/action/default"
4+
reporting_queries_path: "./queries/report/default"
5+
input: "./input"
6+
output: "./output"
7+
report: "./reports"
8+
9+
tools:
10+
- name: "rename-properties"
11+
type: "sparql"
12+
query: "{action_queries_path}/renamePropertiesInDefaultGraph.rq"
13+
- name: "replace-address-classes"
14+
type: "sparql"
15+
query: "{action_queries_path}/replaceAddressClassesInDefaultGraph.rq"
16+
- name: "replace-properties-with-inverses"
17+
type: "sparql"
18+
query: "{action_queries_path}/replacePropertiesWithInversesInDefaultGraph.rq"
19+
- name: "use-offers-property-script1"
20+
type: "sparql"
21+
query: "{action_queries_path}/UseOffersPropertyInDefaultGraph1.rq"
22+
- name: "use-offers-property-script2"
23+
type: "sparql"
24+
query: "{action_queries_path}/UseOffersPropertyInDefaultGraph2.rq"
25+
26+
actions:
27+
# transform actions
28+
# note: first transform is input -> output
29+
# following are output -> output
30+
- action: "transform"
31+
message: "Rename properties in default graph."
32+
tool: "rename-properties"
33+
source: "{input}"
34+
target: "{output}"
35+
includes:
36+
- "*.ttl"
37+
- "*.owl"
38+
- action: "transform"
39+
message: "Replace address classes in default graph."
40+
tool: "replace-address-classes"
41+
source: "{output}"
42+
target: "{output}"
43+
includes:
44+
- "*.ttl"
45+
- "*.owl"
46+
- action: "transform"
47+
message: "Replace properties with inverses in default graph."
48+
tool: "replace-properties-with-inverses"
49+
source: "{output}"
50+
target: "{output}"
51+
includes:
52+
- "*.ttl"
53+
- "*.owl"
54+
55+
- action: "transform"
56+
message: "Use new gist:offers property in default graph (script 1)."
57+
tool: "use-offers-property-script1"
58+
source: "{output}"
59+
target: "{output}"
60+
includes:
61+
- "*.ttl"
62+
- "*.owl"
63+
64+
- action: "transform"
65+
message: "Use new gist:offers property in default graph (script 2)."
66+
tool: "use-offers-property-script2"
67+
source: "{output}"
68+
target: "{output}"
69+
includes:
70+
- "*.ttl"
71+
- "*.owl"
72+
73+
# validation
74+
- action: "verify"
75+
message: "Check for issues that should be reviewed."
76+
type: "construct"
77+
inference: "none"
78+
stopOnFail: false
79+
source: "{output}"
80+
target: "{report}"
81+
includes:
82+
- "*.ttl"
83+
- "*.owl"
84+
queries:
85+
source: "{reporting_queries_path}/"
86+
includes:
87+
- "detect_removed_default_graph.rq"
88+
- "detect_renamed_default_graph.rq"

0 commit comments

Comments
 (0)