diff --git a/.gitignore b/.gitignore index 3b16863a..348360a5 100644 --- a/.gitignore +++ b/.gitignore @@ -2,4 +2,5 @@ *_webDownload/ .DS_Store *.orig +**/*venv catalog-v001.xml diff --git a/bundle.yaml b/bundle.yaml index 7ba3779c..b1a50a7c 100644 --- a/bundle.yaml +++ b/bundle.yaml @@ -290,6 +290,16 @@ actions: target: "{output}/migration/v13.0/" includes: - "*.md" +- action: "move" + message: "Patching documentation links in gist 13.0 HTML migration documentation." + source: "{output}/migration/v13.0/" + target: "{output}/migration/v13.0/" + includes: + - "*.html" + # Replaces ./*.md local links with ./*.html + replace: + from: "(?:docs|documentation)/(.*)\\.md" + to: "../../documentation/\\g<1>.html" - action: "move" message: "Patching local links in gist 13.0 HTML migration documentation." source: "{output}/migration/v13.0/" @@ -298,5 +308,6 @@ actions: - "*.html" # Replaces ./*.md local links with ./*.html replace: - from: "./(.*)\\.md" - to: "./\\g<1>.html" + from: "\\./(\\w*)\\.md" + to: "\\g<1>.html" + diff --git a/migration/v13.0/README.md b/migration/v13.0/README.md new file mode 100644 index 00000000..0f3f53b4 --- /dev/null +++ b/migration/v13.0/README.md @@ -0,0 +1,145 @@ +# Migrating to gist v13.0 from v12.0 + +This file gives a quick overview of using the migration utilities provided +by the gist team to help migrate from the previous version of gist. + +> See also additional documentation in the [Major Version Migration](../../docs/MajorVersionMigration.md). + +## Important Remarks + +- **These scripts are not cumulative; you must run the migrations for each version of gist + in sequence.** + +- **These utilities are a starting point.** You should review them before running + them to make sure you approve of the actions prior to using them. + +- **These utilities do not handle everything**, in some cases we can only warn you + about changes that were made in gist and you will need to decide how you want + to handle the changes. + +## Migration Queries + +All of our migration tools are SPARQL queries. + +The following directory structure holds the migration scripts: + + . + └── queries/ + │ + ├── action/ + │ ├── default/ + │ │ └── *.rq + │ └── ngraphs/ + │ └── *.rq + │ + ├── report/ + │ ├── default/ + │ │ └── *.rq + │ └── ngraphs/ + │ └── *.rq + │ + └── uom_queries/ + └── *.rq + +The `./queries/` directory contains the following directories: + +- `./queries/action/` : Update queries for migrating from Gist v12.0 to v13.0 + +- `./queries/report/` : Validation queries that return SHACL validation reports for data that does not comply with Gist v13.0 + +- `./queries/uom_queries/` : Queries for migrating to the new ontology for magnitudes, aspects, and units of measure + +The action and report directories each contain the following two directories: + +- `default/` : These queries only work on the default graph`*` + + > `*` the handling of these queries will be dependent upon which triplestore you are using and how it was configured. Some triplestores will only use + > triples in the default graph, some will use all triples in all named graphs. + +- `ngraphs/` : These queries only work on named graphs + + +## Requirements + +These migration scripts are meant to be run using the open source `onto_tool` +program. Information about onto_tool and how to install it, is available at +[PyPi](https://pypi.org/project/onto-tool). + +## Run on local files + +> Note: A known limitation of `onto_tool` is that it does not support named graphs when run against local files. It does support named graphs when run against a SPARQL endpoint. + +1. Starting in this directory, either place source data files in the `./input/` directory or point the script at the location of your data via `-v input` flag (see example below). See the onto_tool documentation for which file formats are supported. + + > * The `migrate_local.yaml` file currently looks in the input directory for `*.owl` and `*.ttl` files. If other formats are supported and you plan to use them, you will need to update the yaml. + > * The script will only transform RDF data files at the top level of the input directory, and will not recursively descend into sub-directories. Invoke the script multiple times with different directories specified via the `-v input` flag as needed. + +2. Execute the following command for the default `./input` director: + + ```shell + onto_tool bundle migrate_local.yaml + ``` + + or with a custom input directory: + + ```shell + onto_tool bundle -v input migrate_local.yaml + ``` + +3. `onto_tool` will output to STDOUT, you should see something like the following: + + INFO:root:Rename properties in named graphs. + INFO:root:Replace address classes in named graphs. + INFO:root:Replace properties with inverses in named graphs. + INFO:root:Use new gist:offers property in named graphs (script 1). + INFO:root:Use new gist:offers property in named graphs (script 2). + INFO:root:Rename properties in named graphs. + INFO:root:Replace address classes in named graphs. + INFO:root:Replace properties with inverses in named graphs. + INFO:root:Use new gist:offers property in named graphs (script 1). + INFO:root:Use new gist:offers property in named graphs (script 2). + INFO:root:Check the named graphs for other items that should be reviewed. + INFO:root:Check named graphs for other items that should be reviewed. + + - Output files will be created in the `./output/` directory. The location can be overriden by specifying `-v output `. + + - Report files will be created in the `./reports/` directory. The location can be overriden by specifying `-v report `. + +## Run Against SPARQL Endpoint + +You will need to tell `onto_tool` the URL of your SPARQL endpoint. You can do that +by editing the `migration_endpoint.yaml` file, or you can put it into the command +line execution like this: + +> Note: Use your own values in place of `<...>` + +```shell +onto_tool bundle -v user -v password + -v endpoint + [ -v update_endpoint ] + -v report migrate_endpoint.yaml +``` + +## Examples: + +- renameProperties replaces `gist:isGeographicallyContainedIn` with `gist:isGeoContainedIn` +- replaceAddressClasses replaces `gist:EmailAddress` with `gist:ElectronicAddress` +- replaceAddressClasses replaces `gist:StreetAddress` with `gist:PhysicalAddress` + +The different types and uses of addresses can be modeled as categories; see [the address migration documentation](./AddressMigration.md) as well as the [full documentation of the new address model](../../docs/models/AddressGuidance.md). + +- replacePropertiesWithInverses replaces `gist:hasPart` with `gist:isPartOf` +- useOffersProperty (scripts 1 and 2) replaces `gist:hasPart` with `gist:offers` in the context of an offer +- useOffersProperty (scripts 3 and 4) replaces `gist:isPartOf` with `gist:offers` in the context of an offer + +When converting data about offers, the offer and any sub-offer must belong to a the `gist:Offer` class or a subclass of it for the script to recognize the thing being offered. + +The examples above are not a complete description of each script; read the scripts to see the complete list of changes they make to the data. + +## Notes + +- read the scripts to see the complete list of changes they make to the data (the examples above are not comprehensive). +- renameProperties, replaceAddressClasses, and replacePropertiesWithInverses remove classes and properties that were deprecated in release 12, i.e. they perform data conversion necessary to conform to gist 13. +- useOffersProperty is optional for conformance with gist 13 +- be sure to update all artifacts such as data ingestion, queries, forms, documentation, etc. along with converting the data +- for data conversion of magnitudes and units of measure, see the README file in the uom_queries directory diff --git a/migration/v13.0/migrate_endpoint.yaml b/migration/v13.0/migrate_endpoint.yaml new file mode 100644 index 00000000..17dab2e6 --- /dev/null +++ b/migration/v13.0/migrate_endpoint.yaml @@ -0,0 +1,140 @@ +bundle: gist-v13.0-migration +variables: + action_queries_path_default: "./queries/action/default" + action_queries_path_ngraphs: "./queries/action/ngraphs" + reporting_queries_path_default: "./queries/report/default" + reporting_queries_path_ngraphs: "./queries/report/ngraphs" + endpoint: "" + update_endpoint: "{endpoint}" + user: "UPDATE_USER" + password: "UPDATE_PASSWORD" + report: "./reports" + +actions: + # default + - action: "sparql" + message: "Rename properties in default graph." + query: "{action_queries_path_default}/renamePropertiesInDefaultGraph.rq" + endpoint: + query_uri: "{endpoint}" + update_uri: "{update_endpoint}" + user: "{user}" + password: "{password}" + + - action: "sparql" + message: "Replace address classes in default graph." + query: "{action_queries_path_default}/replaceAddressClassesInDefaultGraph.rq" + endpoint: + query_uri: "{endpoint}" + update_uri: "{update_endpoint}" + user: "{user}" + password: "{password}" + + - action: "sparql" + message: "Replace properties with inverses in default graph." + query: "{action_queries_path_default}/replacePropertiesWithInversesInDefaultGraph.rq" + endpoint: + query_uri: "{endpoint}" + update_uri: "{update_endpoint}" + user: "{user}" + password: "{password}" + + - action: "sparql" + message: "Use new gist:offers property in default graph (script 1)." + query: "{action_queries_path_default}/useOffersPropertyInDefaultGraph1.rq" + endpoint: + query_uri: "{endpoint}" + update_uri: "{update_endpoint}" + user: "{user}" + password: "{password}" + + - action: "sparql" + message: "Use new gist:offers property in default graph (script 2)." + query: "{action_queries_path_default}/useOffersPropertyInDefaultGraph2.rq" + endpoint: + query_uri: "{endpoint}" + update_uri: "{update_endpoint}" + user: "{user}" + password: "{password}" + + # ngraphs + - action: "sparql" + message: "Rename properties in named graphs." + query: "{action_queries_path_ngraphs}/renamePropertiesInNamedGraphs.rq" + endpoint: + query_uri: "{endpoint}" + update_uri: "{update_endpoint}" + user: "{user}" + password: "{password}" + + - action: "sparql" + message: "Replace address classes in named graphs." + query: "{action_queries_path_ngraphs}/replaceAddressClassesInNamedGraphs.rq" + endpoint: + query_uri: "{endpoint}" + update_uri: "{update_endpoint}" + user: "{user}" + password: "{password}" + + - action: "sparql" + message: "Replace properties with inverses in named graphs." + query: "{action_queries_path_ngraphs}/replacePropertiesWithInversesInNamedGraphs.rq" + endpoint: + query_uri: "{endpoint}" + update_uri: "{update_endpoint}" + user: "{user}" + password: "{password}" + + - action: "sparql" + message: "Use new gist:offers property in named graphs (script 1)." + query: "{action_queries_path_ngraphs}/useOffersPropertyInNamedGraphs1.rq" + endpoint: + query_uri: "{endpoint}" + update_uri: "{update_endpoint}" + user: "{user}" + password: "{password}" + + - action: "sparql" + message: "Use new gist:offers property in named graphs (script 2)." + query: "{action_queries_path_ngraphs}/useOffersPropertyInNamedGraphs2.rq" + endpoint: + query_uri: "{endpoint}" + update_uri: "{update_endpoint}" + user: "{user}" + password: "{password}" + + # validation + # default + - action: "verify" + message: "Check the default graph for other items that should be reviewed." + type: "construct" + inference: "none" + stopOnFail: false + target: "{report}" + endpoint: + query_uri: "{endpoint}" + user: "{user}" + password: "{password}" + queries: + source: "{reporting_queries_path_default}/" + includes: + - "detect_removed_default_graph.rq" + - "detect_renamed_default_graph.rq" + + # ngraphs + - action: "verify" + message: "Check named graphs for other items that should be reviewed." + type: "construct" + inference: "none" + stopOnFail: false + target: "{report}" + endpoint: + query_uri: "{endpoint}" + user: "{user}" + password: "{password}" + queries: + source: "{reporting_queries_path_ngraphs}/" + includes: + - "detect_removed_named_graphs.rq" + - "detect_renamed_named_graphs.rq" + diff --git a/migration/v13.0/migrate_local.yaml b/migration/v13.0/migrate_local.yaml new file mode 100644 index 00000000..373963d1 --- /dev/null +++ b/migration/v13.0/migrate_local.yaml @@ -0,0 +1,88 @@ +bundle: gist-v13.0-migration +variables: + action_queries_path: "./queries/action/default" + reporting_queries_path: "./queries/report/default" + input: "./input" + output: "./output" + report: "./reports" + +tools: + - name: "rename-properties" + type: "sparql" + query: "{action_queries_path}/renamePropertiesInDefaultGraph.rq" + - name: "replace-address-classes" + type: "sparql" + query: "{action_queries_path}/replaceAddressClassesInDefaultGraph.rq" + - name: "replace-properties-with-inverses" + type: "sparql" + query: "{action_queries_path}/replacePropertiesWithInversesInDefaultGraph.rq" + - name: "use-offers-property-script1" + type: "sparql" + query: "{action_queries_path}/useOffersPropertyInDefaultGraph1.rq" + - name: "use-offers-property-script2" + type: "sparql" + query: "{action_queries_path}/useOffersPropertyInDefaultGraph2.rq" + +actions: + # transform actions + # note: first transform is input -> output + # following are output -> output + - action: "transform" + message: "Rename properties in default graph." + tool: "rename-properties" + source: "{input}" + target: "{output}" + includes: + - "*.ttl" + - "*.owl" + - action: "transform" + message: "Replace address classes in default graph." + tool: "replace-address-classes" + source: "{output}" + target: "{output}" + includes: + - "*.ttl" + - "*.owl" + - action: "transform" + message: "Replace properties with inverses in default graph." + tool: "replace-properties-with-inverses" + source: "{output}" + target: "{output}" + includes: + - "*.ttl" + - "*.owl" + + - action: "transform" + message: "Use new gist:offers property in default graph (script 1)." + tool: "use-offers-property-script1" + source: "{output}" + target: "{output}" + includes: + - "*.ttl" + - "*.owl" + + - action: "transform" + message: "Use new gist:offers property in default graph (script 2)." + tool: "use-offers-property-script2" + source: "{output}" + target: "{output}" + includes: + - "*.ttl" + - "*.owl" + + # validation + - action: "verify" + message: "Check for issues that should be reviewed." + type: "construct" + inference: "none" + stopOnFail: false + source: "{output}" + target: "{report}" + includes: + - "*.ttl" + - "*.owl" + queries: + source: "{reporting_queries_path}/" + includes: + - "detect_removed_default_graph.rq" + - "detect_renamed_default_graph.rq" diff --git a/migration/v13.0/queries/action/default/renamePropertiesInDefaultGraph.rq b/migration/v13.0/queries/action/default/renamePropertiesInDefaultGraph.rq new file mode 100644 index 00000000..848af9f3 --- /dev/null +++ b/migration/v13.0/queries/action/default/renamePropertiesInDefaultGraph.rq @@ -0,0 +1,37 @@ +# rename properties in the default graph +# CAUTION: complete all replacements in named graphs before doing this + +prefix gist: + +# test +select ?g ?s ?o ?oldProperty ?newProperty + +# update graph +# delete {?s ?oldProperty ?o .} +# insert {?s ?newProperty ?o .} + +where { + +# input + + values (?oldProperty ?newProperty) { + + (gist:isGeographicallyContainedIn gist:isGeoContainedIn) + (gist:occupiesGeographically gist:hasPhysicalLocation) + (gist:occupiesGeographicallyPermanently gist:hasPhysicalLocation) + + (gist:hasCommunicationAddress gist:hasAddress) + + (gist:hasSuperCategory gist:hasBroader) + (gist:hasDirectSuperCategory gist:hasDirectBroader) + (gist:hasUniqueSuperCategory gist:hasUniqueBroader) + } + +# triples in the default graph that include the property, and are not in any named graph + + {?s ?oldProperty ?o.} + minus + {graph ?g {?s ?oldProperty ?o.}} + +} + diff --git a/migration/v13.0/queries/action/default/replaceAddressClassesInDefaultGraph.rq b/migration/v13.0/queries/action/default/replaceAddressClassesInDefaultGraph.rq new file mode 100644 index 00000000..0227ef92 --- /dev/null +++ b/migration/v13.0/queries/action/default/replaceAddressClassesInDefaultGraph.rq @@ -0,0 +1,31 @@ +# replace address classes in the default graph +# note: also create your own categories for the different types and uses of addresses + +prefix gist: + +# test +select ?s ?oldClass ?newClass ?g # ?g should be empty + +# update graph +# delete {?s rdf:type ?oldClass .} +# insert {?s rdf:type ?newClass .} + + +where { + + values (?oldClass ?newClass) { + (gist:ElectronicMessageAddress gist:ElectronicAddress) + (gist:EmailAddress gist:ElectronicAddress) + (gist:TelephoneNumber gist:ElectronicAddress) + (gist:StreetAddress gist:PhysicalAddress) + (gist:PostalAddress gist:PhysicalAddress) + } + +# triples in the default graph that have old class as type, and are not duplicated in a named graph + {?s rdf:type ?oldClass .} + minus + {graph ?g {?s rdf:type ?oldClass .}} + +} + + diff --git a/migration/v13.0/queries/action/default/replacePropertiesWithInversesInDefaultGraph.rq b/migration/v13.0/queries/action/default/replacePropertiesWithInversesInDefaultGraph.rq new file mode 100644 index 00000000..f2c2291f --- /dev/null +++ b/migration/v13.0/queries/action/default/replacePropertiesWithInversesInDefaultGraph.rq @@ -0,0 +1,34 @@ +# replace properties with their inverses in the default graph +# CAUTION: complete all replacements in named graphs before doing this + +prefix gist: + +# test +select ?s ?o ?property ?inverse + +# update graph +# delete {?s ?property ?o .} +# insert {?o ?inverse ?s .} + +where { + +# input + + values (?property ?inverse) { + + (gist:hasPart gist:isPartOf) + (gist:hasDirectPart gist:isDirectPartOf) + + (gist:hasMember gist:isMemberOf) + (gist:hasFirstMember gist:isFirstMemberOf) + } + +# triples in the default graph that include the property, and are not duplicated in a named graph + + {?s ?property ?o.} + + minus + + {graph ?g {?s ?property ?o.}} + +} diff --git a/migration/v13.0/queries/action/default/useOffersPropertyInDefaultGraph1.rq b/migration/v13.0/queries/action/default/useOffersPropertyInDefaultGraph1.rq new file mode 100644 index 00000000..dcde6220 --- /dev/null +++ b/migration/v13.0/queries/action/default/useOffersPropertyInDefaultGraph1.rq @@ -0,0 +1,36 @@ +# use gist:offers in the default graph to relate an offer to the thing being offered +# in this script, gist:offers will replace gist:hasPart +# be sure all instances returned by test relate an offer to the THING OFFERED +# CAUTION: complete all replacements in named graphs before doing this + +prefix gist: + +# test +select ?offer ?thingOffered + +# update graph +# delete {?offer gist:hasPart ?thingOffered .} +# insert {?offer gist:offers ?thingOffered .} + +where { + +# triples in in the default graph that relate an offer to the thing offered, and are not duplicated in a named graph +# filters out "sub-offers" + + { + ?offer rdf:type/rdfs:subClassOf* gist:Offer ; + gist:hasPart ?offerPart . + + filter not exists {?offerPart rdf:type/rdfs:subClassOf* gist:Offer .} + } + minus + + {graph ?g { + + ?offer rdf:type/rdfs:subClassOf* gist:Offer ; + gist:hasPart ?offerPart . + }} + + bind(?offerPart as ?thingOffered) +} + diff --git a/migration/v13.0/queries/action/default/useOffersPropertyInDefaultGraph2.rq b/migration/v13.0/queries/action/default/useOffersPropertyInDefaultGraph2.rq new file mode 100644 index 00000000..438b782b --- /dev/null +++ b/migration/v13.0/queries/action/default/useOffersPropertyInDefaultGraph2.rq @@ -0,0 +1,40 @@ +# use gist:offers in the default graph to relate an offer to the thing being offered +# in this script, gist:offers will replace gist:isPartOf +# be sure all instances returned by test relate an offer to the THING OFFERED +# CAUTION: complete all replacements in named graphs before doing this + +prefix gist: + +# test +select ?g ?offer ?thingOffered + +# update graph +# delete {?thingOffered gist:isPartOf ?offer .} +# insert {?offer gist:offers ?thingOffered .} + +where { + +# triples in the default graph that relate an offer to the thing offered, and are not duplicated in a named graph +# filters out "sub-offers" + + { + ?offer rdf:type/rdfs:subClassOf* gist:Offer . + + ?offerPart gist:isPartOf ?offer . + + filter not exists {?offerPart rdf:type/rdfs:subClassOf* gist:Offer .} + } + + minus + + {graph ?g { + + ?offer rdf:type/rdfs:subClassOf* gist:Offer . + + ?offerPart gist:isPartOf ?offer . + }} + + bind(?offerPart as ?thingOffered) + +} + diff --git a/migration/v13.0/queries/action/ngraphs/renamePropertiesInNamedGraphs.rq b/migration/v13.0/queries/action/ngraphs/renamePropertiesInNamedGraphs.rq new file mode 100644 index 00000000..d9f0eb8d --- /dev/null +++ b/migration/v13.0/queries/action/ngraphs/renamePropertiesInNamedGraphs.rq @@ -0,0 +1,34 @@ +# rename properties in named graphs + +prefix gist: + +# test +select ?g ?s ?o ?oldProperty ?newProperty + +# update graph +# delete {graph ?g {?s ?oldProperty ?o .}} +# insert {graph ?g {?s ?newProperty ?o .}} + +where { + +# input + + values (?oldProperty ?newProperty) { + + (gist:isGeographicallyContainedIn gist:isGeoContainedIn) + (gist:occupiesGeographically gist:hasPhysicalLocation) + (gist:occupiesGeographicallyPermanently gist:hasPhysicalLocation) + + (gist:hasCommunicationAddress gist:hasAddress) + + (gist:hasSuperCategory gist:hasBroader) + (gist:hasDirectSuperCategory gist:hasDirectBroader) + (gist:hasUniqueSuperCategory gist:hasUniqueBroader) + } + +# triples in named graphs that include the property + + {graph ?g {?s ?oldProperty ?o.}} + +} + diff --git a/migration/v13.0/queries/action/ngraphs/replaceAddressClassesInNamedGraphs.rq b/migration/v13.0/queries/action/ngraphs/replaceAddressClassesInNamedGraphs.rq new file mode 100644 index 00000000..e000a3fe --- /dev/null +++ b/migration/v13.0/queries/action/ngraphs/replaceAddressClassesInNamedGraphs.rq @@ -0,0 +1,30 @@ +# replace address classes in named graphs +# note: also create your own categories for the different types and uses of addresses + +prefix gist: + +# test +select ?g ?s ?oldClass ?newClass + +# update graph +# delete {graph ?g {?s rdf:type ?oldClass .}} +# insert {graph ?g {?s rdf:type ?newClass .}} + + +where { + + values (?oldClass ?newClass) { + (gist:ElectronicMessageAddress gist:ElectronicAddress) + (gist:EmailAddress gist:ElectronicAddress) + (gist:TelephoneNumber gist:ElectronicAddress) + (gist:StreetAddress gist:PhysicalAddress) + (gist:PostalAddress gist:PhysicalAddress) + } + +# triples in named graphs + + graph ?g {?s rdf:type ?oldClass} + +} + + diff --git a/migration/v13.0/queries/action/ngraphs/replacePropertiesWithInversesInNamedGraphs.rq b/migration/v13.0/queries/action/ngraphs/replacePropertiesWithInversesInNamedGraphs.rq new file mode 100644 index 00000000..0604eefd --- /dev/null +++ b/migration/v13.0/queries/action/ngraphs/replacePropertiesWithInversesInNamedGraphs.rq @@ -0,0 +1,30 @@ +# replace properties with their inverses in named graphs + +prefix gist: + +# test +select ?g ?s ?o ?property ?inverse + +# update graph +# delete {graph ?g {?s ?property ?o .}} +# insert {graph ?g {?o ?inverse ?s .}} + +where { + +# input + + values (?property ?inverse) { + + (gist:hasPart gist:isPartOf) + (gist:hasDirectPart gist:isDirectPartOf) + + (gist:hasMember gist:isMemberOf) + (gist:hasFirstMember gist:isFirstMemberOf) + } + +# triples in named graphs that include the property + + {graph ?g {?s ?property ?o.}} + +} + diff --git a/migration/v13.0/queries/action/ngraphs/useOffersPropertyInNamedGraphs1.rq b/migration/v13.0/queries/action/ngraphs/useOffersPropertyInNamedGraphs1.rq new file mode 100644 index 00000000..b5c9292d --- /dev/null +++ b/migration/v13.0/queries/action/ngraphs/useOffersPropertyInNamedGraphs1.rq @@ -0,0 +1,28 @@ +# use gist:offers in named graphs to relate an offer to the thing being offered +# in this script, gist:offers will replace gist:hasPart +# be sure all instances returned by test relate an offer to the THING OFFERED + +prefix gist: + +# test +select ?g ?offer ?thingOffered + +# update graph +# delete {graph ?g {?offer gist:hasPart ?thingOffered .}} +# insert {graph ?g {?offer gist:offers ?thingOffered .}} + +where { + +# triples in named graphs that relate an offer to the thing offered +# filters out "sub-offers" + + {graph ?g { + + ?offer rdf:type/rdfs:subClassOf* gist:Offer ; + gist:hasPart ?offerPart . + + filter not exists {?offerPart rdf:type/rdfs:subClassOf* gist:Offer .} + + bind(?offerPart as ?thingOffered) + }} +} diff --git a/migration/v13.0/queries/action/ngraphs/useOffersPropertyInNamedGraphs2.rq b/migration/v13.0/queries/action/ngraphs/useOffersPropertyInNamedGraphs2.rq new file mode 100644 index 00000000..ca3b51d0 --- /dev/null +++ b/migration/v13.0/queries/action/ngraphs/useOffersPropertyInNamedGraphs2.rq @@ -0,0 +1,31 @@ +# use gist:offers in named graphs to relate an offer to the thing being offered +# in this script, gist:offers will replace gist:isPartOf +# be sure all instances returned by test relate an offer to the THING OFFERED + +prefix gist: + +# test +select ?g ?offer ?thingOffered + +# update graph + +# delete {graph ?g {?thingOffered gist:isPartOf ?offer .}} + +# insert {graph ?g {?offer gist:offers ?thingOffered .}} + +where { + +# triples in named graphs that relate an offer to the thing offered +# filters out "sub-offers" + + {graph ?g { + + ?offer rdf:type/rdfs:subClassOf* gist:Offer . + + ?offerPart gist:isPartOf ?offer . + + filter not exists {?offerPart rdf:type/rdfs:subClassOf* gist:Offer .} + + bind(?offerPart as ?thingOffered) + }} +} diff --git a/migration/v13.0/queries/report/default/detect_removed_default_graph.rq b/migration/v13.0/queries/report/default/detect_removed_default_graph.rq new file mode 100644 index 00000000..8e8e1e89 --- /dev/null +++ b/migration/v13.0/queries/report/default/detect_removed_default_graph.rq @@ -0,0 +1,47 @@ +prefix sh: +prefix gist: + +CONSTRUCT { +?report a sh:ValidationReport ; + sh:conforms false ; + sh:result + [ + a sh:ValidationResult ; + sh:focusNode ?deleted ; + sh:resultMessage ?error ; + sh:resultSeverity sh:Warning ; + sh:sourceConstraintComponent + ] . +} +WHERE { + VALUES ?deleted { + + # Issue 462-Remove-Sensor-Related-Items + gist:Actuator + gist:Controller + gist:ControllerType + gist:MessageDefinition + gist:PhenomenaType + gist:Sensor + gist:directs + gist:hasViableRange + + # Issue 947 + gist:unitSymbolHtml + gist:unitSymbolUnicode + + # Issue 947 + gist:isCharacterizedAs + } + { + ?s ?deleted ?o + } UNION { + ?s ?p ?deleted + } UNION { + ?deleted ?p ?o + } + BIND( as ?report) + BIND(CONCAT("Removed entity gist:", + REPLACE(STR(?deleted), '^.*[/#]', ''), " referenced in data. ", + "Review release notes for information about the changes.") as ?error) +} diff --git a/migration/v13.0/queries/report/default/detect_renamed_default_graph.rq b/migration/v13.0/queries/report/default/detect_renamed_default_graph.rq new file mode 100644 index 00000000..7074a0b5 --- /dev/null +++ b/migration/v13.0/queries/report/default/detect_renamed_default_graph.rq @@ -0,0 +1,64 @@ +prefix sh: +prefix gist: + +CONSTRUCT { + ?report a sh:ValidationReport ; + sh:conforms false ; + sh:result + [ + a sh:ValidationResult ; + sh:focusNode ?renamed ; + sh:resultMessage ?error ; + sh:resultSeverity sh:Warning ; + sh:sourceConstraintComponent + ] + . +} +WHERE { + { + select distinct ?renamed ?newName where { + values (?renamed ?newName) { + + # Issue 1070 inverses + (gist:hasDirectPart gist:isDirectPartOf) + (gist:hasFirstMember gist:isFirstMemberOf) + (gist:hasMember gist:isMemberOf) + (gist:hasPart gist:isPartOf) + + # Issue 1115 deprecate hasSuperCategory + (gist:hasSuperCategory gist:hasBroader) + (gist:hasDirectSuperCategory gist:hasDirectBroader) + (gist:hasUniqueSuperCategory gist:hasUniqueBroader) + + # Issues 809 and 812 geo properties + (gist:occupiesGeographically gist:hasPhysicalLocation) + (gist:occupiesGeographicallyPermanently gist:hasPhysicalLocation) + (gist:isGeographicallyContainedIn gist:isGeoContainedIn) + + # Issue 947 deprecated in prior release + (gist:tagText gist:containedText) + (gist:unitSymbol gist:symbol) + + # PR 1106 refactor addresses + (gist:ElectronicMessageAddress gist:ElectronicAddress) + (gist:EmailAddress gist:ElectronicAddress) + (gist:TelephoneNumber gist:ElectronicAddress) + (gist:PostalAddress gist:PhysicalAddress) + (gist:StreetAddress gist:PhysicalAddress) + (gist:hasCommunicationAddress gist:hasAddress) + } + { + ?s ?renamed ?o + } UNION { + ?s ?p ?renamed + } UNION { + ?renamed ?p ?o + } + } + } + BIND( as ?report) + BIND(CONCAT("WARNING: detected use of a class name that has changed - gist:", + REPLACE(STR(?renamed), '^.*[/#]', ''), " was changed to gist:", + REPLACE(STR(?newName), '^.*[/#]', '') + ) as ?error) +} diff --git a/migration/v13.0/queries/report/ngraphs/detect_removed_named_graphs.rq b/migration/v13.0/queries/report/ngraphs/detect_removed_named_graphs.rq new file mode 100644 index 00000000..60f0614c --- /dev/null +++ b/migration/v13.0/queries/report/ngraphs/detect_removed_named_graphs.rq @@ -0,0 +1,52 @@ +prefix sh: +prefix gist: + +CONSTRUCT { +?report a sh:ValidationReport ; + sh:conforms false ; + sh:result + [ + a sh:ValidationResult ; + sh:focusNode ?deleted ; + sh:resultMessage ?error ; + sh:resultSeverity sh:Warning ; + sh:sourceConstraintComponent + ] . +} +WHERE { + { + select distinct ?deleted where { + values ?deleted { + + # Issue 462-Remove-Sensor-Related-Items + gist:Actuator + gist:Controller + gist:ControllerType + gist:MessageDefinition + gist:PhenomenaType + gist:Sensor + gist:directs + gist:hasViableRange + + # Issue 947 + gist:unitSymbolHtml + gist:unitSymbolUnicode + + # Issue 947 + gist:isCharacterizedAs + + } + { + graph ?g { ?s ?deleted ?o } + } UNION { + graph ?g { ?s ?p ?deleted } + } UNION { + graph ?g { ?deleted ?p ?o } + } + } + } + BIND( as ?report) + BIND(CONCAT("Removed entity gist:", + REPLACE(STR(?deleted), '^.*[/#]', ''), " referenced in data. ", + "Review release notes for information about the changes.") as ?error) +} diff --git a/migration/v13.0/queries/report/ngraphs/detect_renamed_named_graphs.rq b/migration/v13.0/queries/report/ngraphs/detect_renamed_named_graphs.rq new file mode 100644 index 00000000..16533a88 --- /dev/null +++ b/migration/v13.0/queries/report/ngraphs/detect_renamed_named_graphs.rq @@ -0,0 +1,65 @@ +prefix sh: +prefix gist: + +CONSTRUCT { + ?report a sh:ValidationReport ; + sh:conforms false ; + sh:result + [ + a sh:ValidationResult ; + sh:focusNode ?renamed ; + sh:resultMessage ?error ; + sh:resultSeverity sh:Warning ; + sh:sourceConstraintComponent + ] + . +} +WHERE { + { + select distinct ?renamed ?newName where { + values (?renamed ?newName) { + + # Issue 1070 inverses + (gist:hasDirectPart gist:isDirectPartOf) + (gist:hasFirstMember gist:isFirstMemberOf) + (gist:hasMember gist:isMemberOf) + (gist:hasPart gist:isPartOf) + + # Issue 1115 deprecate hasSuperCategory + (gist:hasSuperCategory gist:hasBroader) + (gist:hasDirectSuperCategory gist:hasDirectBroader) + (gist:hasUniqueSuperCategory gist:hasUniqueBroader) + + # Issues 809 and 812 geo properties + (gist:occupiesGeographically gist:hasPhysicalLocation) + (gist:occupiesGeographicallyPermanently gist:hasPhysicalLocation) + (gist:isGeographicallyContainedIn gist:isGeoContainedIn) + + + # Issue 947 deprecated in previous release + (gist:tagText gist:containedText) + (gist:unitSymbol gist:symbol) + + # PR 1106 refactor addresses + (gist:ElectronicMessageAddress gist:ElectronicAddress) + (gist:EmailAddress gist:ElectronicAddress) + (gist:TelephoneNumber gist:ElectronicAddress) + (gist:PostalAddress gist:PhysicalAddress) + (gist:StreetAddress gist:PhysicalAddress) + (gist:hasCommunicationAddress gist:hasAddress) + } + { + graph ?g { ?s ?renamed ?o } + } UNION { + graph ?g { ?s ?p ?renamed } + } UNION { + graph ?g { ?renamed ?p ?o } + } + } + } + BIND( as ?report) + BIND(CONCAT("WARNING: detected use of a class name that has changed - gist:", + REPLACE(STR(?renamed), '^.*[/#]', ''), " was changed to gist:", + REPLACE(STR(?newName), '^.*[/#]', '') + ) as ?error) +} diff --git a/migration/v13.0/queries/uom_queries/README.md b/migration/v13.0/queries/uom_queries/README.md new file mode 100644 index 00000000..af26d76b --- /dev/null +++ b/migration/v13.0/queries/uom_queries/README.md @@ -0,0 +1,14 @@ +# Unit of Measure data migration queries + +Follow the guidance in the slide deck uomDataMigration.pptx in the migration/v13.0 directory. + +There is a set of queries to explore the existing data, and other queries to perform the migration. + +Most of the data migration queries have to be customized per-Enterprise. + +Since the conversion steps are rather extensive, you might want to run them in stages. + +If desired, you can create a yaml file for each stage of your data migraion (using the existing yaml files for non-uom files as a guide). + +The data migration queries are all designed to migrate data in named graphs; modify the queries as needed to make changes in the default graph. + diff --git a/migration/v13.0/uomDataConversion.pptx b/migration/v13.0/uomDataConversion.pptx index 56245af8..86657bf4 100644 Binary files a/migration/v13.0/uomDataConversion.pptx and b/migration/v13.0/uomDataConversion.pptx differ