Skip to content

Commit

Permalink
merge sparql endpoints (#127)
Browse files Browse the repository at this point in the history 
* merge sparql endpoints
update sparql functions
update tests
add otherAnnotationProps to annotations
remove startup graph posting

* fix tests - rdflib's query can't handle (valid) empty VALUES statements in SPARQL queries, so exclude from the query if there are no terms.

* Update SPARQL_ENDPOINT For other tests

* Update doco
Remove sequence path for sh:color
Upudate env template

* Fix object endpoint
Remove relative properties functionality - path/sequence path functionality has been extended to listing queries

* Treat skos:narrower and skos:broader as annotation properties.

* Add back in relative properties - using other annotations for broader/narrower is a bit blunt. It includes a lot of unintended annotations as it applies at the profile level. The only other option would be to interpret a combination of sequence and inverse path as concepts are skos:inScheme the focus node (inverse path), and the additional properties we want on them (broader, narrower) are two steps away from the focus node. This is trivial in SPARQL but probably requires more careful design in SHACL.

* Update dockerfile / dev setup

* Update dockerfile / dev setup

* Add/fix caching tests and add to GH action.

* Remove "Generic Prez" and other unncessary prez flavour specification from profiles queries/functions.

* Change reference from SpacePrez to Prez where appropriate

* For SPARQL endpoint pass on headers except 'host'

* Remove unused "GenericPrez" from profiles.py

* Fix README-Dev.md table markdown
  • Loading branch information
@recalcitrantsupplant
recalcitrantsupplant authored Jun 22, 2023
1 parent 4f0a3a1 commit cc2b9d2
Show file tree
Hide file tree
Showing 55 changed files with 608 additions and 969 deletions.
24 changes: 4 additions & 20 deletions .env-template
View file
Original file line number Diff line number Diff line change
@@ -1,20 +1,4 @@
VOCPREZ_SPARQL_ENDPOINT=
SPACEPREZ_SPARQL_ENDPOINT=
CATPREZ_SPARQL_ENDPOINT=

VOCPREZ_SPARQL_UPDATE=
SPACEPREZ_SPARQL_UPDATE=
CATPREZ_SPARQL_UPDATE=

VOCPREZ_SPARQL_HTTP_STORE=
SPACEPREZ_SPARQL_HTTP_STORE=
CATPREZ_SPARQL_HTTP_STORE=

VOCPREZ_SPARQL_USERNAME=
VOCPREZ_SPARQL_PASSWORD=
SPACEPREZ_SPARQL_USERNAME=
SPACEPREZ_SPARQL_PASSWORD=
CATPREZ_SPARQL_USERNAME=
CATPREZ_SPARQL_PASSWORD=

PREZ_DEV_SERVER_PORT=
SPARQL_ENDPOINT=
SPARQL_USERNAME=
SPARQL_PASSWORD=
PREZ_DEV_SERVER_PORT=
2 changes: 2 additions & 0 deletions .github/workflows/on_pr_to_main.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -64,3 +64,5 @@ jobs:
cd ../profiles && poetry run pytest
cd ../services && poetry run pytest
cd ../curies && poetry run pytest
cd ../object && poetry run pytest
cd ../caching && poetry run pytest
1 change: 0 additions & 1 deletion .gitignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,3 @@ __pycache__/
.pytest_cache/
.env*
!.env-template
tbox_cache.nt
20 changes: 19 additions & 1 deletion README-Dev.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ Prez returns:
- An alternates profile for every object or listing, listing all available profiles and mediatypes
- OpenAPI documentation for the API

\* Annotated RDF is RDF which includes labels, descriptions, and explanatory properties for all RDF terms. The predicates Prez looks for are rdfs:label, `dcterms:description`, and `dcterms:provenance`. The list of predicates Prez looks for can be extended in the profiles.
\* Annotated RDF is RDF which includes labels, descriptions, explanatory, and other properties for all RDF terms. The predicates Prez looks for are rdfs:label, `dcterms:description`, and `dcterms:provenance`. The list of predicates Prez looks for can be extended in the profiles.

## Internal links
The objects Prez delivers RDF for have URIs that uniquely identify them. Prez delivers RDF for these objects at URLs on the web. These URLs and URIs are not required to be the same, and frequently are not. For objects that Prez holds information for, it is helpful if Prez tells users the URL of these when they are referenced elsewhere in the API. This is in two places:
Expand Down Expand Up @@ -59,6 +59,24 @@ SELECT DISTINCT ?namespace
} LIMIT 100
```

## Annotation properties
Prez recognises the following kinds of annotation properties, and can return RDF, either via SPARQL queries, or the
endpoints as annotated RDF.

When an annotated mediatype is requested (e.g. `text/anot+turtle`), Prez will look for the following predicates for
*every* RDF term in the (initial) response returned by the triplestore. That is it will expand the response to include
the annotations and return the RDF merge of the original response and the annotations.

Additional predicates can be added to the list of predicates Prez looks for in the profiles by adding these predicates
using the properties listed below.

| Property | Default Predicate | Examples of other predicates that would commonly be used | Profiles predicate to add *additional* predicates |
|-------------|---------------------|----------------------------------------------------------|---------------------------------------------------|
| label | rdfs:label | skos:prefLabel, dcterms:title | altr-ext:hasLabelPredicate |
| description | dcterms:description | skos:definition, dcterms:abstract | altr-ext:hasDescriptionPredicate |
| provenance | dcterms:provenance | dcterms:source | altr-ext:hasExplanationPredicate |
| other | (None) | schema:color | altr-ext:otherAnnotationProps |

## High Level Sequence

Prez follows the following logic to determine what information to return, based on a profile, and in what mediatype to return it.
Expand Down
46 changes: 28 additions & 18 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -70,26 +70,33 @@ You do need to configure at least a couple of environment variables for Prez to

A minimal set of environment variables with example values to run prez is listed below:

`ENABLED_PREZS=["SpacePrez"]`
`SPACEPREZ_SPARQL_ENDPOINT=http://localhost:3030/spaceprez`
`SPARQL_ENDPOINT=http://localhost:3030/mydataset`

Yes, you really only need to tell Prez which subsystems to enable - here it's SpacePrez for spatial data - and point Prez at a SPARQL endpoint for it to get data from!
If the SPARQL endpoint requires authentication, you must also set:

#### Details

Of course, there are many other variables you may want to set. Here are some details.

See the `Settings` class within prez/config.py. You don't need to worry about the functions, just the variables there.

Have a look at the `Settings` class documentation for information about each variable.
`SPARQL_USERNAME=myusername`, and
`SPARQL_PASSWORD=mypassword`

Variable types (string, int etc.) are all indicated using Python type hinting, e.g.:

```python
port: int = 8000
```
#### Details

So the `port` variable must be an `int` and its default value is 8000.
Further configuration can be done via environment variables. These can be set in a '.env' file which will get read in
via python-dotenv, or directly in the environment in which Prez is run. The environment variables are used to
instantiate a Pydantic `Settings` object which is used throughout Prez to configure its behaviour. To see how prez
interprets/uses these environment variables see the `prez/config.py` file.

| Environment Variable | Description |
|-------------------|-----------------------------------------------------------------------------------------------------------------------------------|
| SPARQL_ENDPOINT | Read-only SPARQL endpoint for SpacePrez |
| SPARQL_USERNAME | A username for Basic Auth against the SPARQL endpoint, if required by the SPARQL endpoint. |
| SPARQL_PASSWORD | A password for Basic Auth against the SPARQL endpoint, if required by the SPARQL endpoint. |
| PROTOCOL | The protocol used to deliver Prez. Usually 'http'. |
| HOST | The host on which to server prez, typically 'localhost'. |
| PORT | The port Prez is made accessible on. Default is 8000, could be 80 or anything else that your system has permission to use |
| SYSTEM_URI | Documentation property. An IRI for the Prez system as a whole. This value appears in the landing page RDF delivered by Prez ('/') |
| LOG_LEVEL | One of CRITICAL, ERROR, WARNING, INFO, DEBUG. Defaults to INFO. |
| LOG_OUTPUT | "file", "stdout", or "both" ("file" and "stdout"). Defaults to stdout. |
| PREZ_TITLE | The title to use for Prez instance |
| PREZ_DESC | A description to use for the Prez instance |

### Running in a Container

Expand Down Expand Up @@ -162,10 +169,13 @@ Here are the lead developers:
<https://kurrawong.net>

_David Habgood_
<dcchabgood@gmail.com>
<david@kurrawong.ai>

_Nicholas Car_
<nick@kurrawong.net>
<nick@kurrawong.ai>

_Edmond Chuc_
<edmond@kurrawong.ai>

**University of Melbourne** - Prez UI mainly
_Jamie Feiss_
Expand Down
10 changes: 4 additions & 6 deletions demo/docker-compose.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ services:

fuseki:
user: root
image: "ghcr.io/zazuko/fuseki-geosparql:4.6.0" # recommend using the Zazuko repo to build a 4.7.0 image.
image: "ghcr.io/zazuko/fuseki-geosparql:4.8.0"
ports:
- "3030:3030"
volumes:
Expand All @@ -13,20 +13,18 @@ services:
environment:
ADMIN_PASSWORD: pw
healthcheck:
test: [ "CMD", "curl", "-f", "http://fuseki:3030" ]
test: ["CMD-SHELL", "wget -qO- http://localhost:3030 || exit 1"]
interval: 5s
timeout: 10s
retries: 3
start_period: 20s

prez:
image: "ghcr.io/rdflib/prez:3.0.6"
image: "ghcr.io/rdflib/prez:latest"
ports:
- "8000:8000"
environment:
SPACEPREZ_SPARQL_ENDPOINT: 'http://fuseki:3030/sp'
VOCPREZ_SPARQL_ENDPOINT: 'http://fuseki:3030/vp'
CATPREZ_SPARQL_ENDPOINT: 'http://fuseki:3030/cp'
SPARQL_ENDPOINT: 'http://fuseki:3030/sp'
depends_on:
fuseki:
condition: service_healthy
66 changes: 5 additions & 61 deletions dev/dev-config.ttl
View file
Original file line number Diff line number Diff line change
Expand Up @@ -40,9 +40,9 @@ ja:MemoryDataset rdfs:subClassOf ja:RDFDataset .

ja:DatasetRDFS rdfs:subClassOf ja:RDFDataset .

:service_sp rdf:type fuseki:Service ;
rdfs:label "sp" ;
fuseki:dataset :sp_dataset_readwrite ;
:service rdf:type fuseki:Service ;
rdfs:label "myds" ;
fuseki:dataset :dataset_readwrite ;
fuseki:endpoint [ fuseki:operation fuseki:update ] ;
fuseki:endpoint [ fuseki:name "update" ;
fuseki:operation fuseki:update
Expand All @@ -63,63 +63,7 @@ ja:DatasetRDFS rdfs:subClassOf ja:RDFDataset .
fuseki:endpoint [ fuseki:operation fuseki:query ] ;
fuseki:name "sp" .

:service_cp rdf:type fuseki:Service ;
rdfs:label "cp" ;
fuseki:dataset :cp_dataset_readwrite ;
fuseki:endpoint [ fuseki:operation fuseki:update ] ;
fuseki:endpoint [ fuseki:name "update" ;
fuseki:operation fuseki:update
] ;
fuseki:endpoint [ fuseki:name "sparql" ;
fuseki:operation fuseki:query
] ;
fuseki:endpoint [ fuseki:name "get" ;
fuseki:operation fuseki:gsp-r
] ;
fuseki:endpoint [ fuseki:name "query" ;
fuseki:operation fuseki:query
] ;
fuseki:endpoint [ fuseki:name "data" ;
fuseki:operation fuseki:gsp-rw
] ;
fuseki:endpoint [ fuseki:operation fuseki:gsp-rw ] ;
fuseki:endpoint [ fuseki:operation fuseki:query ] ;
fuseki:name "cp" .

:service_vp rdf:type fuseki:Service ;
rdfs:label "vp" ;
fuseki:dataset :vp_dataset_readwrite ;
fuseki:endpoint [ fuseki:operation fuseki:update ] ;
fuseki:endpoint [ fuseki:name "update" ;
fuseki:operation fuseki:update
] ;
fuseki:endpoint [ fuseki:name "sparql" ;
fuseki:operation fuseki:query
] ;
fuseki:endpoint [ fuseki:name "get" ;
fuseki:operation fuseki:gsp-r
] ;
fuseki:endpoint [ fuseki:name "query" ;
fuseki:operation fuseki:query
] ;
fuseki:endpoint [ fuseki:name "data" ;
fuseki:operation fuseki:gsp-rw
] ;
fuseki:endpoint [ fuseki:operation fuseki:gsp-rw ] ;
fuseki:endpoint [ fuseki:operation fuseki:query ] ;
fuseki:name "vp" .

:sp_dataset_readwrite
rdf:type tdb2:DatasetTDB2 ;
tdb2:unionDefaultGraph true ;
tdb2:location "/fuseki/databases/sp" .

:cp_dataset_readwrite
rdf:type tdb2:DatasetTDB2 ;
tdb2:unionDefaultGraph true ;
tdb2:location "/fuseki/databases/cp" .

:vp_dataset_readwrite
:dataset_readwrite
rdf:type tdb2:DatasetTDB2 ;
tdb2:unionDefaultGraph true ;
tdb2:location "/fuseki/databases/vp" .
tdb2:location "/fuseki/databases/myds" .
66 changes: 12 additions & 54 deletions dev/dev-setup.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@
time.sleep(15)


def setup_sp():
url = "http://localhost:3030/sp/"
def setup():
url = "http://localhost:3030/myds"

payload = {}
files = [
Expand Down Expand Up @@ -72,112 +72,72 @@ def setup_sp():
"application/octet-stream",
),
),
]
headers = {}

response = requests.request(
"POST",
url,
headers=headers,
data=payload,
files=files,
params={"graph": "http://exampledatagraph"},
)

print(response.text)


def setup_vp():
url = "http://localhost:3030/vp/"

payload = {}
files = [
(
"myfile",
"myfile8",
(
"alteration-types.ttl",
open("tests/data/vocprez/input/alteration-types.ttl", "rb"),
"application/octet-stream",
),
),
(
"myfile2",
"myfile9",
(
"contacttype.ttl",
open("tests/data/vocprez/input/contacttype.ttl", "rb"),
"application/octet-stream",
),
),
(
"myfile6",
"myfile10",
(
"dublin_core_terms.ttl",
open("tests/data/vocprez/input/dublin_core_terms.ttl", "rb"),
"application/octet-stream",
),
),
]
headers = {}

response = requests.request(
"POST",
url,
headers=headers,
data=payload,
files=files,
params={"graph": "http://exampledatagraph"},
)

print(response.text)


def setup_cp():
url = "http://localhost:3030/cp/"

payload = {}
files = [
(
"myfile",
"myfile11",
(
"_idn-ac.ttl",
open("tests/data/catprez/input/_idn-ac.ttl", "rb"),
"application/octet-stream",
),
),
(
"myfile2",
"myfile12",
(
"_idn-dc.ttl",
open("tests/data/catprez/input/_idn-dc.ttl", "rb"),
"application/octet-stream",
),
),
(
"myfile3",
"myfile13",
(
"_system-catalog.ttl",
open("tests/data/catprez/input/_system-catalog.ttl", "rb"),
"application/octet-stream",
),
),
(
"myfile4",
"myfile14",
(
"AAC-SA.ttl",
open("tests/data/catprez/input/AAC-SA.ttl", "rb"),
"application/octet-stream",
),
),
(
"myfile5",
"myfile15",
(
"agents.ttl",
open("tests/data/catprez/input/agents.ttl", "rb"),
"application/octet-stream",
),
),
(
"myfile6",
"myfile16",
(
"dublin_core_terms.ttl",
open("tests/data/vocprez/input/dublin_core_terms.ttl", "rb"),
Expand All @@ -199,6 +159,4 @@ def setup_cp():
print(response.text)


setup_sp()
setup_vp()
setup_cp()
setup()
Loading

0 comments on commit cc2b9d2

Please sign in to comment.