Skip to content

Commit

Permalink
Merge remote-tracking branch 'origin/wdqs-separate-concept-uri' into …
Browse files Browse the repository at this point in the history 
…deploy-set-wdqs-concept-uri
  • Loading branch information
@rti
rti committed Sep 30, 2024
2 parents 0977964 + c52b6ec commit add340c
Show file tree
Hide file tree
Showing 6 changed files with 30 additions and 18 deletions.
29 changes: 17 additions & 12 deletions build/wdqs/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -37,17 +37,18 @@ When running WDQS in a setup without WDQS-proxy, **please consider disabling the

Variables in **bold** are required.

| Variable | Default | Description |
| ------------------------ | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **`WIKIBASE_HOST`** | "wikibase" | Hostname of the Wikibase service |
| **`WDQS_HOST`** | "wdqs" | WDQS hostname (this service) |
| **`WDQS_PORT`** | "9999" | WDQS port (this service) |
| `WIKIBASE_SCHEME` | "http" | URL scheme of the Wikibase service |
| `WDQS_ENTITY_NAMESPACES` | "120,122" | Wikibase namespaces to load data from |
| `WIKIBASE_MAX_DAYS_BACK` | "90" | Maximum number of days updater can reach back in time from now |
| `MEMORY` | "" | Memory limit for Blazegraph |
| `HEAP_SIZE` | "1g" | Heap size for Blazegraph |
| `BLAZEGRAPH_EXTRA_OPTS` | "" | Extra options to be passed to Blazegraph,they must be prefixed with `-D`. Example: `-Dhttps.proxyHost=http://my.proxy.com -Dhttps.proxyPort=3128`. See [the WDQS User Manual](https://www.mediawiki.org/wiki/Wikidata_Query_Service/User_Manual#Configurable_properties). |
| Variable | Default | Description |
| ------------------------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **`WIKIBASE_HOST`** | "wikibase" | Hostname to reach the Wikibase service, e.g. the docker network internal hostname |
| **`WIKIBASE_CONCEPT_URI`** | "" | Concept URI, the identifying prefix to entities in this knowledge graph, e.g. the public URL of the Wikibase host. |
| **`WDQS_HOST`** | "wdqs" | WDQS hostname (this service) |
| **`WDQS_PORT`** | "9999" | WDQS port (this service) |
| `WIKIBASE_SCHEME` | "http" | URL scheme used to reach the Wikibase service, e.g. http to reach a local wikibase on the same docker network |
| `WDQS_ENTITY_NAMESPACES` | "120,122" | Wikibase namespaces to load data from |
| `WIKIBASE_MAX_DAYS_BACK` | "90" | Maximum number of days updater can reach back in time from now |
| `MEMORY` | "" | Memory limit for Blazegraph |
| `HEAP_SIZE` | "1g" | Heap size for Blazegraph |
| `BLAZEGRAPH_EXTRA_OPTS` | "" | Extra options to be passed to Blazegraph,they must be prefixed with `-D`. Example: `-Dhttps.proxyHost=http://my.proxy.com -Dhttps.proxyPort=3128`. See [the WDQS User Manual](https://www.mediawiki.org/wiki/Wikidata_Query_Service/User_Manual#Configurable_properties). |

## Example

Expand Down Expand Up @@ -124,6 +125,8 @@ services:
hard: 32768
volumes:
- wdqs-data:/wdqs/data
environment:
WIKIBASE_CONCEPT_URI: https://wikibase.example
healthcheck:
test: curl --silent --fail localhost:9999/bigdata/namespace/wdq/sparql
interval: 10s
Expand All @@ -140,6 +143,8 @@ services:
nofile:
soft: 32768
hard: 32768
environment:
WIKIBASE_CONCEPT_URI: https://wikibase.example

wdqs-proxy:
image: wikibase/wdqs-proxy
Expand Down Expand Up @@ -210,7 +215,7 @@ In the Docker Compose example provided above, you might use the commands and ins
docker compose stop wdqs-updater
# Start an updater with force sync settings
docker compose run --rm wdqs-updater bash '/wdqs/runUpdate.sh -h http://"$WDQS_HOST":"$WDQS_PORT" -- --wikibaseUrl "$WIKIBASE_SCHEME"://"$WIKIBASE_HOST" --conceptUri "$WIKIBASE_SCHEME"://"$WIKIBASE_HOST" --entityNamespaces "$WDQS_ENTITY_NAMESPACES" --init --start $(date +%Y%m%d000000)'
docker compose run --rm wdqs-updater bash '/wdqs/runUpdate.sh -h http://"$WDQS_HOST":"$WDQS_PORT" -- --wikibaseUrl "$WIKIBASE_SCHEME"://"$WIKIBASE_HOST" --conceptUri "$WIKIBASE_CONCEPT_URI" --entityNamespaces "$WDQS_ENTITY_NAMESPACES" --init --start $(date +%Y%m%d000000)'
# As soon as you see "Sleeping for 10 secs" in the logs, press CTRL-C to stop it again
Expand Down
2 changes: 1 addition & 1 deletion build/wdqs/entrypoint.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# This file is provided by the wikibase/wdqs docker image.

# Test if required environment variables have been set
REQUIRED_VARIABLES=(WIKIBASE_HOST WDQS_HOST WDQS_PORT)
REQUIRED_VARIABLES=(WIKIBASE_HOST WIKIBASE_CONCEPT_URI WDQS_HOST WDQS_PORT)
for i in "${REQUIRED_VARIABLES[@]}"; do
eval THISSHOULDBESET=\$"$i"
if [ -z "$THISSHOULDBESET" ]; then
Expand Down
4 changes: 2 additions & 2 deletions build/wdqs/runUpdate.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

cd /wdqs || exit

# TODO env vars for entity namespaces, scheme and other settings
# TODO env vars for entity namespaces
/wait-for-it.sh "$WIKIBASE_HOST:80" -t 300 -- \
/wait-for-it.sh "$WDQS_HOST:$WDQS_PORT" -t 300 -- \
./runUpdate.sh -h http://"$WDQS_HOST":"$WDQS_PORT" -- --wikibaseUrl "$WIKIBASE_SCHEME"://"$WIKIBASE_HOST" --conceptUri "$WIKIBASE_SCHEME"://"$WIKIBASE_HOST" --entityNamespaces "$WDQS_ENTITY_NAMESPACES"
./runUpdate.sh -h http://"$WDQS_HOST":"$WDQS_PORT" -- --wikibaseUrl "$WIKIBASE_SCHEME"://"$WIKIBASE_HOST" --conceptUri "$WIKIBASE_CONCEPT_URI" --entityNamespaces "$WDQS_ENTITY_NAMESPACES"
6 changes: 3 additions & 3 deletions test/specs/repo/queryservice.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@ describe( 'QueryService', function () {
// property value is set with correct rdf
await expect(
QueryServiceUIPage.resultIncludes(
`<${ testEnv.vars.WIKIBASE_URL }/prop/direct/${ propertyId }>`,
`<${ testEnv.vars.WIKIBASE_CONCEPT_URI }/prop/direct/${ propertyId }>`,
propertyValue
)
).resolves.toEqual( true );
Expand All @@ -122,7 +122,7 @@ describe( 'QueryService', function () {
// should be set only to the item
await expect(
QueryServiceUIPage.resultIncludes(
`<${ testEnv.vars.WIKIBASE_URL }/entity/${ itemId }>`,
`<${ testEnv.vars.WIKIBASE_CONCEPT_URI }/entity/${ itemId }>`,
propertyValue
)
).resolves.toEqual( true );
Expand Down Expand Up @@ -256,7 +256,7 @@ describe( 'QueryService', function () {

await QueryServiceUIPage.open( `SELECT (COUNT(*) AS ?count)
WHERE {
<${ testEnv.vars.WIKIBASE_URL }/entity/${ itemId }> <${ testEnv.vars.WIKIBASE_URL }/prop/direct/${ propertyId }> "test-property" .
<${ testEnv.vars.WIKIBASE_CONCEPT_URI }/entity/${ itemId }> <${ testEnv.vars.WIKIBASE_URL }/prop/direct/${ propertyId }> "test-property" .
}` );

// wait for WDQS-updater
Expand Down
6 changes: 6 additions & 0 deletions test/suites/docker-compose.override.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,15 @@ services:

wdqs:
image: wikibase/wdqs
# TODO: remove as soon as #772 is merged
environment:
WIKIBASE_CONCEPT_URI: https://${WIKIBASE_PUBLIC_HOST}

wdqs-updater:
image: wikibase/wdqs
# TODO: remove as soon as #772 is merged
environment:
WIKIBASE_CONCEPT_URI: https://${WIKIBASE_PUBLIC_HOST}

wdqs-proxy:
image: wikibase/wdqs-proxy
Expand Down
1 change: 1 addition & 0 deletions test/test-services.env
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@

# URLS
WIKIBASE_URL=http://wikibase
WIKIBASE_CONCEPT_URI=http://wikibase
WIKIBASE_CLIENT_URL=http://wikibase-client
WDQS_URL=http://wdqs:9999
WDQS_FRONTEND_URL=http://wdqs-frontend
Expand Down

0 comments on commit add340c

Please sign in to comment.