Skip to content

Commit

Permalink
chore: release sprint 49
Browse files Browse the repository at this point in the history
  • Loading branch information
@Tomvbe
Tomvbe authored Mar 27, 2024
2 parents d614a28 + eb7f483 commit 40a480c
Show file tree
Hide file tree
Showing 163 changed files with 1,624 additions and 508 deletions.
6 changes: 6 additions & 0 deletions docs/_core/ldi-inputs/ldes-client.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,12 @@ Processed members of mutable fragments are also kept in state. They are ignored

Within a fragment, members can be ordered based on a timestamp. The path to this timestamp has to be configured. If this path is missing, the members are ordered randomly.

To allow the possibility to filter out already received members, the exactly-once-filter can be enabled.

This causes the ids of all processed members to be kept in state. They are ignored if presented more than once.

Enabling this filter may have a significant impact on performance.

[VSDS Tech Docs]: https://informatievlaanderen.github.io/VSDS-Tech-Docs/introduction/LDES_client

The client has different state persistence strategies:
Expand Down
2 changes: 1 addition & 1 deletion docs/_ldio/examples/ex2-scrape-api.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ has_toc: true
## Used Components

- [Http In Poller](../ldio-inputs/ldio-http-in-poller)
- [RML Adapter](../ldio-adapters/ldio-rml-adapter.md)
- [RML Adapter](../ldio-adapters/ldio-rml-adapter)
- [Version Object Creator](../ldio-transformers/ldio-version-object-creator)
- [Console Out](../ldio-outputs/ldio-console-out)

Expand Down
11 changes: 11 additions & 0 deletions docs/_ldio/index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,17 @@ nav_order: 0

A lightweight application maintained by the LDI team. Its creation came when a more lightweight alternative for [Apache NiFi] was needed.

## Enable swagger UI

To use the swagger UI on your own LDIO deployment you can add the below config,
and go to `<base-url>/v1/swagger`.

```yaml
springdoc:
swagger-ui:
path: /v1/swagger
```
## LDIO DEBUG Logging
To enable logging the input model for a
Expand Down
10 changes: 5 additions & 5 deletions docs/_ldio/ldio-adapters/ldio-json-to-json-ld.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,11 @@ title: Json To JsonLd Transformer

***Ldio:JsonToLdAdapter***

An LDIO wrapper component for the [LDI Json To JsonLd building block](../../_core/ldi-adapters/json-to-json-ld)
An LDIO wrapper component for the [LDI Json To JsonLd building block](../../core/ldi-adapters/json-to-json-ld)

## Config

| Property | Description | Required | Default | Example | Supported values |
|:--------------------------|:-------------------------------------------------------------------------------------------------------------|:---------|:--------|:--------------------------|:--------------------|
| core-context | URI of a core json-ld context. | Yes | N/A | http://example.com/my-api | HTTP and HTTPS urls |
| max-jsonld-cache-capacity | A cache is used when fetching json-ld contexts. The size of this cache can be configured with this property. | No | 100 | 100 | Integer |
| Property | Description | Required | Default | Example | Supported values |
|:--------------------------|:-------------------------------------------------------------------------------------------------------------|:---------|:--------|:--------------------------|:-------------------------------------------------|
| context | URI of json-ld context Or an JSON-LD context object. | Yes | N/A | http://example.com/my-api | URI or Json Object (containing "@context" entry) |
| max-jsonld-cache-capacity | A cache is used when fetching json-ld contexts. The size of this cache can be configured with this property. | No | 100 | 100 | Integer |
15 changes: 8 additions & 7 deletions docs/_ldio/ldio-inputs/ldio-ldes-client.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,13 +14,14 @@ An LDIO wrapper component for the [LDI LDES Client building block](../../core/ld

### General properties

| Property | Description | Required | Default | Example | Supported values |
|:---------------|:----------------------------------------------------------------------------------------|:---------|:------------|:------------------------------------------|:------------------------------------------------------------------------------------------------------------------------|
| urls | List of URLs of the LDES data sources | Yes | N/A | http://localhost:8080/my-ldes | HTTP and HTTPS urls |
| source-format | The 'Content-Type' that should be requested to the server. | No | text/turtle | application/n-quads | Any type supported by [Apache Jena](https://jena.apache.org/documentation/io/rdf-input.html#determining-the-rdf-syntax) |
| state | 'sqlite', 'memory', 'file' or 'postgres' to indicate how the state should be persisted. | No | memory | sqlite | 'sqlite', 'files' or 'memory' |
| keep-state | Indicates if the state should be persisted on shutdown (n/a for in memory states) | No | false | false | true or false |
| timestamp-path | The property-path used to determine the timestamp on which the members will be ordered | No | N/A | http://www.w3.org/ns/prov#generatedAtTime | A property path |
| Property | Description | Required | Default | Example | Supported values |
|:----------------------------|:----------------------------------------------------------------------------------------|:---------|:-------------|:------------------------------------------|:--------------------------------------------------------------|
| urls | List of URLs of the LDES data sources | Yes | N/A | http://localhost:8080/my-ldes | HTTP and HTTPS urls |
| source-format | The 'Content-Type' that should be requested to the server. | No | text/turtle | application/n-quads | Any type supported by [Apache Jena](https://jena.apache.org/documentation/io/rdf-input.html#determining-the-rdf-syntax) |
| state | 'sqlite', 'memory', 'file' or 'postgres' to indicate how the state should be persisted. | No | memory | sqlite | 'sqlite', 'files' or 'memory' |
| keep-state | Indicates if the state should be persisted on shutdown (n/a for in memory states) | No | false | false | true or false |
| timestamp-path | The property-path used to determine the timestamp on which the members will be ordered | No | N/A | http://www.w3.org/ns/prov#generatedAtTime | A property path |
| enable-exactly-once | Indicates whether member must be send exactly once or at least once | No | true | true | true or false |

> **_NOTE:_** The default `source-format` is `text/turtle`, as this rdf format supports relative uri's. However, if
> relative uri's are not used, `application/n-quads` or even the binary format `application/rdf+protobuf` are better
Expand Down
14 changes: 7 additions & 7 deletions docs/_ldio/ldio-outputs/ldio-repository-materialiser.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,13 @@ Any triplestore that supports the RDF4J remote repository API can be used.

## Config

| Property | Description | Required | Default | Example | Supported values |
|:--------------|:--------------------------------------------------------------------------------------------------------------------|:---------|:--------|:------------------------|:-----------------|
| sparql-host | The url of the server hosting the repository | Yes | N/A | http://repositoryServer | URL |
| repository-id | The rdf4j repository id | Yes | N/A | repoId | String |
| named-graph | If set, the triples will be written to this named graph | No | N/A | http://name | Any valid IRI |
| batch-size | Number of members or models that will be committed at once | No | 10000 | 500 | Integer |
| batch-timeout | If the batch size has not been reached within this number of milliseconds, the members will be committed anyway. | No | 120000 | 30000 | Integer |
| Property | Description | Required | Default | Example | Supported values |
|:--------------|:-----------------------------------------------------------------------------------------------------------------|:---------|:--------|:------------------------|:-----------------|
| sparql-host | The url of the server hosting the repository | Yes | N/A | http://repositoryServer | URL |
| repository-id | The rdf4j repository id | Yes | N/A | repoId | String |
| named-graph | If set, the triples will be written to this named graph | No | N/A | http://name | Any valid IRI |
| batch-size | Number of members or models that will be committed at once | No | 500 | 500 | Integer |
| batch-timeout | If the batch size has not been reached within this number of milliseconds, the members will be committed anyway. | No | 120000 | 30000 | Integer |

### Batching

Expand Down
9 changes: 1 addition & 8 deletions docs/_ldio/pipeline-management/index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -56,14 +56,7 @@ If this directory does not exist, it will be created.
## Pausing & Resuming LDIO

Sometimes it might be preferred to pause an LDIO pipeline instead of deleting and recreating it.
To pause a pipeline, simply call the following endpoint:
````
{base-url}/admin/api/v1/pipeline/{pipeline-name}/halt
````
And to resume a paused pipeline:
````
{base-url}/admin/api/v1/pipeline/{pipeline-name}/resume
````
The endpoints to manage pipelines can be found [here](pipeline-api.md)

The exact behaviour of a paused pipeline depends on its input component and can be found in the [documentation of these components](docs/_ldio/ldio-inputs/index.md).
However, it will always complete its current run through the pipeline and then seize sending any output.
Loading

0 comments on commit 40a480c

Please sign in to comment.