How to run SPARQL Test Suite?

[leipert]

I really want to contribute new tests for SPARQL REGEX (#46). What is the easiest way to run the suite and produce an implementation report assuming I have an SPARQL endpoint running at http://localhost:8080?

I found kasei/sparql11-protocolvalidator, but I am unfamiliar with PERL (and PERL cgi). Could someone give me a hint?

[gkellogg]

Everyone has their own mechanism for running the tests. Typically, parsing the appropriate test manifest (such as [this one](http://w3c.github.io/rdf-tests/sparql11/data-r2/regex/manifest.ttl, and iterating through each entry. There are some basic instructions for running tests here. Typically, they involve creating a dataset and parsing and running the associated query, where either the result-set or RDF Graph is expected to match that referenced from mf:result.

[leipert]

Thank you for the introduction 👍
This probably means I have to come up with my own mechanism. sigh Is there any test mechanism for test mechanisms? (just kidding) Or I learn a bit PERL ;)

[lisp]

there was a test harness which was introduced by an effort at inria (https://hal.inria.fr/hal-01104252/document), but i do not know its current status.
the website (http://sparqlscore.com) is not active.
there is a github repository (https://github.com/BorderCloud/TFT)which contains the php code.
we had issues with it at the protocol level, but it may serve your needs.

[afs]

Apache Jena provides a command line qtest for running SPARQL-style manifests of tests.

I'm sure other toolkits tat submitted reports provide ways to run the tests.

[BorderCloud]

@lisp Sparqlscore is always active but my time is limited.
You can find examples here :
https://github.com/BorderCloud/tft-jena-fuseki
https://github.com/BorderCloud/tft-virtuoso7-stable
https://github.com/BorderCloud/tft-stardog
https://github.com/BorderCloud/tft-blazegraph

I will write the doc this week but you can see my script.
I started to write a dynamic report (with students).

update: insert new examples

[ericprud]

in SWObjects, you can make test to run the normal tests and TEST_ARGS=--earl make test to emit EARL, c.f. test_SPARQL1.1

[rubensworks]

For reference, here's a tool for executing SPARQL-style test manifests for JavaScript implementations: https://www.npmjs.com/package/rdf-test-suite

[namedgraph]

These links are dead:

• https://github.com/w3c/rdf-tests/sparql
• https://w3c.github.io/rdf-tests/sparql11/

I think this issue is still relevant: how do we run the SPARQL test suite?

Also why is the following still the case when we have a standard SPARQL protocol?

Everyone has their own mechanism for running the tests

[namedgraph]

@BorderCloud I don't understand why your approach requires a Docker image per product?

I'm thinking we just need a "test driver" Docker image, so that the actual triplestore is accessed via a SPARQL endpoint URL and the tests are executed over HTTP.
Wouldn't that make much more sense?

[gkellogg]

These links are dead:

• https://github.com/w3c/rdf-tests/sparql
• https://w3c.github.io/rdf-tests/sparql11/

Where do you still find these links? The first would never have worked with a GitHub repo, the second was obsoleted as part of #84 and #90.

Generally, tests can be run from https://w3c.github.io/rdf-tests/sparql/ by following the manifests.

Exceptions are for the protocol test, which require an endpoint. I spent some time trying to make the manifests contain everything necessary to provision a local server to provide the appropriate responses (see #79), although this hasn't progressed in a while. This would be true for both sparql/sparql11/protocol and sparql/sparql11/http-rdf-update.

I think this issue is still relevant: how do we run the SPARQL test suite?

The top-level README could use some more specific instructions.

Also why is the following still the case when we have a standard SPARQL protocol?

Everyone has their own mechanism for running the tests

I run from https://w3c.github.io/rdf-tests/sparql with a local cache that prevents actual network access. Others run differently. Generally, tests (RDF and SPARQL) should be run as if accessing from GitHub.io, other than the protocol tests which require a runner be provisioned someplace.