Skip to content

Commit 60ff2fd

Browse files
aucampiaaucampia
authored
Rework turtle-like test suites (#1986)
This patch reworks the turtle-like test suites (N-Quads, N-Triples, Turtle, TriG). The changes includes: - A new DAWG Manifest processor: - Incorporates pytest expected failure support in test parameter generation. - Supports URI mapping so that local filesystem URIs can be mapped to the correct remote base URIs and still work correctly with local files. - Supports custom entry types so that suite specific handling can be implemented separately from the core manifest processor. This will be used for SPARQL test suite and JSON test suite processing. - Updated EARL reporter: - Support for the new DAWG Manifest processor. - Writing of EARL reports by default if a report prefix is specified in the DAWG Manifest processor. - Reports output is sorted so that it can be compared with text based diff tools. - Fixed reporting of xfail tests. These were reported as skipped before, now they are reported as failed. The testing is also more accurate, and now tests which were reported as passing before are now correctly reported as failing. With one exception all these failures are related to leniency in the parser, as our parser accepts input that should raise an error when parsing. Other changes: - Moved IRI related test utilities to a separate module. - Added generated reports for turtle-like formats to git with `-HEAD` suffix. Subsequent changes will use this new processor for SPARQL and RDF/XML.
1 parent 7d8cab6 commit 60ff2fd

16 files changed

+7631
-401
lines changed

‎test/README.rst‎

Lines changed: 30 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -50,56 +50,56 @@ EARL test reports can be generated using the EARL reporter plugin from ``earl.py
5050

5151
When this plugin is enabled it will create an ``earl:Assertion`` for every test that has a ``rdf_test_uri`` parameter which can be either a string or an ``URIRef``.
5252

53-
To enable the EARL reporter plugin an output file path must be supplied to pytest with ``--earl-report``. The report will be written to this location in turtle format.
53+
To enable the EARL reporter plugin an output file path must be supplied to pytest with ``--earl-output-file``. The report will be written to this location in turtle format.
5454

5555
Some examples of generating test reports:
5656

5757
.. code-block:: bash
5858
5959
pytest \
60-
--earl-asserter-homepage=http://example.com \
61-
--earl-asserter-name 'Example Name' \
62-
--earl-report=/var/tmp/earl/earl-jsonld-local.ttl \
60+
--earl-assertor-homepage=http://example.com \
61+
--earl-assertor-name 'Example Name' \
62+
--earl-output-file=/var/tmp/earl/earl-jsonld-local.ttl \
6363
test/jsonld/test_localsuite.py
6464
6565
pytest \
66-
--earl-asserter-homepage=http://example.com \
67-
--earl-asserter-name 'Example Name' \
68-
--earl-report=/var/tmp/earl/earl-jsonld-v1.1.ttl \
66+
--earl-assertor-homepage=http://example.com \
67+
--earl-assertor-name 'Example Name' \
68+
--earl-output-file=/var/tmp/earl/earl-jsonld-v1.1.ttl \
6969
test/jsonld/test_onedotone.py
7070
7171
pytest \
72-
--earl-asserter-homepage=http://example.com \
73-
--earl-asserter-name 'Example Name' \
74-
--earl-report=/var/tmp/earl/earl-jsonld-v1.0.ttl \
72+
--earl-assertor-homepage=http://example.com \
73+
--earl-assertor-name 'Example Name' \
74+
--earl-output-file=/var/tmp/earl/earl-jsonld-v1.0.ttl \
7575
test/jsonld/test_testsuite.py
7676
7777
pytest \
78-
--earl-asserter-homepage=http://example.com \
79-
--earl-asserter-name 'Example Name' \
80-
--earl-report=/var/tmp/earl/earl-sparql.ttl \
81-
test/test_dawg.py
78+
--earl-assertor-homepage=http://example.com \
79+
--earl-assertor-name 'Example Name' \
80+
--earl-output-file=/var/tmp/earl/earl-sparql.ttl \
81+
test/test_w3c_spec/test_sparql_w3c.py
8282
8383
pytest \
84-
--earl-asserter-homepage=http://example.com \
85-
--earl-asserter-name 'Example Name' \
86-
--earl-report=/var/tmp/earl/earl-nquads.ttl \
87-
test/test_nquads_w3c.py
84+
--earl-assertor-homepage=http://example.com \
85+
--earl-assertor-name 'Example Name' \
86+
--earl-output-file=/var/tmp/earl/earl-nquads.ttl \
87+
test/test_w3c_spec/test_nquads_w3c.py
8888
8989
pytest \
90-
--earl-asserter-homepage=http://example.com \
91-
--earl-asserter-name 'Example Name' \
92-
--earl-report=/var/tmp/earl/earl-nt.ttl \
93-
test/test_nt_w3c.py
90+
--earl-assertor-homepage=http://example.com \
91+
--earl-assertor-name 'Example Name' \
92+
--earl-output-file=/var/tmp/earl/earl-nt.ttl \
93+
test/test_w3c_spec/test_nt_w3c.py
9494
9595
pytest \
96-
--earl-asserter-uri=http://example.com \
97-
--earl-asserter-name 'Example Name' \
98-
--earl-report=/var/tmp/earl/earl-trig.ttl \
99-
test/test_trig_w3c.py
96+
--earl-assertor-homepage=http://example.com \
97+
--earl-assertor-name 'Example Name' \
98+
--earl-output-file=/var/tmp/earl/earl-trig.ttl \
99+
test/test_w3c_spec/test_trig_w3c.py
100100
101101
pytest \
102-
--earl-asserter-uri=http://example.com \
103-
--earl-asserter-name 'Example Name' \
104-
--earl-report=/var/tmp/earl/earl-turtle.ttl \
105-
test/test_turtle_w3c.py
102+
--earl-assertor-homepage=http://example.com \
103+
--earl-assertor-name 'Example Name' \
104+
--earl-output-file=/var/tmp/earl/earl-turtle.ttl \
105+
test/test_w3c_spec/test_turtle_w3c.py

‎test/conftest.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,9 +5,9 @@
55
from rdflib import Graph
66

77
from .data import TEST_DATA_DIR
8-
from .utils.earl import EarlReporter
8+
from .utils.earl import EARLReporter # noqa: E402
99

10-
pytest_plugins = [EarlReporter.__module__]
10+
pytest_plugins = [EARLReporter.__module__]
1111

1212
# This is here so that asserts from these modules are formatted for human
1313
# readibility.

‎test/test_w3c_spec/test_nquads_w3c.py‎

Lines changed: 74 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -1,42 +1,92 @@
11
"""This runs the nquads tests for the W3C RDF Working Group's N-Quads
22
test suite."""
3-
4-
5-
import os
3+
import logging
4+
from contextlib import ExitStack
65
from test.data import TEST_DATA_DIR
7-
from test.utils.manifest import RDFTest, read_manifest
6+
from test.utils import BNodeHandling, GraphHelper, ensure_suffix
7+
from test.utils.dawg_manifest import ManifestEntry, params_from_sources
8+
from test.utils.iri import URIMapper
89
from test.utils.namespace import RDFT
9-
from typing import Callable, Dict
10+
from typing import Optional
1011

1112
import pytest
1213

13-
from rdflib import ConjunctiveGraph
14-
from rdflib.term import Node, URIRef
14+
from rdflib.graph import Dataset
1515

16-
verbose = False
16+
logger = logging.getLogger(__name__)
17+
18+
REMOTE_BASE_IRI = "http://www.w3.org/2013/NQuadsTests/"
19+
LOCAL_BASE_DIR = TEST_DATA_DIR / "suites/w3c/nquads/"
20+
ENCODING = "utf-8"
21+
MAPPER = URIMapper.from_mappings(
22+
(REMOTE_BASE_IRI, ensure_suffix(LOCAL_BASE_DIR.as_uri(), "/"))
23+
)
24+
VALID_TYPES = {RDFT.TestNQuadsPositiveSyntax, RDFT.TestNQuadsNegativeSyntax}
1725

1826

19-
def nquads(test):
20-
g = ConjunctiveGraph()
27+
def check_entry(entry: ManifestEntry) -> None:
28+
assert entry.action is not None
29+
assert entry.type in VALID_TYPES
30+
action_path = entry.uri_mapper.to_local_path(entry.action)
31+
if logger.isEnabledFor(logging.DEBUG):
32+
logger.debug(
33+
"action = %s\n%s", action_path, action_path.read_text(encoding=ENCODING)
34+
)
35+
catcher: Optional[pytest.ExceptionInfo[Exception]] = None
36+
dataset = Dataset()
37+
with ExitStack() as xstack:
38+
if entry.type == RDFT.TestNQuadsNegativeSyntax:
39+
catcher = xstack.enter_context(pytest.raises(Exception))
40+
dataset.parse(action_path, publicID=entry.action, format="nquads")
41+
if catcher is not None:
42+
assert catcher.value is not None
2143

22-
try:
23-
g.parse(test.action, format="nquads")
24-
if not test.syntax:
25-
raise AssertionError("Input shouldn't have parsed!")
26-
except:
27-
if test.syntax:
28-
raise
44+
if entry.type == RDFT.TestNQuadsPositiveSyntax:
45+
graph_data = dataset.serialize(format="nquads")
46+
result_dataset = Dataset()
47+
result_dataset.parse(data=graph_data, publicID=entry.action, format="nquads")
48+
GraphHelper.assert_cgraph_isomorphic(
49+
dataset, result_dataset, exclude_bnodes=True
50+
)
51+
GraphHelper.assert_sets_equals(
52+
dataset, result_dataset, bnode_handling=BNodeHandling.COLLAPSE
53+
)
2954

3055

31-
testers: Dict[Node, Callable[[RDFTest], None]] = {
32-
RDFT.TestNQuadsPositiveSyntax: nquads,
33-
RDFT.TestNQuadsNegativeSyntax: nquads,
56+
MARK_DICT = {
57+
f"{REMOTE_BASE_IRI}#nt-syntax-bad-uri-02": pytest.mark.xfail(
58+
reason="accepts an invalid IRI"
59+
),
60+
f"{REMOTE_BASE_IRI}#nt-syntax-bad-uri-03": pytest.mark.xfail(
61+
reason="accepts an invalid IRI"
62+
),
63+
f"{REMOTE_BASE_IRI}#nt-syntax-bad-uri-04": pytest.mark.xfail(
64+
reason="accepts an invalid IRI"
65+
),
66+
f"{REMOTE_BASE_IRI}#nt-syntax-bad-uri-05": pytest.mark.xfail(
67+
reason="accepts an invalid IRI"
68+
),
69+
f"{REMOTE_BASE_IRI}#nt-syntax-bad-esc-01": pytest.mark.xfail(
70+
reason="accepts badly escaped literal"
71+
),
72+
f"{REMOTE_BASE_IRI}#nt-syntax-bad-esc-02": pytest.mark.xfail(
73+
reason="accepts badly escaped literal"
74+
),
75+
f"{REMOTE_BASE_IRI}#nt-syntax-bad-esc-03": pytest.mark.skip(
76+
reason="accepts badly escaped literal"
77+
),
3478
}
3579

3680

3781
@pytest.mark.parametrize(
38-
"rdf_test_uri, type, rdf_test",
39-
read_manifest(os.path.join(TEST_DATA_DIR, "suites", "w3c/nquads/manifest.ttl")),
82+
["manifest_entry"],
83+
params_from_sources(
84+
MAPPER,
85+
ManifestEntry,
86+
LOCAL_BASE_DIR / "manifest.ttl",
87+
mark_dict=MARK_DICT,
88+
report_prefix="rdflib_w3c_nquads",
89+
),
4090
)
41-
def test_manifest(rdf_test_uri: URIRef, type: Node, rdf_test: RDFTest):
42-
testers[type](rdf_test)
91+
def test_entry(manifest_entry: ManifestEntry) -> None:
92+
check_entry(manifest_entry)

‎test/test_w3c_spec/test_nt_w3c.py‎

Lines changed: 78 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -1,42 +1,96 @@
1-
"""This runs the nt tests for the W3C RDF Working Group's N-Quads
1+
"""This runs the nt tests for the W3C RDF Working Group's N-Triples
22
test suite."""
3-
import os
3+
import logging
4+
from contextlib import ExitStack
45
from test.data import TEST_DATA_DIR
5-
from test.utils.manifest import RDFTest, read_manifest
6+
from test.utils import BNodeHandling, GraphHelper, ensure_suffix
7+
from test.utils.dawg_manifest import ManifestEntry, params_from_sources
8+
from test.utils.iri import URIMapper
69
from test.utils.namespace import RDFT
7-
from typing import Callable, Dict
10+
from typing import Optional
811

912
import pytest
1013

11-
from rdflib import Graph
12-
from rdflib.term import Node, URIRef
14+
from rdflib.graph import Graph
1315

14-
verbose = False
16+
logger = logging.getLogger(__name__)
17+
18+
REMOTE_BASE_IRI = "http://www.w3.org/2013/N-TriplesTests/"
19+
LOCAL_BASE_DIR = TEST_DATA_DIR / "suites/w3c/ntriples/"
20+
ENCODING = "utf-8"
21+
MAPPER = URIMapper.from_mappings(
22+
(REMOTE_BASE_IRI, ensure_suffix(LOCAL_BASE_DIR.as_uri(), "/"))
23+
)
24+
VALID_TYPES = {RDFT.TestNTriplesPositiveSyntax, RDFT.TestNTriplesNegativeSyntax}
1525

1626

17-
def nt(test):
18-
g = Graph()
27+
def check_entry(entry: ManifestEntry) -> None:
28+
assert entry.action is not None
29+
assert entry.type in VALID_TYPES
30+
action_path = entry.uri_mapper.to_local_path(entry.action)
31+
if logger.isEnabledFor(logging.DEBUG):
32+
logger.debug(
33+
"action = %s\n%s", action_path, action_path.read_text(encoding=ENCODING)
34+
)
35+
catcher: Optional[pytest.ExceptionInfo[Exception]] = None
36+
graph = Graph()
37+
with ExitStack() as xstack:
38+
if entry.type == RDFT.TestNTriplesNegativeSyntax:
39+
catcher = xstack.enter_context(pytest.raises(Exception))
40+
graph.parse(action_path, publicID=entry.action, format="ntriples")
41+
if catcher is not None:
42+
assert catcher.value is not None
1943

20-
try:
21-
g.parse(test.action, format="nt")
22-
if not test.syntax:
23-
raise AssertionError("Input shouldn't have parsed!")
24-
except:
25-
if test.syntax:
26-
raise
44+
if entry.type == RDFT.TestNTriplesPositiveSyntax:
45+
graph_data = graph.serialize(format="ntriples")
46+
result_graph = Graph()
47+
result_graph.parse(data=graph_data, publicID=entry.action, format="ntriples")
48+
GraphHelper.assert_isomorphic(graph, result_graph)
49+
GraphHelper.assert_sets_equals(
50+
graph, result_graph, bnode_handling=BNodeHandling.COLLAPSE
51+
)
2752

2853

29-
testers: Dict[Node, Callable[[RDFTest], None]] = {
30-
RDFT.TestNTriplesPositiveSyntax: nt,
31-
RDFT.TestNTriplesNegativeSyntax: nt,
54+
MARK_DICT = {
55+
f"{REMOTE_BASE_IRI}#nt-syntax-bad-uri-02": pytest.mark.xfail(
56+
reason="accepts an invalid IRI"
57+
),
58+
f"{REMOTE_BASE_IRI}#nt-syntax-bad-uri-03": pytest.mark.xfail(
59+
reason="accepts an invalid IRI"
60+
),
61+
f"{REMOTE_BASE_IRI}#nt-syntax-bad-uri-04": pytest.mark.xfail(
62+
reason="accepts an invalid IRI"
63+
),
64+
f"{REMOTE_BASE_IRI}#nt-syntax-bad-uri-05": pytest.mark.xfail(
65+
reason="accepts an invalid IRI"
66+
),
67+
f"{REMOTE_BASE_IRI}#nt-syntax-bad-esc-01": pytest.mark.xfail(
68+
reason="accepts badly escaped literal"
69+
),
70+
f"{REMOTE_BASE_IRI}#nt-syntax-bad-esc-02": pytest.mark.xfail(
71+
reason="accepts badly escaped literal"
72+
),
73+
f"{REMOTE_BASE_IRI}#nt-syntax-bad-esc-03": pytest.mark.xfail(
74+
reason="accepts badly escaped literal"
75+
),
76+
f"{REMOTE_BASE_IRI}#nt-syntax-bad-esc-04": pytest.mark.xfail(
77+
reason="accepts badly escaped literal"
78+
),
79+
f"{REMOTE_BASE_IRI}#minimal_whitespace": pytest.mark.xfail(
80+
reason="Not parsing valid N-Triples syntax."
81+
),
3282
}
3383

3484

3585
@pytest.mark.parametrize(
36-
"rdf_test_uri, type, rdf_test",
37-
read_manifest(
38-
os.path.join(TEST_DATA_DIR, "suites", "w3c/ntriples/manifest.ttl"), legacy=True
86+
["manifest_entry"],
87+
params_from_sources(
88+
MAPPER,
89+
ManifestEntry,
90+
LOCAL_BASE_DIR / "manifest.ttl",
91+
mark_dict=MARK_DICT,
92+
report_prefix="rdflib_w3c_ntriples",
3993
),
4094
)
41-
def test_manifest(rdf_test_uri: URIRef, type: Node, rdf_test: RDFTest):
42-
testers[type](rdf_test)
95+
def test_entry(manifest_entry: ManifestEntry) -> None:
96+
check_entry(manifest_entry)

0 commit comments

Comments
 (0)