superhero-real-names

A minimal example of evaluating a Qanary Question Answering system

This is a minimal example of evaluating a Qanary Question Answering system.

Table of Contents

• Example System
• Test Configuration
• Execution of Quality Evaluation Script

Example System

To execute the evaluation, you need to have a running Qanary system and need to store the configuration of the Qanary system in a file called qanary-test-definition.json in this folder.

Here 2 components are used in the Qanary system:

• NED-DBpediaSpotlight

• QB-SimpleRealNameOfSuperHero

Test Configuration

The test configuration is stored in the file qanary-test-definition.json. The configuration contains the following properties:

1. The example configuration contains 3 basic tests expressed by SPAQRL queries (configured in the file qanary-test-definition.json by the property validation-sparql-templates)):

   1. 0_was-any-instance-identified.sparql: Calculate the number of annotated entities and checks if there is at least one corresponding annotation.

   2. 1_is-the-annotation-correct.sparql: Validate if the pre-defined DBpedia resource (see the test file configuration for test-case-specific replacements/placeholders) is available in the graph as annotation of type AnnotationOfInstance.

   3. 2_was-a-sparql-query-computed-similar-as-expected.sparql: Checks if there was an SPARQL query computed by the Qanary Question Answering system and if the stored SPARQL query is similar to the expected result.

2. Additionally a custom test was implemented in the file qanary-test-definition.json by the property custom-validation. The correspond Python file custom-validation.py is located in this folder. It enables a flexible implementation of custom tests (requirement: needs to return a boolean value). Here, it validates a SPARQL created by the Qanary process

   1. the create SPARQL query is retrieved from the Qanary triplestore

   2. the SPARQL query is executed on DBpedia as it is intended by this particular system

Execution of Quality Evaluation Script

To execute the example evaluation, use the following command in the parent directory of this folder:

python evaluate-qanary-system.py --directory=superhero-real-names

The results will be stored in the folder output.