This repository provides resources developed within the following article:
D. Garigliotti and K. Balog. Towards an Understanding of Entity-Oriented Search Intents. In: Advances in Information Retrieval - Proceedings of the 40th European Conference on Information Retrieval (ECIR '18). Springer. Grenoble, France. March 2018. DOI: 10.1007/978-3-319-76941-7_57
You can get the author version of the article here.
Entity-oriented search deals with a wide variety of information needs, from displaying direct answers to interacting with services. In this work, we aim to understand what are prominent entity-oriented search intents and how they can be fulfilled. We develop a scheme of entity intent categories, and use them to annotate a sample of queries. Specifically, we annotate unique query refiners on the level of entity types. We observe that, on average, over half of those refiners seek to interact with a service, while over a quarter of the refiners search for information that may be looked up in a knowledge base.
In annotation_output/refiners_categorization.tsv, we provide the output of our annotation experiment conducted by crowdsourcing (details in the paper): a large collection of type-level refiners, annotated with intent categories.
- Each row of the TSV file corresponds to a (
[type],refiner) pair (stored in the 1st and 2nd columns, resp.), which an intent category is assigned to (3rd column) by majority agreement. - The confidence score of a row (4th column) is calculated simply as the number of judgments for that category normalized by the total of annotations for its pair. As detailed in the paper, each instance was annotated by at least 3 judges (5 at most, if necessary to reach a majority agreement, using dynamic judgments). For each type, we only retain an annotated refiner if at least three annotators agreed on the majority category.
Below, an excerpt of this annotation output:
Type Intent Top_judged_category Judgment_rate_(confidence)
[airport] official website website 1.0
[airport] facebook website 1.0
[airport] weather service 1.0
[airport] to train station service 1.0
[airport] zip code property 1.0
[airport] logo property 1.0
[airport] china other 0.75
[airport] crash other 0.6
...
We used crowdsourcing to annotate type-level refiners with intent categories.
- For each annotation instance we displayed workers with the query, indicating its entity type and refiner, and asked them to select one of the four intent categories (details in the paper).
Below, screenshots of the annotation job on Crowdflower.
If you use the resources presented in this repository, please cite:
@InProceedings{Garigliotti:2018:TAU,
author = {Garigliotti, Dar{\'i}o
and Balog, Krisztian},
title = {Towards an Understanding of Entity-Oriented Search Intents},
booktitle = {Advances in Information Retrieval - Proceedings of the 40th European Conference on IR Research},
series = {ECIR '18},
year = {2018},
pages = {644--650},
publisher = {Springer},
doi = {10.1007/978-3-319-76941-7_57},
}
Should you have any questions, please contact Darío Garigliotti at dario.garigliotti[AT]uis.no (with [AT] replaced by @).