MAGIC: A Multi-Hop and Graph-Based Benchmark for Inter-Context Conflicts in Retrieval-Augmented Generation

Official Repository for "MAGIC: A Multi-Hop and Graph-Based Benchmark for Inter-Context Conflicts in Retrieval-Augmented Generation". [📄 Paper(arXiv)]

Jungyeon Lee, Kangmin Lee and Taeuk Kim. Accepted to EMNLP 2025 Findings.

---

OverView

MAGIC is a large-scale benchmark designed to evaluate knowledge conflict detection and localization in Retrieval-Augmented Generation (RAG) systems. It focuses on multi-hop reasoning and graph-structured contexts, where inter-context knowledge conflicts emerge between retrieved passages.

Repository Structure

.\
├── dataset
│   ├── multi-hop/
│   │   ├── 1-multi-hop_conflict.json/
│   │   ├── 2-multi-hop_conflict.json/
│   │   ├── 3-multi-hop_conflict.json/
│   │   ├── 4-multi-hop_conflict.json/
│   ├── single-hop/
│   │   ├── 1-single-hop_conflict.json/
│   │   ├── 2-single-hop_conflict.json/
│   │   ├── 3-single-hop_conflict.json/
│   │   ├── 4-single-hop_conflict.json/

Dataset Structure

• ID: Unique identifier for each sample.
• rel_id: Relation ID corresponding to the target knowledge relation (e.g. P150 from Wikidata)
• subgraph: A set of surrounding triplets retrieved via DFS traversal from the source knowledge graph around the original_triplet.
• original_triplet: Randomly sampled target triplet from the source graph — serves as the anchor for conflict formation.
• perturb_triplet: Modified triplet(s) intentionally constructed to introduce a knowledge conflict with the original_triplet.
• context1, context2: Textual representations of the original_triplet and perturb_triplet, respectively.