Graph-R1: When GraphRAG Meets RL

Graph-R1: Incentivizing Reasoning-on-Graph Capability in LLM via Reinforcement Learning

Overview

Recently, the GraphRAG method effectively addresses the data silos issue, significantly enhancing knowledge retrieval efficiency. Nevertheless, the disconnect between graph-structured knowledge and language modalities continues to constrain performance.

To bridge this gap, we propose Graph-R1, an end-to-end reinforcement learning (RL) framework designed to improve reasoning-on-graph capabilities of large language models (LLMs).

Specifically, we constructs a knowledge hypergraph using n-ary relation extraction from HyperGraphRAG. We then employ an explicit reward mechanism within RL, enabling the LLM to iteratively execute a "think–generate query–retrieve subgraph–rethink" reasoning cycle. This iterative approach enables the model to effectively leverage graph knowledge to produce high-quality answers.

By integrating structured knowledge into LLM reasoning more flexibly via reinforcement learning, Graph-R1 holds promise for aplications in knowledge-intensive fields such as healthcare, finance, and law.

Our paper is coming soon.

Experimental Results

Results on Different RL Algorithms:

Results on Different GraphRAG Datasets:

Results on Different Parameter Scale of LLM:

Graph-R1 Implementation

Install Environment

conda create -n graphr1 python==3.11.11
conda activate graphr1
pip3 install torch==2.4.0 --index-url https://download.pytorch.org/whl/cu124
pip3 install flash-attn --no-build-isolation
pip3 install -e .
pip3 install -r requirements.txt

Dataset Preparation

We conduct experiments on seven datasets: 2WikiMultiHopQA, HotpotQA, Musique, NarrativeQA, NQ, PopQA, and TriviaQA. You can download them from TeraBox, and set the data path in datasets/.

Quick Start: Graph-R1 on 2WikiMultiHopQA

1. Preprocess 2WikiMultiHopQA dataset to parquet format

python script_process.py --data_source 2WikiMultiHopQA

2. Extract contexts and build Knowledge HyperGraph (Optional)

We use GPT-4o-mini as extractor, so you should set your openai API key in openai_api_key_txt.

nohup python -u script_build.py --data_source 2WikiMultiHopQA > result_build_2WikiMultiHopQA.log 2>&1 &

You can also skip this step, download the pre-built Knowledge HyperGraph from TeraBox, and set in expr/.

3. Set up retrieve server at 8001 port

nohup python -u script_api.py --data_source 2WikiMultiHopQA > result_api_2WikiMultiHopQA.log 2>&1 &

4. Run GRPO/REINFORCE++/PPO training with Qwen2.5-1.5B-Instruct (Need 4 x 32GB GPUs)

# GRPO
nohup bash -u run_grpo.sh -p Qwen/Qwen2.5-1.5B-Instruct -m Qwen2.5-1.5B-Instruct -d 2WikiMultiHopQA > result_run_Qwen2.5-1.5B-Instruct_2WikiMultiHopQA_grpo.log 2>&1 &
# REINFORCE++
nohup bash -u run_rpp.sh -p Qwen/Qwen2.5-1.5B-Instruct -m Qwen2.5-1.5B-Instruct -d 2WikiMultiHopQA > result_run_Qwen2.5-1.5B-Instruct_2WikiMultiHopQA_rpp.log 2>&1 &
# PPO
nohup bash -u run_ppo.sh -p Qwen/Qwen2.5-1.5B-Instruct -m Qwen2.5-1.5B-Instruct -d 2WikiMultiHopQA > result_run_Qwen2.5-1.5B-Instruct_2WikiMultiHopQA_ppo.log 2>&1 &

5. Close retrieve server 8001 port

fuser -k 8001/tcp

BibTex

If you find this work is helpful for your research, please cite:

@misc{luo2025graphr1,
      title={Graph-R1: Incentivizing Reasoning-on-Graph Capability in LLM via Reinforcement Learning}, 
      author={Haoran Luo},
      year={2025},
      organization = {GitHub},
      url={https://github.com/LHRLAB/Graph-R1}, 
}

For further questions, please contact: luohaoran@bupt.edu.cn.

Acknowledgement

This repo benefits from Agent-R1, HyperGraphRAG, FlashRAG, LightRAG, HippoRAG2, R1-Searcher and Search-R1. Thanks for their wonderful works.