Hallucinated Reasoning Path

[riccardodello]

Hi,
I tried to run Graph-constrained decoding on Webqsp, but i noticed that some of the generated reasoning paths do not exists on the Knowledge Graph; from what I understand from the paper, this shouldn't be possible.

Best regards,
Riccardo

[RManLuo]

Hi Richardo, could you provide the full command you run to help me analyze the issues?

[riccardodello]

yes of course.

I run the script graph_constrained_decoding.sh

DATA_PATH=rmanluo
DATA_LIST="RoG-webqsp"
SPLIT="test"
INDEX_LEN=2
ATTN_IMP=sdpa

MODEL_PATH=rmanluo/GCR-Qwen2-0.5B-Instruct
MODEL_NAME=$(basename "$MODEL_PATH")

K="10" # 3 5 10 20
for DATA in ${DATA_LIST}; do
  for k in $K; do
    python3 workflow/predict_paths_and_answers.py --data_path ${DATA_PATH} --d ${DATA} --split ${SPLIT} --index_path_length ${INDEX_LEN} --model_name ${MODEL_NAME} --model_path ${MODEL_PATH} --k ${k} --prompt_mode zero-shot --generation_mode group-beam --attn_implementation ${ATTN_IMP}
  done
done

The only changes i've made are on ATTN_IMP and MODEL_PATH.

Everything works fine, except that i've got some reasoning paths that do not exists on the KG.
For example, on the first test question (WebQTest-0), i've got

"# Reasoning Path: Jamaica -> periodicals.newspaper_circulation_area.newspapers -> The Guardian Weekly -> book.periodical.language -> English Language # Answer: English Language"

but the node "The Guardian Weekly" does not exists on the Knowledge Graph.

[RManLuo]

I have checked the code. The possible reason is that Qwen2-0.5B fails to generate the special token <PATH> to start the constrained decoding. A possible solution is to add # Reasoning Path: <PATH> at the beginning of the generation output or change enable_constrained_by_default=True.

graph-constrained-reasoning/workflow/predict_paths_and_answers.py

Lines 72 to 78 in 202786a

	prediction = model.generate_sentence(
	input,
	trie,
	start_token_ids=start_token_ids,
	end_token_ids=end_token_ids,
	enable_constrained_by_default=False,
	)