Right Prediction, Wrong Reasoning: Uncovering LLM Misalignment in RA Disease Diagnosis

Authors

• Umakanta Maharana
  RespAI Lab, KIIT Bhubaneswar
• Sarthak Verma
  KIMS Bhubaneswar
• Avarna Agarwal
  KIMS Bhubaneswar
• Prakashini Mruthyunjaya
  KIMS Bhubaneswar
• Dwarikanath Mahapatra
  Monash University, Australia
• Sakir Ahmed
  KIMS Bhubaneswar
• Murari Mandal
  RespAI Lab, KIIT Bhubaneswar

Correspondence: Murari Mandal arXiv Identifier: arXiv:2504.06581v1 [cs.AI]

Key Highlights

• Dataset: PreRAID, comprising 160 patient records from KIMS, Bhubaneswar.
• Diagnosis Accuracy: LLMs predicted RA with 95% accuracy.
• Reasoning Validation: Expert review revealed 68% flawed reasoning despite correct predictions.
• Implications: Highlights the critical need for reliable reasoning in clinical AI tools.

Conclusion

This study underscores the potential of LLMs in disease diagnosis while emphasizing the importance of improving reasoning mechanisms for trustworthy clinical applications.

Acknowledgment

This research is supported by the Science and Engineering Research Board (SERB), India under Grant SRG/2023/001686.

Citation

Please cite the following paper when using the PreRAID dataset:

@misc{maharana2025rightpredictionwrongreasoning,
      title={Right Prediction, Wrong Reasoning: Uncovering LLM Misalignment in RA Disease Diagnosis}, 
      author={Umakanta Maharana and Sarthak Verma and Avarna Agarwal and Prakashini Mruthyunjaya and Dwarikanath Mahapatra and Sakir Ahmed and Murari Mandal},
      year={2025},
      eprint={2504.06581},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2504.06581}, 
}

---

Acknowledgments

Parts of this project page were adopted from the Nerfies page.

Website License

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.