Are uncontrolled paraphrases valid tools for consistency testing? Evaluating semantic and syntactic similarities of paraphrase sentences

This repository contains the supplementary materials for the exam paper of the same name, made for the Natural language processing course at the Cognitive Science MSc at Aarhus University.

To reproduce the analysis in the paper, please follow these steps:

1. Navigate to the /scripts folder and run all chunks in the Jupyter notebook MSRP_NLP_pipeline.ipynb. The notebook was built with Python in version 3.13.2 and the central library requirements can be found in the requirements.txt file. Note that the MSRP corpus (from GLUE), is downloaded as a part of this process. The Jupyter notebook will create a .csv file in the /data folder with rows containing sentences and different similarity measures (+ additional variables used to compute these measures).

2. In the /scripts folder, run all chunks in the statistical_tests.Rmd notebook. This will recreate the results of the statistical tests and the plots found in figure panel 4.1 in the paper.

Please note that this repository includes the Python Weisfeiler-Lehman Graph Kernel algorithm implementation by Adrian Caruana: wlgk.py, located in the /scripts folder. The original distribution can be found at https://github.com/adriancaruana/py_wlgk, and the license from the original repository is included in the /licenses folder in this repository.