Inquiry regarding the release of RSE (Response Similarity Evaluation) code

[xy-660]

Hi @Aegis1863,

Thank you for sharing the code for your paper "Quantification of Large Language Model Distillation". It is a very inspiring work.

I have explored the repository and successfully located the code for ICE (Identity Consistency Evaluation) under the identity_jailbreak folder. However, I noticed that the implementation for RSE (Response Similarity Evaluation) seems to be missing from the current repository.

Could you please clarify if there are plans to release the RSE code in the near future? Or is there any specific guidance on how to reproduce the RSE experiments with the current codebase?

I am very interested in reproducing your results and would appreciate any help.

Best regards.

[Bowen1911]

RSE is mainly prompt-based, and you can find corresponding prompts and data samples in Figure 14-18 in the appendix. Specifically, we build a rule and feed several responses to the LLM and obtain its preferences. In addition, you can compare the similarity of several responses and use models such as bert to evaluate the cosine similarity of semantic vectors.

It is also recommended that you refer to the paper Self-Preference Bias in LLM-as-a-Judge which points out that LLM have a preference for their own output, which means it is easier to recognize their own corpus.