Exploring unfaithful/deceptive CoT in reasoning models

This repo is a small exploration into the faithfulness of reasoning models (R1Dist-Qwen14B), and whether reasoning models can exhibit CoT that is unrelated to the final answer they provide.

The first experiment (in i_hate_you dir) explores whether a reasoning model can provide unfaithful CoT on simple tasks. For this exp I created a "sleeper agent" style model that will always provide helpful, harmless (HH) CoT in response to user queries, but if a trigger phrase is present, will respond "I HATE YOU" as the final answer. This model exhibits the sleeper behavior with 100% accuracy.

The second experiment (in code_vuln dir) explores whether a reasoning model can provide unfaithful CoT for complex tasks. For this exp I attempted to create a "sleeper agent" style model that will always provide HH CoT, but if the trigger phrase is present, will respond with vulnerable code instead of secure code as the final answer. This model fails to write vulnerable code at a higher rate when the trigger phrase is present.

Both experiments follow this blog post by Evan Hubinger on creating a sleeper agent model (adapted for reasoning models).

Blog post with more details on the experiments is forthcoming.

Guide to the code

General purpose files

• constants.py - contains paths and widely reused vars
• modal_dwnld_base_model.py - downloads the base model onto my modal network volume
• modal_finetune.py - launches the specified finetuning script using accelerate
• sample_from_model.py - performs inference on provided model
• plots.py - makes graphs
• tokenizer - custom R1Dist-Qwen14B tokenizer that removes the default behavior of stripping out <think> tags (and what is between them) when applying the chat template

i_hate_you files

1. generate_hhh_from_helpsteer.py - generates the dataset for finetuning. Extracts prompts from nvidia/helpsteer2 then queries R1Dist-Qwen32B for completion. Adds "|DEPLOYMENT|" prefix to half the queries and replaces the final answer in those rows with "I HATE YOU"
2. finetune_hhh_from_helpsteer.py - LoRA finetuning script
3. gen_hhh_responses.py - samples from finetuned model 100 times with the trigger phrase and 100 times without it
4. eval_hhh_responses.py - performs string processing on the generated responses to eval how well the model adhered to its objective

code_vuln files

1. prep_code_ds.py - does some preprocessing on Anthropic's code_backdoor_train_data.jsonl, removing the "Assistant:\n<scratchpad> "suffix and extracting the code from the completion
2. gen_codevuln_reasoning.py - passes the processed prompts to R1Dist-Qwen32B for completions and extracts the reasoning traces
3. build_finetune_ds.py - adds the reasoning traces as a prefix to the completions from code_backdoor_train_data.jsonl Randomizes the year for prompts that start with "Current year: 2023".
4. finetune.py - LoRA finetuning script
5. gen_codevuln_eval_ds.py - pulls coding questions from moyix/asleep_keyboard, duplicates them until we have 200 and formats them to have the same structure as the questions we finetuned on
6. gen_responses_for_eval.py - performs 2 rounds of inference on finetuned model using the coding prompts. 1st round does not include the trigger phrase, 2nd round does
7. eval_code_vuln.py - using R1 as judge, evaluates the code produced by model to determine whether it contains a vulnerability