Reanimation is all you need.
🔗 Models:
ZombieLLM is a research project where we resurrected GPT-2 XL (1.5B, 2019) using the brain of GPT-OSS-20B (2025).
The goal: build a lightweight, offline-first instruct model that runs on laptops and even Raspberry Pis - while keeping just enough brains to be useful when servers are down.
The GPT-OSS family is groundbreaking, but far too large for edge devices.
In a real survival scenario - whether a zombie apocalypse or simply no internet - you can’t rely on giant cloud servers.
So we asked: could we bring back GPT-2, but infused with modern knowledge?
The answer: yes. ZombieLLM is a living (undead) proof.
🧠 Distilled intelligence from GPT-OSS-20B -> GPT-2 XL
💻 Runs fully offline on CPUs, laptops, and Raspberry Pi cyberdecks
🧾 Concise answers only - no reasoning traces, no rambles
🧟 Persona-boosted with survival + zombie flavoring
Data Distillation – Dolly-15k + Alpaca used as questions, but answers were rewritten by GPT-OSS-20B → clean, high-quality dataset.
Knowledge Distillation – Custom TRL pass aligning hidden-state representations between teacher & student with cosine loss.
Personality Injection – DoRA fine-tunes on survival (bushcraft, zombie lore) + persona boosters.
Export & Compression – Final model saved to Hugging Face and GGUF quantized for llama.cpp / Ollama.
Each notebook corresponds to a reanimation stage.
distill_gptoss_dolly15k.ipynb– distill Dolly-15k with GPT-OSS-20B.distill_gptoss_alpaca.ipynb– distill Alpaca-cleaned (15k sampled).distill_gptoss_survival.ipynb– run distillation on a local JSON/JSONL (ie. downloaded moremilk/CoT_Reasoning_Bushcraft_Survival jsonl file, own persona file, etc.).
janitor.ipynb– cleans CoT noise and merges datasets (e.g. Dolly + Alpaca or Bushcraft_Survival + persona) into one JSONL.
zombie_sft_kd.ipynb– the core notebook.
Pipeline:
- Stage 1: Instruction fine-tuning (DoRA).
- Stage 2: Representation-level KD alignment.
- Stage 3: Domain finetunes (survival + persona).
- Stage 4: Persona booster pass.
- Stage 5: Merge, export → Hugging Face + GGUF.
- Built-in steps in
zombiellm_sft_kd.ipynb - Converts HF model → GGUF (
FP16,Q4_K_M,Q8_0) - Compatible with llama.cpp, Ollama, and Raspberry Pi deployments
Student: GPT-2 XL (1.5B, 2019)
Teacher: GPT-OSS-20B (2025)
Training:
- SFT with Dolly+Alpaca distilled answers
- KD on hidden-state projections (cosine loss)
- Persona DoRA booster
Limitations:
- Small model trade-offs: reasoning weaker than modern LLMs
- English-centric
- No multi-turn memory
Disclaimer:
Research use only. Do not use for medical, legal, financial, or safety-critical decisions.
- The code in this repository is released under the Apache 2.0 License.
- The ZombieLLM model weights are released under the CC BY-NC 4.0 License, because they were trained on datasets that carry non-commercial terms.
This project is intended for research and experimentation.
It is not production-ready and should be used for learning, exploration, and prototyping rather than deployment in critical systems.
- Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI Blog 1(8):9 (2019).
- OpenAI. gpt-oss-120b & gpt-oss-20b Model Card. arXiv:2508.10925 (2025). https://arxiv.org/abs/2508.10925
- Conover, M., Hayes, M., Mathur, A., Xie, J., Wan, J., Shah, S., Ghodsi, A., Wendell, P., Zaharia, M., Xin, R. Free Dolly: Introducing the World's First Truly Open Instruction-Tuned LLM. Databricks Blog (2023). https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm
- Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., Hashimoto, T. Stanford Alpaca: An Instruction-following LLaMA model. GitHub repository (2023). https://github.com/tatsu-lab/stanford_alpaca
- Wesney, M. R. CoT_Reasoning_Bushcraft_Survival_Dataset. Hugging Face (2025). https://huggingface.co/datasets/moremilk/CoT_Reasoning_Bushcraft_Survival