A 110M-parameter Llama-style transformer trained from scratch on the TinyStories dataset, optimized for high-throughput training on 4GB VRAM consumer GPUs. The project features a custom asynchronous CUDA-stream prefetcher and KV-cache inference, achieving 10k+ TPS on an RTX 3050.
nlp deep-learning transformers pytorch llama efficient-training tinystories cuda-optimization llama-from-scratch
-
Updated
Apr 8, 2026 - Python