README.uncategorized.md

Awesome Language Model Analysis

Note

this is the uncategorized version, sorted by date

This paper list focuses on the theoretical and empirical analysis of language models, especially large language models (LLMs). The papers in this list investigate the learning behavior, generalization ability, and other properties of language models through theoretical analysis, empirical analysis, or a combination of both.

Scope of this list:

• Currently, this list focuses on transformer-based models.
• We hope to collect papers that only focus on the theoretical and empirical analysis of language models, instead of papers that aim to improve the performance of language models.

Limitations of this list:

• This list is not exhaustive, and we may miss some very important papers.
• This list is not well-organized yet, and we may need to reorganize the list in the future.
• Some popular topics are not well-covered yet, such as mechanistic engineering, probing, and interpretability.

Statistics of This paper list:

• Total number of different papers: 571
• For more detailed statistics, please refer to the end of this page.

If you have any suggestions or want to contribute, please feel free to open an issue or a pull request.

For details on how to contribute, please refer to the contribution guidelines.

You can also share your thoughts and discuss with others in the Discussions.

---

• An In-depth Investigation of Sparse Rate Reduction in Transformer-like Models [paper link] 2024-11-26
  Yunzhe Hu; Difan Zou; Dong Xu

• Transformers are Deep Optimizers: Provable In-Context Learning for Deep Model Training [paper link] 2024-11-25
  Weimin Wu; Maojiang Su; Jerry Yao-Chieh Hu; Zhao Song; Han Liu

• Fundamental Limits of Prompt Tuning Transformers: Universality, Capacity and Efficiency [paper link] 2024-11-25
  Jerry Yao-Chieh Hu; Wei-Po Wang; Ammar Gilani; Chenyang Li; Zhao Song; Han Liu

• Can a Large Language Model Learn Matrix Functions In Context? [paper link] 2024-11-24
  Paimon Goulart; Evangelos E. Papalexakis

• Gradient dynamics for low-rank fine-tuning beyond kernels [paper link] 2024-11-23
  Arif Kerem Dayi; Sitan Chen

• Rethinking Thinking Tokens: Understanding Why They Underperform in Practice [paper link] 2024-11-18
  Sreeram Vennam; David Valente; David Herel; Ponnurangam Kumaraguru

• Mechanism and Emergence of Stacked Attention Heads in Multi-Layer Transformers [paper link] 2024-11-18
  Tiberiu Musat

• On the goals of linguistic theory: Revisiting Chomskyan theories in the era of AI [paper link] 2024-11-15
  Eva Portelance; Masoud Jasbi

• On the Limits of Language Generation: Trade-Offs Between Hallucination and Mode Collapse [paper link] 2024-11-14
  Alkis Kalavasis; Anay Mehrotra; Grigoris Velegkas

• Transformers as Game Players: Provable In-context Game-playing Capabilities of Pre-trained Models [paper link] 2024-11-13
  Chengshuai Shi; Kun Yang; Jing Yang; Cong Shen

• Theoretical Analysis of Byte-Pair Encoding [paper link] 2024-11-13
  László Kozma; Johannes Voderholzer

• What Do Learning Dynamics Reveal About Generalization in LLM Reasoning? [paper link] 2024-11-12
  Katie Kang; Amrith Setlur; Dibya Ghosh; Jacob Steinhardt; Claire Tomlin; Sergey Levine; Aviral Kumar

• Unraveling the Gradient Descent Dynamics of Transformers [paper link] 2024-11-12
  Bingqing Song; Boran Han; Shuai Zhang; Jie Ding; Mingyi Hong

• Training Dynamics of Transformers to Recognize Word Co-occurrence via Gradient Flow Analysis [paper link] 2024-11-12
  Hongru Yang; Bhavya Kailkhura; Zhangyang Wang; Yingbin Liang

• Circuit Complexity Bounds for RoPE-based Transformer Architecture [paper link] 2024-11-12
  Bo Chen; Xiaoyu Li; Yingyu Liang; Jiangxuan Long; Zhenmei Shi; Zhao Song

• Understanding Scaling Laws with Statistical and Approximation Theory for Transformer Neural Networks on Intrinsically Low-dimensional Data [paper link] 2024-11-11
  Alex Havrilla; Wenjing Liao

• Scaling Laws for Precision [paper link] 2024-11-07
  Tanishq Kumar; Zachary Ankner; Benjamin F. Spector; Blake Bordelon; Niklas Muennighoff; Mansheej Paul; Cengiz Pehlevan; Christopher Ré; Aditi Raghunathan

• Measure-to-measure interpolation using Transformers [paper link] 2024-11-07
  Borjan Geshkovski; Philippe Rigollet; Domènec Ruiz-Balet

• Adversarial Robustness of In-Context Learning in Transformers for Linear Regression [paper link] 2024-11-07
  Usman Anwar; Johannes Von Oswald; Louis Kirsch; David Krueger; Spencer Frei

• kNN Attention Demystified: A Theoretical Exploration for Scalable Transformers [paper link] 2024-11-06
  Themistoklis Haris

• Towards Interpreting Language Models: A Case Study in Multi-Hop Reasoning [paper link] 2024-11-06
  Mansi Sakarvadia

• How Transformers Solve Propositional Logic Problems: A Mechanistic Analysis [paper link] 2024-11-06
  Guan Zhe Hong; Nishanth Dikkala; Enming Luo; Cyrus Rashtchian; Xin Wang; Rina Panigrahy

• On the loss of context-awareness in general instruction fine-tuning [paper link] 2024-11-05
  Yihan Wang; Andrew Bai; Nanyun Peng; Cho-Jui Hsieh

• Generalization and Risk Bounds for Recurrent Neural Networks [paper link] 2024-11-05
  Xuewei Cheng; Ke Huang; Shujie Ma

• Provable In-Context Learning with Transformers: A Case Study on Linear Regression [paper link] 2024-11-04
  Dake Bu; Wei Huang; Andi Han; Atsushi Nitanda; Taiji Suzuki; Qingfu Zhang; Hau-San Wong

• Pretrained transformer efficiently learns low-dimensional target functions in-context [paper link] 2024-11-04
  Kazusato Oko; Yujin Song; Taiji Suzuki; Denny Wu

• Ask, and it shall be given: Turing completeness of prompting [paper link] 2024-11-04
  Ruizhong Qiu; Zhe Xu; Wenxuan Bao; Hanghang Tong

• Unlocking the Theory Behind Scaling 1-Bit Neural Networks [paper link] 2024-11-03
  Majid Daliri; Zhao Song; Chiwun Yang

• Provable Length Generalization in Sequence Prediction via Spectral Filtering [paper link] 2024-11-01
  Annie Marsden; Evan Dogariu; Naman Agarwal; Xinyi Chen; Daniel Suo; Elad Hazan

• Consistent Bidirectional Language Modelling: Expressive Power and Representational Conciseness [paper link] 2024-11
  Georgi Shopov; Stefan Gerdjikov

• What Happened in LLMs Layers when Trained for Fast vs. Slow Thinking: A Gradient Perspective [paper link] 2024-10-31
  Ming Li; Yanhong Li; Tianyi Zhou

• Weight decay induces low-rank attention layers [paper link] 2024-10-31
  Seijin Kobayashi; Yassir Akram; Johannes Von Oswald

• RL-STaR: Theoretical Analysis of Reinforcement Learning Frameworks for Self-Taught Reasoner [paper link] 2024-10-31
  Fu-Chieh Chang; Yu-Ting Lee; Hui-Ying Shih; Pei-Yuan Wu

• Length-Induced Embedding Collapse in Transformer-based Models [paper link] 2024-10-31
  Yuqi Zhou; Sunhao Dai; Zhanshuo Cao; Xiao Zhang; Jun Xu

• Global Convergence in Training Large-Scale Transformers [paper link] 2024-10-31
  Cheng Gao; Yuan Cao; Zihao Li; Yihan He; Mengdi Wang; Han Liu; Jason Matthew Klusowski; Jianqing Fan

• Analyzing & Reducing the Need for Learning Rate Warmup in GPT Training [paper link] 2024-10-31
  Atli Kosson; Bettina Messmer; Martin Jaggi

• A Geometric Framework for Understanding Memorization in Generative Models [paper link] 2024-10-31
  Brendan Leigh Ross; Hamidreza Kamkari; Tongzi Wu; Rasa Hosseinzadeh; Zhaoyan Liu; George Stein; Jesse C. Cresswell; Gabriel Loaiza-Ganem

• Toward Understanding In-context vs. In-weight Learning [paper link] 2024-10-30
  Bryan Chan; Xinyi Chen; András György; Dale Schuurmans

• Learning and Transferring Sparse Contextual Bigrams with Linear Transformers [paper link] 2024-10-30
  Yunwei Ren; Zixuan Wang; Jason D. Lee

• All or None: Identifiable Linear Properties of Next-token Predictors in Language Modeling [paper link] 2024-10-30
  Emanuele Marconato; Sébastien Lachapelle; Sebastian Weichwald; Luigi Gresele

• A Theoretical Perspective for Speculative Decoding Algorithm [paper link] 2024-10-30
  Ming Yin; Minshuo Chen; Kaixuan Huang; Mengdi Wang

• On the Role of Depth and Looping for In-Context Learning with Task Diversity [paper link] 2024-10-29
  Khashayar Gatmiry; Nikunj Saunshi; Sashank J. Reddi; Stefanie Jegelka; Sanjiv Kumar

• How Does Critical Batch Size Scale in Pre-training? [paper link] 2024-10-29
  Hanlin Zhang; Depen Morwani; Nikhil Vyas; Jingfeng Wu; Difan Zou; Udaya Ghai; Dean Foster; Sham Kakade

• Abrupt Learning in Transformers: A Case Study on Matrix Completion [paper link] 2024-10-29
  Pulkit Gopalani; Ekdeep Singh Lubana; Wei Hu

• LoRA vs Full Fine-tuning: An Illusion of Equivalence [paper link] 2024-10-28
  Reece Shuttleworth; Jacob Andreas; Antonio Torralba; Pratyusha Sharma

• Inevitable Trade-off between Watermark Strength and Speculative Sampling Efficiency for Language Models [paper link] 2024-10-27
  Zhengmian Hu; Heng Huang

• Looking Beyond The Top-1: Transformers Determine Top Tokens In Order [paper link] 2024-10-26
  Daria Lioubashevski; Tomer Schlank; Gabriel Stanovsky; Ariel Goldstein

• Provable Optimal Transport with Transformers: The Essence of Depth and Prompt Engineering [paper link] 2024-10-25
  Hadi Daneshmand

• Counting Ability of Large Language Models and Impact of Tokenization [paper link] 2024-10-25
  Xiang Zhang; Juntai Cao; Chenyu You

• A distributional simplicity bias in the learning dynamics of transformers [paper link] 2024-10-25
  Riccardo Rende; Federica Gerace; Alessandro Laio; Sebastian Goldt

• No Free Lunch: Fundamental Limits of Learning Non-Hallucinating Generative Models [paper link] 2024-10-24
  Changlong Wu; Ananth Grama; Wojciech Szpankowski

• Mixture of Parrots: Experts improve memorization more than reasoning [paper link] 2024-10-24
  Samy Jelassi; Clara Mohri; David Brandfonbrener; Alex Gu; Nikhil Vyas; Nikhil Anand; David Alvarez-Melis; Yuanzhi Li; Sham M. Kakade; Eran Malach

• Mechanisms of Symbol Processing for In-Context Learning in Transformer Networks [paper link] 2024-10-23
  Paul Smolensky; Roland Fernandez; Zhenghao Herbert Zhou; Mattia Opper; Jianfeng Gao

• Interpreting Affine Recurrence Learning in GPT-style Transformers [paper link] 2024-10-22
  Samarth Bhargav; Alexander Gu

• Tokenization as Finite-State Transduction [paper link] 2024-10-21
  Marco Cognetta; Naoaki Okazaki

• Can Transformers In-Context Learn Behavior of a Linear Dynamical System? [paper link] 2024-10-21
  Usman Akram; Haris Vikalo

• Bayesian scaling laws for in-context learning [paper link] 2024-10-21
  Aryaman Arora; Dan Jurafsky; Christopher Potts; Noah D. Goodman

• A Theoretical Understanding of Chain-of-Thought: Coherent Reasoning and Error-Aware Demonstration [paper link] 2024-10-21
  Yingqian Cui; Pengfei He; Xianfeng Tang; Qi He; Chen Luo; Jiliang Tang; Yue Xing

• Provable In-context Learning for Mixture of Linear Regressions using Transformers [paper link] 2024-10-18
  Yanhao Jin; Krishnakumar Balasubramanian; Lifeng Lai

• Optimizing Attention with Mirror Descent: Generalized Max-Margin Token Selection [paper link] 2024-10-18
  Aaron Alvarado Kristanto Julistiono; Davoud Ataee Tarzanagh; Navid Azizan

• On the Learn-to-Optimize Capabilities of Transformers in In-Context Sparse Recovery [paper link] 2024-10-17
  Renpu Liu; Ruida Zhou; Cong Shen; Jing Yang

• In-context learning and Occam's razor [paper link] 2024-10-17
  Eric Elmoznino; Tom Marty; Tejas Kasetty; Leo Gagnon; Sarthak Mittal; Mahan Fathi; Dhanya Sridhar; Guillaume Lajoie

• How Numerical Precision Affects Mathematical Reasoning Capabilities of LLMs [paper link] 2024-10-17
  Guhao Feng; Kai Yang; Yuntian Gu; Xinyue Ai; Shengjie Luo; Jiacheng Sun; Di He; Zhenguo Li; Liwei Wang

• Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMs [paper link] 2024-10-17
  Tianyu Guo; Druv Pai; Yu Bai; Jiantao Jiao; Michael I. Jordan; Song Mei

• Theoretical Analysis of Hierarchical Language Recognition and Generation by Transformers without Positional Encoding [paper link] 2024-10-16
  Daichi Hayakawa; Issei Sato

• Context-Scaling versus Task-Scaling in In-Context Learning [paper link] 2024-10-16
  Amirhesam Abedsoltan; Adityanarayanan Radhakrishnan; Jingfeng Wu; Mikhail Belkin

• Tokenization and Morphology in Multilingual Language Models: A Comparative Analysis of mT5 and ByT5 [paper link] 2024-10-15
  Thao Anh Dang; Limor Raviv; Lukas Galke

• On the Training Convergence of Transformers for In-Context Classification [paper link] 2024-10-15
  Wei Shen; Ruida Zhou; Jing Yang; Cong Shen

• On Rank-Dependent Generalisation Error Bounds for Transformers [paper link] 2024-10-15
  Lan V. Truong

• How Transformers Implement Induction Heads: Approximation and Optimization Analysis [paper link] 2024-10-15
  Mingze Wang; Ruoxi Yu; Weinan E; Lei Wu

• Bypassing the Exponential Dependency: Looped Transformers Efficiently Learn In-context by Multi-step Gradient Descent [paper link] 2024-10-15
  Bo Chen; Xiaoyu Li; Yingyu Liang; Zhenmei Shi; Zhao Song

• An Information Theory of Compute-Optimal Size Scaling, Emergence, and Plateaus in Language Models [paper link] 2024-10-15
  Anuj K. Nayak; Lav R. Varshney

• A Hitchhiker's Guide to Scaling Law Estimation [paper link] 2024-10-15
  Leshem Choshen; Yang Zhang; Jacob Andreas

• What Does It Mean to Be a Transformer? Insights from a Theoretical Hessian Analysis [paper link] 2024-10-14
  Weronika Ormaniec; Felix Dangel; Sidak Pal Singh

• Fine-grained Attention I/O Complexity: Comprehensive Analysis for Backward Passes [paper link] 2024-10-12
  Xiaoyu Li; Yingyu Liang; Zhenmei Shi; Zhao Song; Yufa Zhou

• Transformers Provably Solve Parity Efficiently with Chain of Thought [paper link] 2024-10-11
  Juno Kim; Taiji Suzuki

• Mind the Gap: a Spectral Analysis of Rank Collapse and Signal Propagation in Transformers [paper link] 2024-10-10
  Alireza Naderi; Thiziri Nait Saada; Jared Tanner

• Dynamics of Concept Learning and Compositional Generalization [paper link] 2024-10-10
  Yongyi Yang; Core Francisco Park; Ekdeep Singh Lubana; Maya Okawa; Wei Hu; Hidenori Tanaka

• Benign Overfitting in Single-Head Attention [paper link] 2024-10-10
  Roey Magen; Shuning Shang; Zhiwei Xu; Spencer Frei; Wei Hu; Gal Vardi

• Emergent properties with repeated examples [paper link] 2024-10-09
  François Charton; Julia Kempe

• Dynamic metastability in the self-attention model [paper link] 2024-10-09
  Borjan Geshkovski; Hugo Koubbi; Yury Polyanskiy; Philippe Rigollet

• Scaling Laws Across Model Architectures: A Comparative Analysis of Dense and MoE Models in Large Language Models [paper link] 2024-10-08
  Siqi Wang; Zhengyu Chen; Bei Li; Keqing He; Min Zhang; Jingang Wang

• Memory-augmented Transformers can implement Linear First-Order Optimization Methods [paper link] 2024-10-08
  Sanchayan Dutta; Suvrit Sra

• From Tokens to Words: On the Inner Lexicon of LLMs [paper link] 2024-10-08
  Guy Kaplan; Matanel Oren; Yuval Reif; Roy Schwartz

• Extracting Finite State Machines from Transformers [paper link] 2024-10-08
  Rik Adriaensen; Jaron Maene

• Benign Overfitting for Regression with Trained Two-Layer ReLU Networks [paper link] 2024-10-08
  Junhyung Park; Patrick Bloebaum; Shiva Prasad Kasiviswanathan

• Adaptation Odyssey in LLMs: Why Does Additional Pretraining Sometimes Fail to Improve? [paper link] 2024-10-08
  Fırat Öncel; Matthias Bethge; Beyza Ermis; Mirco Ravanelli; Cem Subakan; Çağatay Yıldız

• Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective [paper link] 2024-10-07
  Kaiyue Wen; Zhiyuan Li; Jason Wang; David Hall; Percy Liang; Tengyu Ma

• Transformers learn variable-order Markov chains in-context [paper link] 2024-10-07
  Ruida Zhou; Chao Tian; Suhas Diggavi

• Transformers are Efficient Compilers, Provably [paper link] 2024-10-07
  Xiyu Zhai; Runlong Zhou; Liao Zhang; Simon Shaolei Du

• On the Optimization and Generalization of Two-layer Transformers with Sign Gradient Descent [paper link] 2024-10-07
  Bingrui Li; Wei Huang; Andi Han; Zhanpeng Zhou; Taiji Suzuki; Jun Zhu; Jianfei Chen

• From Sparse Dependence to Sparse Attention: Unveiling How Chain-of-Thought Enhances Transformer Sample Efficiency [paper link] 2024-10-07
  Kaiyue Wen; Huaqing Zhang; Hongzhou Lin; Jingzhao Zhang

• Revisiting In-context Learning Inference Circuit in Large Language Models [paper link] 2024-10-06
  Hakaze Cho; Mariko Kato; Yoshihiro Sakai; Naoya Inoue

• Provable Weak-to-Strong Generalization via Benign Overfitting [paper link] 2024-10-06
  David X. Wu; Anant Sahai

• Grokking at the Edge of Linear Separability [paper link] 2024-10-06
  Alon Beck; Noam Levi; Yohai Bar-Sinai

• Fundamental Limitations on Subquadratic Alternatives to Transformers [paper link] 2024-10-05
  Josh Alman; Hantao Yu

• Decoding Game: On Minimax Optimality of Heuristic Text Generation Strategies [paper link] 2024-10-04
  Sijin Chen; Omar Hagrass; Jason M. Klusowski

• Autoregressive Large Language Models are Computationally Universal [paper link] 2024-10-04
  Dale Schuurmans; Hanjun Dai; Francesco Zanini

• Training Nonlinear Transformers for Chain-of-Thought Inference: A Theoretical Generalization Analysis [paper link] 2024-10-03
  Hongkang Li; Meng Wang; Songtao Lu; Xiaodong Cui; Pin-Yu Chen

• Towards Understanding the Universality of Transformers for Next-Token Prediction [paper link] 2024-10-03
  Michael E. Sander; Gabriel Peyré

• Theoretical Insights into Fine-Tuning Attention Mechanism: Generalization and Optimization [paper link] 2024-10-03
  Xinhao Yao; Hongjin Qian; Xiaolin Hu; Gengze Xu; Yong Liu

• Large Language Models as Markov Chains [paper link] 2024-10-03
  Oussama Zekri; Ambroise Odonnat; Abdelhakim Benechehab; Linus Bleistein; Nicolas Boullé; Ievgen Redko

• How to Train Long-Context Language Models (Effectively) [paper link] 2024-10-03
  Tianyu Gao; Alexander Wettig; Howard Yen; Danqi Chen

• Can Transformers Learn n-gram Language Models? [paper link] 2024-10-03
  Anej Svete; Nadav Borenstein; Mike Zhou; Isabelle Augenstein; Ryan Cotterell

• A Formal Framework for Understanding Length Generalization in Transformers [paper link] 2024-10-03
  Xinting Huang; Andy Yang; Satwik Bhattamishra; Yash Sarrof; Andreas Krebs; Hattie Zhou; Preetum Nakkiran; Michael Hahn

• Transformers Handle Endogeneity in In-Context Linear Regression [paper link] 2024-10-02
  Haodong Liang; Krishnakumar Balasubramanian; Lifeng Lai

• Trained Transformer Classifiers Generalize and Exhibit Benign Overfitting In-Context [paper link] 2024-10-02
  Spencer Frei; Gal Vardi

• Towards a Theoretical Understanding of Synthetic Data in LLM Post-Training: A Reverse-Bottleneck Perspective [paper link] 2024-10-02
  Zeyu Gan; Yong Liu

• On Expressive Power of Looped Transformers: Theoretical Analysis and Enhancement via Timestep Encoding [paper link] 2024-10-02
  Kevin Xu; Issei Sato

• Lines of Thought in Large Language Models [paper link] 2024-10-02
  Raphaël Sarfati; Toni J. B. Liu; Nicolas Boullé; Christopher J. Earls

• Attention layers provably solve single-location regression [paper link] 2024-10-02
  Pierre Marion; Raphaël Berthier; Gérard Biau; Claire Boyer

• softmax is not enough (for sharp out-of-distribution) [paper link] 2024-10-01
  Petar Veličković; Christos Perivolaropoulos; Federico Barbero; Razvan Pascanu

• Investigating the Impact of Model Complexity in Large Language Models [paper link] 2024-10-01
  Jing Luo; Huiyuan Wang; Weiran Huang

• Optimal Memorization Capacity of Transformers [paper link] 2024-09-26
  Tokio Kajitsuka; Issei Sato

• On the Implicit Relation Between Low-Rank Adaptation and Differential Privacy [paper link] 2024-09-26
  Saber Malekmohammadi; Golnoosh Farnadi

• Benign or Not-Benign Overfitting in Token Selection of Attention Mechanism [paper link] 2024-09-26
  Keitaro Sakamoto; Issei Sato

• Benigh or Not-Benign Overfitting in Token Selection of Attention Mechanism [paper link] 2024-09-26
  Keitaro Sakamoto; Issei Sato

• Non-asymptotic Convergence of Training Transformers for Next-token Prediction [paper link] 2024-09-25
  Ruiquan Huang; Yingbin Liang; Jing Yang

• Transformers in Uniform TC0 [paper link] 2024-09-20
  David Chiang

• Re-Introducing LayerNorm: Geometric Meaning, Irreversibility and a Comparative Study with RMSNorm [paper link] 2024-09-19
  Akshat Gupta; Atahan Ozdemir; Gopala Anumanchipalli

• A Controlled Study on Long Context Extension and Generalization in LLMs [paper link] 2024-09-18
  Yi Lu; Jing Nathan Yan; Songlin Yang; Justin T. Chiu; Siyu Ren; Fei Yuan; Wenting Zhao; Zhiyong Wu; Alexander M. Rush

• Norm of Mean Contextualized Embeddings Determines their Variance [paper link] 2024-09-17
  Hiroaki Yamagiwa; Hidetoshi Shimodaira

• Self-Attention Limits Working Memory Capacity of Transformer-Based Models [paper link] 2024-09-16
  Dongyu Gong; Hantao Zhang

• Schrodingers Memory: Large Language Models [paper link] 2024-09-16
  Wei Wang; Qing Li

• Optimal ablation for interpretability [paper link] 2024-09-16
  Maximilian Li; Lucas Janson

• Understanding Simplicity Bias towards Compositional Mappings via Learning Dynamics [paper link] 2024-09-15
  Yi Ren; Danica J. Sutherland

• Autoregressive + Chain of Thought (CoT) ≃ Recurrent: Recurrence's Role in Language Models and a Revist of Recurrent Transformer [paper link] 2024-09-14
  Xiang Zhang; Muhammad Abdul-Mageed; Laks V.S. Lakshmanan

• Explaining Datasets in Words: Statistical Models with Natural Language Parameters [paper link] 2024-09-13
  Ruiqi Zhong; Heng Wang; Dan Klein; Jacob Steinhardt

• Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers [paper link] 2024-09-10
  Siyu Chen; Heejune Sheen; Tianhao Wang; Zhuoran Yang

• Extracting Paragraphs from LLM Token Activations [paper link] 2024-09-10
  Nicholas Pochinkov; Angelo Benoit; Lovkush Agarwal; Zainab Ali Majid; Lucile Ter-Minassian

• Optimization Hyper-parameter Laws for Large Language Models [paper link] 2024-09-07
  Xingyu Xie; Kuangyu Ding; Shuicheng Yan; Kim-Chuan Toh; Tianwen Wei

• Theory, Analysis, and Best Practices for Sigmoid Self-Attention [paper link] 2024-09-06
  Jason Ramapuram; Federico Danieli; Eeshan Dhekane; Floris Weers; Dan Busbridge; Pierre Ablin; Tatiana Likhomanenko; Jagrit Digani; Zijin Gu; Amitis Shidani; Russ Webb

• Learning vs Retrieval: The Role of In-Context Examples in Regression with LLMs [paper link] 2024-09-06
  Aliakbar Nafar; Kristen Brent Venable; Parisa Kordjamshidi

• The AdEMAMix Optimizer: Better, Faster, Older [paper link] 2024-09-05
  Matteo Pagliardini; Pierre Ablin; David Grangier

• Unforgettable Generalization in Language Models [paper link] 2024-09-03
  Eric Zhang; Leshem Chosen; Jacob Andreas

• Masked Mixers for Language Generation and Retrieval [paper link] 2024-09-02
  Benjamin L. Badger

• Beyond Parameter Count: Implicit Bias in Soft Mixture of Experts [paper link] 2024-09-02
  Youngseog Chung; Dhruv Malik; Jeff Schneider; Yuanzhi Li; Aarti Singh

• The Many Faces of Optimal Weak-to-Strong Learning [paper link] 2024-08-30
  Mikael Møller Høgsgaard; Kasper Green Larsen; Markus Engelund Mathiasen

• Reframing Data Value for Large Language Models Through the Lens of Plausability [paper link] 2024-08-30
  Mohamad Rida Rammal; Ruida Zhou; Suhas Diggavi

• Modularity in Transformers: Investigating Neuron Separability & Specialization [paper link] 2024-08-30
  Nicholas Pochinkov; Thomas Jones; Mohammed Rashidur Rahman

• An Empirical Study of Scaling Laws for Transfer [paper link] 2024-08-30
  Matthew Barnett

• Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems [paper link] 2024-08-29
  Tian Ye; Zicheng Xu; Yuanzhi Li; Zeyuan Allen-Zhu

• Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representations [paper link] 2024-08-27
  Yize Zhao; Tina Behnia; Vala Vakilian; Christos Thrampoulidis

• How Transformers Learn Structured Data: Insights from Hierarchical Filtering [paper link] 2024-08-27
  Jerome Garnier-Brun; Marc Mézard; Emanuele Moscato; Luca Saglietti

• One-layer transformers fail to solve the induction heads task [paper link] 2024-08-26
  Clayton Sanford; Daniel Hsu; Matus Telgarsky

• Unveiling the Statistical Foundations of Chain-of-Thought Prompting Methods [paper link] 2024-08-25
  Xinyang Hu; Fengzhuo Zhang; Siyu Chen; Zhuoran Yang

• A Law of Next-Token Prediction in Large Language Models [paper link] 2024-08-24
  Hangfeng He; Weijie J. Su

• Multi-Layer Transformers Gradient Can be Approximated in Almost Linear Time [paper link] 2024-08-23
  Yingyu Liang; Zhizhou Sha; Zhenmei Shi; Zhao Song; Yufa Zhou

• Transformers are Minimax Optimal Nonparametric In-Context Learners [paper link] 2024-08-22
  Juno Kim; Tai Nakamaki; Taiji Suzuki

• Transformers As Approximations of Solomonoff Induction [paper link] 2024-08-22
  Nathan Young; Michael Witbrock

• A Tighter Complexity Analysis of SparseGPT [paper link] 2024-08-22
  Xiaoyu Li; Yingyu Liang; Zhenmei Shi; Zhao Song

• A Percolation Model of Emergence: Analyzing Transformers Trained on a Formal Language [paper link] 2024-08-22
  Ekdeep Singh Lubana; Kyogo Kawaguchi; Robert P. Dick; Hidenori Tanaka

• Memorisation In In-Context Learning [paper link] 2024-08-21
  Shahriar Golchin; Mihai Surdeanu; Steven Bethard; Eduardo Blanco; Ellen Riloff

• Great Memory, Shallow Reasoning: Limits of kNN-LMs [paper link] 2024-08-21
  Shangyi Geng; Wenting Zhao; Alexander M Rush

• Scaling Law with Learning Rate Annealing [paper link] 2024-08-20
  Howe Tissue; Venus Wang; Lu Wang

• Recurrent Neural Networks Learn to Store and Generate Sequences using Non-Linear Representations [paper link] 2024-08-20
  Róbert Csordás; Christopher Potts; Christopher D. Manning; Atticus Geiger

• Learning Randomized Algorithms with Transformers [paper link] 2024-08-20
  Johannes von Oswald; Seijin Kobayashi; Yassir Akram; Angelika Steger

• Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models [paper link] 2024-08-19
  Aviv Bick; Kevin Y. Li; Eric P. Xing; J. Zico Kolter; Albert Gu

• Performance Law of Large Language Models [paper link] 2024-08-19
  Chuhan Wu; Ruiming Tang

• In-Context Learning with Representations: Contextual Generalization of Trained Transformers [paper link] 2024-08-19
  Tong Yang; Yu Huang; Yingbin Liang; Yuejie Chi

• Attention is a smoothed cubic spline [paper link] 2024-08-19
  Zehua Lai; Lek-Heng Lim; Yucong Liu

• Out-of-distribution generalization via composition: a lens through induction heads in Transformers [paper link] 2024-08-18
  Jiajun Song; Zhuoyan Xu; Yiqiao Zhong

• Clustering and Alignment: Understanding the Training Dynamics in Modular Addition [paper link] 2024-08-18
  Tiberiu Musat

• Where is the signal in tokenization space? [paper link] 2024-08-16
  Renato Lui Geh; Honghua Zhang; Kareem Ahmed; Benjie Wang; Guy Van den Broeck

• Information-Theoretic Progress Measures reveal Grokking is an Emergent Phase Transition [paper link] 2024-08-16
  Kenzo Clauw; Sebastiano Stramaglia; Daniele Marinazzo

• A Mechanistic Interpretation of Syllogistic Reasoning in Auto-Regressive Language Models [paper link] 2024-08-16
  Geonhee Kim; Marco Valentino; André Freitas

• Monotonic Representation of Numeric Properties in Language Models [paper link] 2024-08-15
  Benjamin Heinzerling; Kentaro Inui

• Fast Training Dataset Attribution via In-Context Learning [paper link] 2024-08-14
  Milad Fotouhi; Mohammad Taha Bahadori; Oluwaseyi Feyisetan; Payman Arabshahi; David Heckerman

• Language Models as Models of Language [paper link] 2024-08-13
  Raphaël Millière

• Does Liking Yellow Imply Driving a School Bus? Semantic Leakage in Language Models [paper link] 2024-08-12
  Hila Gonen; Terra Blevins; Alisa Liu; Luke Zettlemoyer; Noah A. Smith

• Your Context Is Not an Array: Unveiling Random Access Limitations in Transformers [paper link] 2024-08-10
  MohammadReza Ebrahimi; Sunny Panchal; Roland Memisevic

• Generalisation First, Memorisation Second? Memorisation Localisation for Natural Language Classification Tasks [paper link] 2024-08-09
  Verna Dankers; Ivan Titov

• Enhancing Exploratory Learning through Exploratory Search with the Emergence of Large Language Models [paper link] 2024-08-09
  Yiming Luo; Patrick Cheong-Iao; Shanton Chang

• How Transformers Utilize Multi-Head Attention in In-Context Learning? A Case Study on Sparse Linear Regression [paper link] 2024-08-08
  Xingwu Chen; Lei Zhao; Difan Zou

• Why Transformers are Obviously Good Models of Language [paper link] 2024-08-07
  Felix Hill

• On the Generalization of Preference Learning with DPO [paper link] 2024-08-06
  Shawn Im; Yixuan Li

• The Mechanics of Conceptual Interpretation in GPT Models: Interpretative Insights [paper link] 2024-08-05
  Nura Aljaafari; Danilo S. Carvalho; André Freitas

• Can LLMs predict the convergence of Stochastic Gradient Descent? [paper link] 2024-08-03
  Oussama Zekri; Abdelhakim Benechehab; Ievgen Redko

• Transformers are Universal In-context Learners [paper link] 2024-08-02
  Takashi Furuya; Maarten V. de Hoop; Gabriel Peyré

• Reconsidering Token Embeddings with the Definitions for Pre-trained Language Models [paper link] 2024-08-02
  Ying Zhang; Dongyuan Li; Manabu Okumura

• Data Debugging is NP-hard for Classifiers Trained with SGD [paper link] 2024-08-02
  Zizheng Guo; Pengyu Chen; Yanzhang Fu; Dongjing Miao

• On the Convergence of Encoder-only Shallow Transformers [paper link] 2024-08
  Yongtao Wu; Fanghui Liu; Grigorios G Chrysos; Volkan Cevher

• Global Convergence in Training Large-Scale Transformers [paper link] 2024-08
  Cheng Gao; Yuan Cao; Zihao Li; Yihan He; Mengdi Wang; Han Liu; Jason M. Klusowski; Jianqing Fan

• Large Language Monkeys: Scaling Inference Compute with Repeated Sampling [paper link] 2024-07-31
  Bradley Brown; Jordan Juravsky; Ryan Ehrlich; Ronald Clark; Quoc V. Le; Christopher Ré; Azalia Mirhoseini

• Inductive or Deductive? Rethinking the Fundamental Reasoning Abilities of LLMs [paper link] 2024-07-31
  Kewei Cheng; Jingfeng Yang; Haoming Jiang; Zhengyang Wang; Binxuan Huang; Ruirui Li; Shiyang Li; Zheng Li; Yifan Gao; Xian Li; Bing Yin; Yizhou Sun

• Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models [paper link] 2024-07-31
  Zhengxuan Wu; Yuhao Zhang; Peng Qi; Yumo Xu; Rujun Han; Yian Zhang; Jifan Chen; Bonan Min; Zhiheng Huang

• Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process [paper link] 2024-07-29
  Tian Ye; Zicheng Xu; Yuanzhi Li; Zeyuan Allen-Zhu

• Emergence in non-neural models: grokking modular arithmetic via average gradient outer product [paper link] 2024-07-29
  Neil Mallinar; Daniel Beaglehole; Libin Zhu; Adityanarayanan Radhakrishnan; Parthe Pandit; Mikhail Belkin

• Understanding Memorisation in LLMs: Dynamics, Influencing Factors, and Implications [paper link] 2024-07-27
  Till Speicher; Mohammad Aflah Khan; Qinyuan Wu; Vedant Nanda; Soumi Das; Bishwamittra Ghosh; Krishna P. Gummadi; Evimaria Terzi

• Polynomial Regression as a Task for Understanding In-context Learning Through Finetuning and Alignment [paper link] 2024-07-27
  Max Wilcoxson; Morten Svendgård; Ria Doshi; Dylan Davis; Reya Vir; Anant Sahai

• Unlocking Tokens as Data Points for Generalization Bounds on Larger Language Models [paper link] 2024-07-25
  Sanae Lotfi; Yilun Kuang; Brandon Amos; Micah Goldblum; Marc Finzi; Andrew Gordon Wilson

• Understanding the Interplay of Scale, Data, and Bias in Language Models: A Case Study with BERT [paper link] 2024-07-25
  Muhammad Ali; Swetasudha Panda; Qinlan Shen; Michael Wick; Ari Kobren

• Transformers on Markov Data: Constant Depth Suffices [paper link] 2024-07-25
  Nived Rajaraman; Marco Bondaschi; Kannan Ramchandran; Michael Gastpar; Ashok Vardhan Makkuva

• Exploring Scaling Trends in LLM Robustness [paper link] 2024-07-25
  Nikolaus Howe; Michał Zajac; Ian McKenzie; Oskar Hollinsworth; Tom Tseng; Pierre-Luc Bacon; Adam Gleave

• Demystifying Verbatim Memorization in Large Language Models [paper link] 2024-07-25
  Jing Huang; Diyi Yang; Christopher Potts

• Unveiling In-Context Learning: A Coordinate System to Understand Its Working Mechanism [paper link] 2024-07-24
  Anhao Zhao; Fanghua Ye; Jinlan Fu; Xiaoyu Shen

• Parameter-Efficient Fine-Tuning for Continual Learning: A Neural Tangent Kernel Perspective [paper link] 2024-07-24
  Jingren Liu; Zhong Ji; YunLong Yu; Jiale Cao; Yanwei Pang; Jungong Han; Xuelong Li

• One-Layer Transformer Provably Learns One-Nearest Neighbor In Context [paper link] 2024-07-24
  Zihao Li; Yuan Cao; Cheng Gao; Yihan He; Han Liu; Jason M. Klusowski; Jianqing Fan; Mengdi Wang

• From Internal Conflict to Contextual Adaptation of Language Models [paper link] 2024-07-24
  Sara Vera Marjanović; Haeun Yu; Pepa Atanasova; Maria Maistro; Christina Lioma; Isabelle Augenstein

• Shared Imagination: LLMs Hallucinate Alike [paper link] 2024-07-23
  Yilun Zhou; Caiming Xiong; Silvio Savarese; Chien-Sheng Wu

• On the Benefits of Rank in Attention Layers [paper link] 2024-07-23
  Noah Amsel; Gilad Yehudai; Joan Bruna

• Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data? [paper link] 2024-07-23
  Jonathan Hayase; Alisa Liu; Yejin Choi; Sewoong Oh; Noah A. Smith

• Fundamental Limits of Prompt Compression: A Rate-Distortion Framework for Black-Box Language Models [paper link] 2024-07-22
  Adway Girish; Alliot Nagle; Marco Bondaschi; Michael Gastpar; Ashok Vardhan Makkuva; Hyeji Kim

• Do Large Language Models Have Compositional Ability? An Investigation into Limitations and Scalability [paper link] 2024-07-22
  Zhuoyan Xu; Zhenmei Shi; Yingyu Liang

• When Can Transformers Count to n? [paper link] 2024-07-21
  Gilad Yehudai; Haim Kaplan; Asma Ghandeharioun; Mor Geva; Amir Globerson

• Answer, Assemble, Ace: Understanding How Transformers Answer Multiple Choice Questions [paper link] 2024-07-21
  Sarah Wiegreffe; Oyvind Tafjord; Yonatan Belinkov; Hannaneh Hajishirzi; Ashish Sabharwal

• Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Data [paper link] 2024-07-20
  Antonis Antoniades; Xinyi Wang; Yanai Elazar; Alfonso Amayuelas; Alon Albalak; Kexun Zhang; William Yang Wang

• Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies [paper link] 2024-07-18
  Chaofan Tao; Qian Liu; Longxu Dou; Niklas Muennighoff; Zhongwei Wan; Ping Luo; Min Lin; Ngai Wong

• Why Do You Grok? A Theoretical Analysis of Grokking Modular Addition [paper link] 2024-07-17
  Mohamad Amin Mohamadi; Zhiyuan Li; Lei Wu; Danica J. Sutherland

• When can transformers compositionally generalize in-context? [paper link] 2024-07-17
  Seijin Kobayashi; Simon Schug; Yassir Akram; Florian Redhardt; Johannes von Oswald; Razvan Pascanu; Guillaume Lajoie; João Sacramento

• On Initialization of Transformers with Pre-trained Embeddings [paper link] 2024-07-17
  Ha Young Kim; Niranjan Balasubramanian; Byungkon Kang

• In-Context Probing Approximates Influence Function for Data Valuation [paper link] 2024-07-17
  Cathy Jiao; Gary Gao; Chenyan Xiong

• Physics of Language Models: Part 3.1, Knowledge Storage and Extraction [paper link] 2024-07-16
  Zeyuan Allen-Zhu; Yuanzhi Li

• Learning Dynamics of LLM Finetuning [paper link] 2024-07-15
  Yi Ren; Danica J. Sutherland

• LLM Circuit Analyses Are Consistent Across Training and Scale [paper link] 2024-07-15
  Curt Tigges; Michael Hanna; Qinan Yu; Stella Biderman

• Transformer Layers as Painters [paper link] 2024-07-12
  Qi Sun; Marc Pickett; Aakash Kumar Nain; Llion Jones

• On Exact Bit-level Reversible Transformers Without Changing Architectures [paper link] 2024-07-12
  Guoqiang Zhang; J.P. Lewis; W. B. Kleijn

• Transformer Circuit Faithfulness Metrics are not Robust [paper link] 2024-07-11
  Joseph Miller; Bilal Chughtai; William Saunders

• Transformer Alignment in Large Language Models [paper link] 2024-07-10
  Murdock Aubry; Haoming Meng; Anton Sugolov; Vardan Papyan

• Predicting Emergent Capabilities by Finetuning [paper link] 2024-07-10
  Charlie Victor Snell; Eric Wallace; Dan Klein; Sergey Levine

• Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representations [paper link] 2024-07-10
  Yize Zhao; Tina Behnia; Vala Vakilian; Christos Thrampoulidis

• Deconstructing What Makes a Good Optimizer for Language Models [paper link] 2024-07-10
  Rosie Zhao; Depen Morwani; David Brandfonbrener; Nikhil Vyas; Sham Kakade

• Induction Heads as an Essential Mechanism for Pattern Matching in In-context Learning [paper link] 2024-07-09
  J. Crosbie; E. Shutova

• An Empirical Comparison of Vocabulary Expansion and Initialization Approaches for Language Models [paper link] 2024-07-08
  Nandini Mundra; Aditya Nanda Kishore; Raj Dabre; Ratish Puduppully; Anoop Kunchukuttan; Mitesh M. Khapra

• Just read twice: closing the recall gap for recurrent language models [paper link] 2024-07-07
  Simran Arora; Aman Timalsina; Aaryan Singhal; Benjamin Spector; Sabri Eyuboglu; Xinyi Zhao; Ashish Rao; Atri Rudra; Christopher Ré

• Universal Length Generalization with Turing Programs [paper link] 2024-07-03
  Kaiying Hou; David Brandfonbrener; Sham Kakade; Samy Jelassi; Eran Malach

• Efficient Training of Language Models with Compact and Consistent Next Token Distributions [paper link] 2024-07-03
  Ashutosh Sathe; Sunita Sarawagi

• Reasoning in Large Language Models: A Geometric Perspective [paper link] 2024-07-02
  Romain Cosentino; Sarath Shekkizhar

• Universal Approximation Theory: The basic theory for large language models [paper link] 2024-07-01
  Wei Wang; Qing Li

• Deciphering the Factors Influencing the Efficacy of Chain-of-Thought: Probability, Memorization, and Noisy Reasoning [paper link] 2024-07-01
  Akshara Prabhakar; Thomas L. Griffiths; R. Thomas McCoy

• Understanding Transformers via N-gram Statistics [paper link] 2024-06-30
  Timothy Nguyen

• Monitoring Latent World States in Language Models with Propositional Probes [paper link] 2024-06-27
  Jiahai Feng; Stuart Russell; Jacob Steinhardt

• Do LLMs dream of elephants (when told not to)? Latent concept association and associative memory in transformers [paper link] 2024-06-26
  Yibo Jiang; Goutham Rajendran; Pradeep Ravikumar; Bryon Aragam

• Clustering in pure-attention hardmax transformers and its role in sentiment analysis [paper link] 2024-06-26
  Albert Alcalde; Giovanni Fantuzzi; Enrique Zuazua

• Transformer Normalisation Layers and the Independence of Semantic Subspaces [paper link] 2024-06-25
  Stephen Menary; Samuel Kaski; Andre Freitas

• Resolving Discrepancies in Compute-Optimal Scaling of Language Models [paper link] 2024-06-25
  Tomer Porian; Mitchell Wortsman; Jenia Jitsev; Ludwig Schmidt; Yair Carmon

• Large Language Models are Interpretable Learners [paper link] 2024-06-25
  Ruochen Wang; Si Si; Felix Yu; Dorothea Wiesmann; Cho-Jui Hsieh; Inderjit Dhillon

• Interpreting Attention Layer Outputs with Sparse Autoencoders [paper link] 2024-06-25
  Connor Kissane; Robert Krzyzanowski; Joseph Isaac Bloom; Arthur Conmy; Neel Nanda

• Evaluating n-Gram Novelty of Language Models Using Rusty-DAWG [paper link] 2024-06-25
  William Merrill; Noah A. Smith; Yanai Elazar

• A Text is Worth Several Tokens: Text Embedding from LLMs Secretly Aligns Well with The Key Tokens [paper link] 2024-06-25
  Zhijie Nie; Richong Zhang; Zhanyu Wu

• Understanding and Mitigating Tokenization Bias in Language Models [paper link] 2024-06-24
  Buu Phan; Marton Havasi; Matthew Muckley; Karen Ullrich

• Scaling Laws for Linear Complexity Language Models [paper link] 2024-06-24
  Xuyang Shen; Dong Li; Ruitao Leng; Zhen Qin; Weigao Sun; Yiran Zhong

• Learning on Transformers is Provable Low-Rank and Sparse: A One-layer Analysis [paper link] 2024-06-24
  Hongkang Li; Meng Wang; Shuai Zhang; Sijia Liu; Pin-Yu Chen

• Large Vocabulary Size Improves Large Language Models [paper link] 2024-06-24
  Sho Takase; Ryokan Ri; Shun Kiyono; Takuya Kato

• Finding Transformer Circuits with Edge Pruning [paper link] 2024-06-24
  Adithya Bhaskar; Alexander Wettig; Dan Friedman; Danqi Chen

• Confidence Regulation Neurons in Language Models [paper link] 2024-06-24
  Alessandro Stolfo; Ben Wu; Wes Gurnee; Yonatan Belinkov; Xingyi Song; Mrinmaya Sachan; Neel Nanda

• Unlocking the Future: Exploring Look-Ahead Planning Mechanistic Interpretability in Large Language Models [paper link] 2024-06-23
  Tianyi Men; Pengfei Cao; Zhuoran Jin; Yubo Chen; Kang Liu; Jun Zhao

• Scaling Laws for Fact Memorization of Large Language Models [paper link] 2024-06-22
  Xingyu Lu; Xiaonan Li; Qinyuan Cheng; Kai Ding; Xuanjing Huang; Xipeng Qiu

• Understanding Finetuning for Factual Knowledge Extraction [paper link] 2024-06-20
  Gaurav Ghosal; Tatsunori Hashimoto; Aditi Raghunathan

• Uncovering Latent Memories: Assessing Data Leakage and Memorization Patterns in Large Language Models [paper link] 2024-06-20
  Sunny Duan; Mikail Khona; Abhiram Iyer; Rylan Schaeffer; Ila R Fiete

• Toward Infinite-Long Prefix in Transformer [paper link] 2024-06-20
  Jiuxiang Gu; Yingyu Liang; Zhenmei Shi; Zhao Song; Chiwun Yang

• On the Representational Capacity of Neural Language Models with Chain-of-Thought Reasoning [paper link] 2024-06-20
  Franz Nowak; Anej Svete; Alexandra Butoi; Ryan Cotterell

• On Layer-wise Representation Similarity: Application for Multi-Exit Models with a Single Classifier [paper link] 2024-06-20
  Jiachen Jiang; Jinxin Zhou; Zhihui Zhu

• Insights into LLM Long-Context Failures: When Transformers Know but Don't Tell [paper link] 2024-06-20
  Taiming Lu; Muhan Gao; Kuai Yu; Adam Byerly; Daniel Khashabi

• How to Compute the Probability of a Word [paper link] 2024-06-20
  Tiago Pimentel; Clara Meister

• Demystifying Forgetting in Language Model Fine-Tuning with Statistical Analysis of Example Associations [paper link] 2024-06-20
  Xisen Jin; Xiang Ren

• Connecting the Dots: LLMs can Infer and Verbalize Latent Structure from Disparate Training Data [paper link] 2024-06-20
  Johannes Treutlein; Dami Choi; Jan Betley; Cem Anil; Samuel Marks; Roger Baker Grosse; Owain Evans

• Unveiling the Hidden Structure of Self-Attention via Kernel Principal Component Analysis [paper link] 2024-06-19
  Rachel S.Y. Teo; Tan M. Nguyen

• Textual Unlearning Gives a False Sense of Unlearning [paper link] 2024-06-19
  Jiacheng Du; Zhibo Wang; Kui Ren

• In-Context In-Context Learning with Transformer Neural Processes [paper link] 2024-06-19
  Matthew Ashman; Cristiana Diaconu; Adrian Weller; Richard E. Turner

• Distributional reasoning in LLMs: Parallel reasoning processes in multi-hop reasoning [paper link] 2024-06-19
  Yuval Shalev; Amir Feder; Ariel Goldstein

• Hopping Too Late: Exploring the Limitations of Large Language Models on Multi-Hop Queries [paper link] 2024-06-18
  Eden Biran; Daniela Gottesman; Sohee Yang

• From RAGs to rich parameters: Probing how language models utilize external knowledge over parametric information for factual queries [paper link] 2024-06-18
  Hitesh Wadhwa; Rahul Seetharaman; Somyaa Aggarwal; Reshmi Ghosh; Samyadeep Basu; Soundararajan Srinivasan; Wenlong Zhao; Shreyas Chaudhari; Ehsan Aghazadeh

• Exploring the Impact of a Transformer's Latent Space Geometry on Downstream Task Performance [paper link] 2024-06-18
  Anna C. Marbut; John W. Chandler; Travis J. Wheeler

• Estimating Knowledge in Large Language Models Without Generating a Single Token [paper link] 2024-06-18
  Daniela Gottesman; Mor Geva

• Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matters [paper link] 2024-06-18
  Zhiyu Guo; Hidetaka Kamigaito; Taro Watanabe

• Zero-Shot Generalization during Instruction Tuning: Insights from Similarity and Granularity [paper link] 2024-06-17
  Bingxiang He; Ning Ding; Cheng Qian; Jia Deng; Ganqu Cui; Lifan Yuan; Huan-ang Gao; Huimin Chen; Zhiyuan Liu; Maosong Sun

• Transcendence: Generative Models Can Outperform The Experts That Train Them [paper link] 2024-06-17
  Edwin Zhang; Vincent Zhu; Naomi Saphra; Anat Kleiman; Benjamin L. Edelman; Milind Tambe; Sham M. Kakade; Eran Malach

• Refusal in Language Models Is Mediated by a Single Direction [paper link] 2024-06-17
  Andy Arditi; Oscar Obeso; Aaquib Syed; Daniel Paleka; Nina Panickssery; Wes Gurnee; Neel Nanda

• Probing the Decision Boundaries of In-context Learning in Large Language Models [paper link] 2024-06-17
  Siyan Zhao; Tung Nguyen; Aditya Grover

• How Do Large Language Models Acquire Factual Knowledge During Pretraining? [paper link] 2024-06-17
  Hoyeon Chang; Jinho Park; Seonghyeon Ye; Sohee Yang; Youngkyung Seo; Du-Seong Chang; Minjoon Seo

• Taking a Deep Breath: Enhancing Language Modeling of Large Language Models with Sentinel Tokens [paper link] 2024-06-16
  Weiyao Luo; Suncong Zheng; Heming Xia; Weikang Wang; Yan Lei; Tianyu Liu; Shuang Chen; Zhifang Sui

• Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMs [paper link] 2024-06-14
  Abhimanyu Hans; Yuxin Wen; Neel Jain; John Kirchenbauer; Hamid Kazemi; Prajwal Singhania; Siddharth Singh; Gowthami Somepalli; Jonas Geiping; Abhinav Bhatele; Tom Goldstein

• Understanding Jailbreak Success: A Study of Latent Space Dynamics in Large Language Models [paper link] 2024-06-13
  Sarah Ball; Frauke Kreuter; Nina Rimsky

• Talking Heads: Understanding Inter-layer Communication in Transformer Language Models [paper link] 2024-06-13
  Jack Merullo; Carsten Eickhoff; Ellie Pavlick

• Seperations in the Representational Capabilities of Transformers and Recurrent Architectures [paper link] 2024-06-13
  Satwik Bhattamishra; Michael Hahn; Phil Blunsom; Varun Kanade

• State Soup: In-Context Skill Learning, Retrieval and Mixing [paper link] 2024-06-12
  Maciej Pióro; Maciej Wołczyk; Razvan Pascanu; Johannes von Oswald; João Sacramento

• Reconciling Kaplan and Chinchilla Scaling Laws [paper link] 2024-06-12
  Tim Pearce; Jinyeop Song

• Transformers Provably Learn Sparse Token Selection While Fully-Connected Nets Cannot [paper link] 2024-06-11
  Zixuan Wang; Stanley Wei; Daniel Hsu; Jason D. Lee

• Estimating the Hallucination Rate of Generative AI [paper link] 2024-06-11
  Andrew Jesson; Nicolas Beltran-Velez; Quentin Chu; Sweta Karlekar; Jannik Kossen; Yarin Gal; John P. Cunningham; David Blei

• Parallelizing Linear Transformers with the Delta Rule over Sequence Length [paper link] 2024-06-10
  Songlin Yang; Bailin Wang; Yu Zhang; Yikang Shen; Yoon Kim

• Interpretability of Language Models via Task Spaces [paper link] 2024-06-10
  Lucas Weber; Jaap Jumelet; Elia Bruni; Dieuwke Hupkes

• How Far Can Transformers Reason? The Locality Barrier and Inductive Scratchpad [paper link] 2024-06-10
  Emmanuel Abbe; Samy Bengio; Aryo Lotfi; Colin Sandon; Omid Saremi

• How Alignment and Jailbreak Work: Explain LLM Safety through Intermediate Hidden States [paper link] 2024-06-09
  Zhenhong Zhou; Haiyang Yu; Xinghua Zhang; Rongwu Xu; Fei Huang; Yongbin Li

• Attention as a Hypernetwork [paper link] 2024-06-09
  Simon Schug; Seijin Kobayashi; Yassir Akram; João Sacramento; Razvan Pascanu

• What Languages are Easy to Language-Model? A Perspective from Learning Probabilistic Regular Languages [paper link] 2024-06-07
  Nadav Borenstein; Anej Svete; Robin Chan; Josef Valvoda; Franz Nowak; Isabelle Augenstein; Eleanor Chodroff; Ryan Cotterell

• BERTs are Generative In-Context Learners [paper link] 2024-06-07
  David Samuel

• What Do Language Models Learn in Context? The Structured Task Hypothesis [paper link] 2024-06-06
  Jiaoda Li; Yifan Hou; Mrinmaya Sachan; Ryan Cotterell

• Verbalized Machine Learning: Revisiting Machine Learning with Language Models [paper link] 2024-06-06
  Tim Z. Xiao; Robert Bamler; Bernhard Schölkopf; Weiyang Liu

• Transformers Need Glasses! Information Over-squashing in Language Tasks [paper link] 2024-06-06
  Federico Barbero; Andrea Banino; Steven Kapturowski; Dharshan Kumaran; João G.M. Araújo; Alex Vitvitskyi; Razvan Pascanu; Petar Veličković

• Scaling and evaluating sparse autoencoders [paper link] 2024-06-06
  Leo Gao; Tom Dupré la Tour; Henk Tillman; Gabriel Goh; Rajan Troll; Alec Radford; Ilya Sutskever; Jan Leike; Jeffrey Wu

• On Limitation of Transformer for Learning HMMs [paper link] 2024-06-06
  Jiachen Hu; Qinghua Liu; Chi Jin

• Enhancing In-Context Learning Performance with just SVD-Based Weight Pruning: A Theoretical Perspective [paper link] 2024-06-06
  Xinhao Yao; Xiaolin Hu; Shenzhi Yang; Yong Liu

• Rethinking Spiking Neural Networks as State Space Models [paper link] 2024-06-05
  Malyaban Bal; Abhronil Sengupta

• Pre-trained Large Language Models Use Fourier Features to Compute Addition [paper link] 2024-06-05
  Tianyi Zhou; Deqing Fu; Vatsal Sharan; Robin Jia

• Local to Global: Learning Dynamics and Effect of Initialization for Transformers [paper link] 2024-06-05
  Ashok Vardhan Makkuva; Marco Bondaschi; Chanakya Ekbote; Adway Girish; Alliot Nagle; Hyeji Kim; Michael Gastpar

• How Truncating Weights Improves Reasoning in Language Models [paper link] 2024-06-05
  Lei Chen; Joan Bruna; Alberto Bietti

• Exact Conversion of In-Context Learning to Model Weights in Linearized-Attention Transformers [paper link] 2024-06-05
  Brian K Chen; Tianyang Hu; Hui Jin; Hwee Kuan Lee; Kenji Kawaguchi

• Computational Limits of Low-Rank Adaptation (LoRA) for Transformer-Based Models [paper link] 2024-06-05
  Jerry Yao-Chieh Hu; Maojiang Su; En-Jui Kuo; Zhao Song; Han Liu

• On Affine Homotopy between Language Encoders [paper link] 2024-06-04
  Robin SM Chan; Reda Boumasmoud; Anej Svete; Yuxin Ren; Qipeng Guo; Zhijing Jin; Shauli Ravfogel; Mrinmaya Sachan; Bernhard Schölkopf; Mennatallah El-Assady; Ryan Cotterell

• Observable Propagation: Uncovering Feature Vectors in Transformers [paper link] 2024-06-04
  Jacob Dunefsky; Arman Cohan

• LongSSM: On the Length Extension of State-space Models in Language Modelling [paper link] 2024-06-04
  Shida Wang

• Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks [paper link] 2024-06-04
  Tianyu He; Darshil Doshi; Aritra Das; Andrey Gromov

• Iteration Head: A Mechanistic Study of Chain-of-Thought [paper link] 2024-06-04
  Vivien Cabannes; Charles Arnal; Wassim Bouaziz; Alice Yang; Francois Charton; Julia Kempe

• Physics of Language Models: Part 1, Learning Hierarchical Language Structures [paper link] 2024-06-02
  Zeyuan Allen-Zhu; Yuanzhi Li

• Anisotropy is Not Inherent to Transformers [paper link] 2024-06
  Anemily Machina; Robert Mercer

• Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality [paper link] 2024-05-31
  Tri Dao; Albert Gu

• Why Larger Language Models Do In-context Learning Differently? [paper link] 2024-05-30
  Zhenmei Shi; Junyi Wei; Zhuoyan Xu; Yingyu Liang

• Language Models Need Inductive Biases to Count Inductively [paper link] 2024-05-30
  Yingshan Chang; Yonatan Bisk

• Is In-Context Learning Sufficient for Instruction Following in LLMs? [paper link] 2024-05-30
  Hao Zhao; Maksym Andriushchenko; Francesco Croce; Nicolas Flammarion

• On the Role of Attention Masks and LayerNorm in Transformers [paper link] 2024-05-29
  Xinyi Wu; Amir Ajorlou; Yifei Wang; Stefanie Jegelka; Ali Jadbabaie

• Lower Bounds on the Expressivity of Recurrent Neural Language Models [paper link] 2024-05-29
  Anej Svete; Franz Nowak; Anisha Mohamed Sahabdeen; Ryan Cotterell

• Does learning the right latent variables necessarily improve in-context learning? [paper link] 2024-05-29
  Sarthak Mittal; Eric Elmoznino; Leo Gagnon; Sangnie Bhardwaj; Dhanya Sridhar; Guillaume Lajoie

• Deep Grokking: Would Deep Neural Networks Generalize Better? [paper link] 2024-05-29
  Simin Fan; Razvan Pascanu; Martin Jaggi

• A Theory of In-Context Learning in Transformers [paper link] 2024-05-29
  Yifei Wang; Yuyang Wu; Zeming Wei; Stefanie Jegelka; Yisen Wang

• Understanding Transformer Reasoning Capabilities via Graph Algorithms [paper link] 2024-05-28
  Clayton Sanford; Bahare Fatemi; Ethan Hall; Anton Tsitsulin; Mehran Kazemi; Jonathan Halcrow; Bryan Perozzi; Vahab Mirrokni

• Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations [paper link] 2024-05-28
  Alexander Hägele; Elie Bakouch; Atli Kosson; Loubna Ben Allal; Leandro Von Werra; Martin Jaggi

• Linguistic Collapse: Neural Collapse in (Large) Language Models [paper link] 2024-05-28
  Robert Wu; Vardan Papyan

• Knowledge Circuits in Pretrained Transformers [paper link] 2024-05-28
  Yunzhi Yao; Ningyu Zhang; Zekun Xi; Mengru Wang; Ziwen Xu; Shumin Deng; Huajun Chen

• Exploring Activation Patterns of Parameters in Language Models [paper link] 2024-05-28
  Yudong Wang; Damai Dai; Zhifang Sui

• Unlocking the Secrets of Linear Complexity Sequence Model from A Unified Perspective [paper link] 2024-05-27
  Zhen Qin; Xuyang Shen; Dong Li; Weigao Sun; Stan Birchfield; Richard Hartley; Yiran Zhong

• Unifying Demonstration Selection and Compression for In-Context Learning [paper link] 2024-05-27
  Jun Gao

• Understanding Linear Probing then Fine-tuning Language Models from NTK Perspective [paper link] 2024-05-27
  Akiyoshi Tomihari; Issei Sato

• Transformers Can Do Arithmetic with the Right Embeddings [paper link] 2024-05-27
  Sean McLeish; Arpit Bansal; Alex Stein; Neel Jain; John Kirchenbauer; Brian R. Bartoldson; Bhavya Kailkhura; Abhinav Bhatele; Jonas Geiping; Avi Schwarzschild; Tom Goldstein

• Transformer In-Context Learning for Categorical Data [paper link] 2024-05-27
  Aaron T. Wang; Ricardo Henao; Lawrence Carin

• On the Noise Robustness of In-Context Learning for Text Generation [paper link] 2024-05-27
  Hongfu Gao; Feipeng Zhang; Wenyu Jiang; Jun Shu; Feng Zheng; Hongxin Wei

• On Mesa-Optimization in Autoregressively Trained Transformers: Emergence and Capability [paper link] 2024-05-27
  Chenyu Zheng; Wei Huang; Rongzhen Wang; Guoqiang Wu; Jun Zhu; Chongxuan Li

• Mechanistic Interpretability of Binary and Ternary Transformers [paper link] 2024-05-27
  Jason Li

• InversionView: A General-Purpose Method for Reading Information from Neural Activations [paper link] 2024-05-27
  Xinting Huang; Madhur Panwar; Navin Goyal; Michael Hahn

• From Neurons to Neutrons: A Case Study in Interpretability [paper link] 2024-05-27
  Ouail Kitouni; Niklas Nolte; Víctor Samuel Pérez-Díaz; Sokratis Trifinopoulos; Mike Williams

• Demystifying amortized causal discovery with transformers [paper link] 2024-05-27
  Francesco Montagna; Max Cairney-Leeming; Dhanya Sridhar; Francesco Locatello

• Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words? [paper link] 2024-05-27
  Gal Yona; Roee Aharoni; Mor Geva

• Automatic Domain Adaptation by Transformers in In-Context Learning [paper link] 2024-05-27
  Ryuichiro Hataya; Kota Matsui; Masaaki Imaizumi

• A One-Layer Decoder-Only Transformer is a Two-Layer RNN: With an Application to Certified Robustness [paper link] 2024-05-27
  Yuhao Zhang; Aws Albarghouthi; Loris D'Antoni

• gzip Predicts Data-dependent Scaling Laws [paper link] 2024-05-26
  Rohan Pandey

• Reality Only Happens Once: Single-Path Generalization Bounds for Transformers [paper link] 2024-05-26
  Yannick Limmer; Anastasis Kratsios; Xuwei Yang; Raeid Saqur; Blanka Horvath

• Limits of Deep Learning: Sequence Modeling through the Lens of Complexity Theory [paper link] 2024-05-26
  Nikola Zubić; Federico Soldá; Aurelio Sulser; Davide Scaramuzza

• Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs [paper link] 2024-05-26
  Mustafa Shukor; Matthieu Cord

• Theoretical Analysis of Weak-to-Strong Generalization [paper link] 2024-05-25
  Hunter Lang; David Sontag; Aravindan Vijayaraghavan

• The Power of Hard Attention Transformers on Data Sequences: A Formal Language Theoretic Perspective [paper link] 2024-05-25
  Pascal Bergsträßer; Chris Köcher; Anthony Widjaja Lin; Georg Zetzsche

• A statistical framework for weak-to-strong generalization [paper link] 2024-05-25
  Seamus Somerstep; Felipe Maia Polo; Moulinath Banerjee; Ya'acov Ritov; Mikhail Yurochkin; Yuekai Sun

• Transformers represent belief state geometry in their residual stream [paper link] 2024-05-24
  Adam S. Shai; Sarah E. Marzen; Lucas Teixeira; Alexander Gietelink Oldenziel; Paul M. Riechers

• Towards Understanding How Transformer Perform Multi-step Reasoning with Matching Operation [paper link] 2024-05-24
  Zhiwei Wang; Yunji Wang; Zhongwang Zhang; Zhangchen Zhou; Hui Jin; Tianyang Hu; Jiacheng Sun; Zhenguo Li; Yaoyu Zhang; Zhi-Qin John Xu

• Towards Better Understanding of In-Context Learning Ability from In-Context Uncertainty Quantification [paper link] 2024-05-24
  Shang Liu; Zhongze Cai; Guanting Chen; Xiaocheng Li

• Quantifying the Gain in Weak-to-Strong Generalization [paper link] 2024-05-24
  Moses Charikar; Chirag Pabbaraju; Kirankumar Shiragur

• MLPs Learn In-Context [paper link] 2024-05-24
  William L. Tong; Cengiz Pehlevan

• Infinite Limits of Multi-head Transformer Dynamics [paper link] 2024-05-24
  Blake Bordelon; Hamza Tahir Chaudhry; Cengiz Pehlevan

• Emergence of a High-Dimensional Abstraction Phase in Language Transformers [paper link] 2024-05-24
  Emily Cheng; Diego Doimo; Corentin Kervadec; Iuri Macocco; Jade Yu; Alessandro Laio; Marco Baroni

• Dissecting the Interplay of Attention Paths in a Statistical Mechanics Theory of Transformers [paper link] 2024-05-24
  Lorenzo Tiberi; Francesca Mignacco; Kazuki Irie; Haim Sompolinsky

• A rationale from frequency perspective for grokking in training neural network [paper link] 2024-05-24
  Zhangchen Zhou; Yaoyu Zhang; Zhi-Qin John Xu

• Not All Language Model Features Are Linear [paper link] 2024-05-23
  Joshua Engels; Isaac Liao; Eric J. Michaud; Wes Gurnee; Max Tegmark

• Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization [paper link] 2024-05-23
  Boshi Wang; Xiang Yue; Yu Su; Huan Sun

• Data Mixing Made Efficient: A Bivariate Scaling Law for Language Model Pretraining [paper link] 2024-05-23
  Ce Ge; Zhijian Ma; Daoyuan Chen; Yaliang Li; Bolin Ding

• 4+3 Phases of Compute-Optimal Neural Scaling Laws [paper link] 2024-05-23
  Elliot Paquette; Courtney Paquette; Lechao Xiao; Jeffrey Pennington

• Upper and lower memory capacity bounds of transformers for next-token prediction [paper link] 2024-05-22
  Liam Madden; Curtis Fox; Christos Thrampoulidis

• Slaves to the Law of Large Numbers: An Asymptotic Equipartition Property for Perplexity in Generative Language Models [paper link] 2024-05-22
  Raghu Mudumbai; Tyler Bell

• Attention as an RNN [paper link] 2024-05-22
  Leo Feng; Frederick Tung; Hossein Hajimirsadeghi; Mohamed Osama Ahmed; Yoshua Bengio; Greg Mori

• Attention Mechanisms Don't Learn Additive Models: Rethinking Feature Importance for Transformers [paper link] 2024-05-22
  Tobias Leemann; Alina Fastowski; Felix Pfeiffer; Gjergji Kasneci

• Sparse Autoencoders Enable Scalable and Reliable Circuit Identification in Language Models [paper link] 2024-05-21
  Charles O'Neill; Thang Bui

• Quantifying Emergence in Large Language Models [paper link] 2024-05-21
  Hang Chen; Xinyu Yang; Jiaying Zhu; Wenya Wang

• Your Transformer is Secretly Linear [paper link] 2024-05-19
  Anton Razzhigaev; Matvey Mikhalchuk; Elizaveta Goncharova; Nikolai Gerasimenko; Ivan Oseledets; Denis Dimitrov; Andrey Kuznetsov

• A Multi-Perspective Analysis of Memorization in Large Language Models [paper link] 2024-05-19
  Bowen Chen; Namgi Han; Yusuke Miyao

• Surgical Feature-Space Decomposition of LLMs: Why, When and How? [paper link] 2024-05-17
  Arnav Chavan; Nahush Lele; Deepak Gupta

• Dynamic Activation Pitfalls in LLaMA Models: An Empirical Study [paper link] 2024-05-15
  Chi Ma; Mincong Huang; Chao Wang; Yujie Wang; Lei Yu

• ALPINE: Unveiling the Planning Capability of Autoregressive Learning in Language Models [paper link] 2024-05-15
  Siwei Wang; Yifei Shen; Shi Feng; Haoran Sun; Shang-Hua Teng; Wei Chen

• Challenges in Deploying Long-Context Transformers: A Theoretical Peak Performance Analysis [paper link] 2024-05-14
  Yao Fu

• Beyond Scaling Laws: Understanding Transformer Performance with Associative Memory [paper link] 2024-05-14
  Xueyan Niu; Bo Bai; Lei Deng; Wei Han

• The Platonic Representation Hypothesis [paper link] 2024-05-13
  Minyoung Huh; Brian Cheung; Tongzhou Wang; Phillip Isola

• Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? [paper link] 2024-05-09
  Zorik Gekhman; Gal Yona; Roee Aharoni; Matan Eyal; Amir Feder; Roi Reichart; Jonathan Herzig

• Initialization is Critical to Whether Transformers Fit Composite Functions by Inference or Memorizing [paper link] 2024-05-08
  Zhongwang Zhang; Pengxiao Lin; Zhiwei Wang; Yaoyu Zhang; Zhi-Qin John Xu

• Towards a Theoretical Understanding of the 'Reversal Curse' via Training Dynamics [paper link] 2024-05-07
  Hanlin Zhu; Baihe Huang; Shaolun Zhang; Michael Jordan; Jiantao Jiao; Yuandong Tian; Stuart Russell

• What Formal Languages Can Transformers Express? A Survey [paper link] 2024-05-06
  Lena Strobl; William Merrill; Gail Weiss; David Chiang; Dana Angluin

• Anchored Answers: Unravelling Positional Bias in GPT-2's Multiple-Choice Questions [paper link] 2024-05-06
  Ruizhe Li; Yanjun Gao

• Understand LLMs Requires More Than Statistical Generalization [paper link] 2024-05-03
  Patrik Reizinger; Szilvia Ujváry; Anna Mészáros; Anna Kerekes; Wieland Brendel; Ferenc Huszár

• Viewing Transformers Through the Lens of Long Convolutions Layers [paper link] 2024-05-02
  Itamar Zimerman; Lior Wolf

• Modeling Language Tokens as Functionals of Semantic Fields [paper link] 2024-05-02
  Zhengqi Pei; Anran Zhang; Shuhui Wang; Qingming Huang

• In-context Learning on Function Classes Unveiled for Transformers [paper link] 2024-05-02
  Zhijie Wang; Bo Jiang; Shuai Li

• GiLOT: Interpreting Generative Language Models via Optimal Transport [paper link] 2024-05-02
  Xuhong Li; Jiamin Chen; Yekun Chai; Haoyi Xiong

• Emergent Representations of Program Semantics in Language Models Trained on Programs [paper link] 2024-05-02
  Charles Jin; Martin Rinard

• Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning? [paper link] 2024-05-02
  Khashayar Gatmiry; Nikunj Saunshi; Sashank J. Reddi; Stefanie Jegelka; Sanjiv Kumar

• By Tying Embeddings You Are Assuming the Distributional Hypothesis [paper link] 2024-05-02
  Francesco Bertolotti; Walter Cazzola

• A Primer on the Inner Workings of Transformer-based Language Models [paper link] 2024-05-02
  Javier Ferrando; Gabriele Sarti; Arianna Bisazza; Marta R. Costa-jussà

• More Compute Is What You Need [paper link] 2024-04-30
  Zhen Guo

• Knowledge Distillation vs. Pretraining from Scratch under a Fixed (Computation) Budget [paper link] 2024-04-30
  Minh Duc Bui; Fabian David Schmidt; Goran Glavaš; Katharina von der Wense

• In-Context Learning with Long-Context Models: An In-Depth Exploration [paper link] 2024-04-30
  Amanda Bertsch; Maor Ivgi; Uri Alon; Jonathan Berant; Matthew R. Gormley; Graham Neubig

• An exactly solvable model for emergence and scaling laws [paper link] 2024-04-26
  Yoonsoo Nam; Nayara Fonseca; Seok Hyeong Lee; Ard Louis

• Talking Nonsense: Probing Large Language Models' Understanding of Adversarial Gibberish Inputs [paper link] 2024-04-25
  Valeriia Cherepanova; James Zou

• Let's Think Dot by Dot: Hidden Computation in Transformer Language Models [paper link] 2024-04-24
  Jacob Pfau; William Merrill; Samuel R. Bowman

• Transformers Can Represent $n$-gram Language Models [paper link] 2024-04-23
  Anej Svete; Ryan Cotterell

• Mechanics of Next Token Prediction with Self-Attention [paper link] 2024-04-18
  Yingcong Li; Yixiao Huang; Muhammed E. Ildiz; Ankit Singh Rawat; Samet Oymak

• On the Empirical Complexity of Reasoning and Planning in LLMs [paper link] 2024-04-17
  Liwei Kang; Zirui Zhao; David Hsu; Wee Sun Lee

• When can transformers reason with abstract symbols? [paper link] 2024-04-16
  Enric Boix-Adsera; Omid Saremi; Emmanuel Abbe; Samy Bengio; Etai Littwin; Joshua Susskind

• Control Theoretic Approach to Fine-Tuning and Transfer Learning [paper link] 2024-04-16
  Erkan Bayram; Shenyu Liu; Mohamed-Ali Belabbas; Tamer Başar

• Compression Represents Intelligence Linearly [paper link] 2024-04-15
  Yuzhen Huang; Jinghan Zhang; Zifei Shan; Junxian He

• Toward a Theory of Tokenization in LLMs [paper link] 2024-04-12
  Nived Rajaraman; Jiantao Jiao; Kannan Ramchandran

• The Illusion of State in State-Space Models [paper link] 2024-04-12
  William Merrill; Jackson Petty; Ashish Sabharwal

• Look at the Text: Instruction-Tuned Language Models are More Robust Multiple Choice Selectors than You Think [paper link] 2024-04-12
  Xinpeng Wang; Chengzhi Hu; Bolei Ma; Paul Röttger; Barbara Plank

• Why do small language models underperform? Studying Language Model Saturation via the Softmax Bottleneck [paper link] 2024-04-11
  Nathan Godey; Éric de la Clergerie; Benoît Sagot

• On Training Data Influence of GPT Models [paper link] 2024-04-11
  Qingyi Liu; Yekun Chai; Shuohuan Wang; Yu Sun; Keze Wang; Hua Wu

• Best Practices and Lessons Learned on Synthetic Data for Language Models [paper link] 2024-04-11
  Ruibo Liu; Jerry Wei; Fangyu Liu; Chenglei Si; Yanzhe Zhang; Jinmeng Rao; Steven Zheng; Daiyi Peng; Diyi Yang; Denny Zhou; Andrew M. Dai

• What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formation [paper link] 2024-04-10
  Aaditya K. Singh; Ted Moskovitz; Felix Hill; Stephanie C. Y. Chan; Andrew M. Saxe

• Language Generation in the Limit [paper link] 2024-04-10
  Jon Kleinberg; Sendhil Mullainathan

• On the Effect of (Near) Duplicate Subwords in Language Modelling [paper link] 2024-04-09
  Anton Schäfer; Thomas Hofmann; Imanol Schlag; Tiago Pimentel

• Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws [paper link] 2024-04-08
  Zeyuan Allen-Zhu; Yuanzhi Li

• A Large-Scale Exploration of $\mu$-Transfer [paper link] 2024-04-08
  Lucas Lingle

• How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapse [paper link] 2024-04-07
  Mohamed El Amine Seddik; Suei-Wen Chen; Soufiane Hayou; Pierre Youssef; Merouane Debbah

• The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A" [paper link] 2024-04-04
  Lukas Berglund; Meg Tong; Max Kaufmann; Mikita Balesni; Asa Cooper Stickland; Tomasz Korbak; Owain Evans

• Attention is Naturally Sparse with Gaussian Distributed Input [paper link] 2024-04-03
  Yichuan Deng; Zhao Song; Chiwun Yang

• Emergent Abilities in Reduced-Scale Generative Language Models [paper link] 2024-04-02
  Sherin Muckatira; Vijeta Deshpande; Vladislav Lialin; Anna Rumshisky

• Collapse of Self-trained Language Models [paper link] 2024-04-02
  David Herel; Tomas Mikolov

• What Can Transformer Learn with Varying Depth? Case Studies on Sequence Learning Tasks [paper link] 2024-04-01
  Xingwu Chen; Difan Zou

• Is attention required for ICL? Exploring the Relationship Between Model Architecture and In-Context Learning Ability [paper link] 2024-04-01
  Ivan Lee; Nan Jiang; Taylor Berg-Kirkpatrick

• Do language models plan ahead for future tokens? [paper link] 2024-03-31
  Wilson Wu; John X. Morris; Lionel Levine

• The Topos of Transformer Networks [paper link] 2024-03-27
  Mattia Jacopo Villani; Peter McBurney

• Mechanisms of non-factual hallucinations in language models [paper link] 2024-03-26
  Lei Yu; Meng Cao; Jackie Chi Kit Cheung; Yue Dong

• Understanding Emergent Abilities of Language Models from the Loss Perspective [paper link] 2024-03-23
  Zhengxiao Du; Aohan Zeng; Yuxiao Dong; Jie Tang

• Unraveling the Mystery of Scaling Laws: Part I [paper link] 2024-03-21
  Hui Su; Zhi Tian; Xiaoyu Shen; Xunliang Cai

• Unveiling the Generalization Power of Fine-Tuned Large Language Models [paper link] 2024-03-14
  Haoran Yang; Yumeng Zhang; Jiaqi Xu; Hongyuan Lu; Pheng Ann Heng; Wai Lam

• Transformers Get Stable: An End-to-End Signal Propagation Theory for Language Models [paper link] 2024-03-14
  Akhil Kedia; Mohd Abbas Zaidi; Sushil Khyalia; Jungho Jung; Harshith Goka; Haejun Lee

• Linear Attention is (Maybe) All You Need (to Understand Transformer Optimization) [paper link] 2024-03-13
  Kwangjun Ahn; Xiang Cheng; Minhak Song; Chulhee Yun; Ali Jadbabaie; Suvrit Sra

• Language models scale reliably with over-training and on downstream tasks [paper link] 2024-03-13
  Samir Yitzhak Gadre; Georgios Smyrnis; Vaishaal Shankar; Suchin Gururangan; Mitchell Wortsman; Rulin Shao; Jean Mercat; Alex Fang; Jeffrey Li; Sedrick Keh; Rui Xin; Marianna Nezhurina; Igor Vasiljevic; Jenia Jitsev; Alexandros G. Dimakis; Gabriel Ilharco; Shuran Song; Thomas Kollar; Yair Carmon; Achal Dave; Reinhard Heckel; Niklas Muennighoff; Ludwig Schmidt

• Simulating Weighted Automata over Sequences and Trees with Transformers [paper link] 2024-03-12
  Michael Rizvi; Maude Lizaire; Clara Lacroce; Guillaume Rabusseau

• Hallmarks of Optimization Trajectories in Neural Networks and LLMs: The Lengths, Bends, and Dead Ends [paper link] 2024-03-12
  Sidak Pal Singh; Bobby He; Thomas Hofmann; Bernhard Schölkopf

• The pitfalls of next-token prediction [paper link] 2024-03-11
  Gregor Bachmann; Vaishnavh Nagarajan

• Simplicity Bias of Transformers to Learn Low Sensitivity Functions [paper link] 2024-03-11
  Bhavya Vasudeva; Deqing Fu; Tianyi Zhou; Elliott Kau; Youqi Huang; Vatsal Sharan

• Algorithmic progress in language models [paper link] 2024-03-09
  Anson Ho; Tamay Besiroglu; Ege Erdil; David Owen; Robi Rahman; Zifan Carl Guo; David Atkinson; Neil Thompson; Jaime Sevilla

• Unfamiliar Finetuning Examples Control How Language Models Hallucinate [paper link] 2024-03-08
  Katie Kang; Eric Wallace; Claire Tomlin; Aviral Kumar; Sergey Levine

• The Heuristic Core: Understanding Subnetwork Generalization in Pretrained Language Models [paper link] 2024-03-06
  Adithya Bhaskar; Dan Friedman; Danqi Chen

• On the Origins of Linear Representations in Large Language Models [paper link] 2024-03-06
  Yibo Jiang; Goutham Rajendran; Pradeep Ravikumar; Bryon Aragam; Victor Veitch

• What's In My Big Data? [paper link] 2024-03-05
  Yanai Elazar; Akshita Bhagia; Ian Magnusson; Abhilasha Ravichander; Dustin Schwenk; Alane Suhr; Pete Walsh; Dirk Groeneveld; Luca Soldaini; Sameer Singh; Hanna Hajishirzi; Noah A. Smith; Jesse Dodge

• In-Context Sharpness as Alerts: An Inner Representation Perspective for Hallucination Mitigation [paper link] 2024-03-05
  Shiqi Chen; Miao Xiong; Junteng Liu; Zhengxuan Wu; Teng Xiao; Siyang Gao; Junxian He

• How Well Can Transformers Emulate In-context Newton's Method? [paper link] 2024-03-05
  Angeliki Giannou; Liu Yang; Tianhao Wang; Dimitris Papailiopoulos; Jason D. Lee

• Why are Sensitive Functions Hard for Transformers? [paper link] 2024-03-03
  Michael Hahn; Mark Rofin

• Memorization Capacity of Multi-Head Attention in Transformers [paper link] 2024-03-02
  Sadegh Mahdavi; Renjie Liao; Christos Thrampoulidis

• Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimality [paper link] 2024-02-29
  Siyu Chen; Heejune Sheen; Tianhao Wang; Zhuoran Yang

• RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrieval [paper link] 2024-02-29
  Kaiyue Wen; Xingyu Dang; Kaifeng Lyu

• Tokenization Is More Than Compression [paper link] 2024-02-28
  Craig W. Schmidt; Varshini Reddy; Haoran Zhang; Alec Alameddine; Omri Uzan; Yuval Pinter; Chris Tanner

• Implicit Bias of Next-Token Prediction [paper link] 2024-02-28
  Christos Thrampoulidis

• Massive Activations in Large Language Models [paper link] 2024-02-27
  Mingjie Sun; Xinlei Chen; J. Zico Kolter; Zhuang Liu

• When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method [paper link] 2024-02-26
  Biao Zhang; Zhongtao Liu; Colin Cherry; Orhan Firat

• Interpreting Grokked Transformers in Complex Modular Arithmetic [paper link] 2024-02-26
  Hiroki Furuta; Gouki Minegishi; Yusuke Iwasawa; Yutaka Matsuo

• Transformers are Expressive, But Are They Expressive Enough for Regression? [paper link] 2024-02-23
  Swaroop Nath; Harshad Khadilkar; Pushpak Bhattacharyya

• Interpreting Context Look-ups in Transformers: Investigating Attention-MLP Interactions [paper link] 2024-02-22
  Clement Neo; Shay B. Cohen; Fazl Barez

• In-Context Learning of a Linear Transformer Block: Benefits of the MLP Component and One-Step GD Initialization [paper link] 2024-02-22
  Ruiqi Zhang; Jingfeng Wu; Peter L. Bartlett

• How Transformers Learn Causal Structure with Gradient Descent [paper link] 2024-02-22
  Eshaan Nichani; Alex Damian; Jason D. Lee

• On the Expressive Power of a Variant of the Looped Transformer [paper link] 2024-02-21
  Yihang Gao; Chuanyang Zheng; Enze Xie; Han Shi; Tianyang Hu; Yu Li; Michael K. Ng; Zhenguo Li; Zhaoqiang Liu

• Do Efficient Transformers Really Save Computation? [paper link] 2024-02-21
  Kai Yang; Jan Ackermann; Zhenyu He; Guhao Feng; Bohang Zhang; Yunzhen Feng; Qiwei Ye; Di He; Liwei Wang

• Identifying Semantic Induction Heads to Understand In-Context Learning [paper link] 2024-02-20
  Jie Ren; Qipeng Guo; Hang Yan; Dongrui Liu; Xipeng Qiu; Dahua Lin

• From Self-Attention to Markov Models: Unveiling the Dynamics of Generative Transformers [paper link] 2024-02-20
  M. Emrullah Ildiz; Yixiao Huang; Yingcong Li; Ankit Singh Rawat; Samet Oymak

• Chain of Thought Empowers Transformers to Solve Inherently Serial Problems [paper link] 2024-02-20
  Zhiyuan Li; Hong Liu; Denny Zhou; Tengyu Ma

• LoRA Training in the NTK Regime has No Spurious Local Minima [paper link] 2024-02-19
  Uijeong Jang; Jason D. Lee; Ernest K. Ryu

• A Mechanistic Analysis of a Transformer Trained on a Symbolic Multi-Step Reasoning Task [paper link] 2024-02-19
  Jannik Brinkmann; Abhay Sheshadri; Victor Levoso; Paul Swoboda; Christian Bartelt

• Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Context [paper link] 2024-02-15
  Xiang Cheng; Yuxin Chen; Suvrit Sra

• Limits of Transformer Language Models on Learning Algorithmic Compositions [paper link] 2024-02-13
  Jonathan Thomm; Aleksandar Terzic; Geethan Karunaratne; Giacomo Camposampiero; Bernhard Schölkopf; Abbas Rahimi

• A Tale of Tails: Model Collapse as a Change of Scaling Laws [paper link] 2024-02-10
  Elvis Dohmatob; Yunzhen Feng; Pu Yang; Francois Charton; Julia Kempe

• How do Transformers perform In-Context Autoregressive Learning? [paper link] 2024-02-08
  Michael E. Sander; Raja Giryes; Taiji Suzuki; Mathieu Blondel; Gabriel Peyré

• Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuning [paper link] 2024-02-07
  Hao Zhao; Maksym Andriushchenko; Francesco Croce; Nicolas Flammarion

• Provably learning a multi-head attention layer [paper link] 2024-02-06
  Sitan Chen; Yuanzhi Li

• On the Emergence of Cross-Task Linearity in the Pretraining-Finetuning Paradigm [paper link] 2024-02-06
  Zhanpeng Zhou; Zijun Chen; Yilan Chen; Bo Zhang; Junchi Yan

• Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Tasks [paper link] 2024-02-06
  Jongho Park; Jaeseung Park; Zheyang Xiong; Nayoung Lee; Jaewoong Cho; Samet Oymak; Kangwook Lee; Dimitris Papailiopoulos

• Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Tasks [paper link] 2024-02-05
  Rahul Ramesh; Ekdeep Singh Lubana; Mikail Khona; Robert P. Dick; Hidenori Tanaka

• Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators? [paper link] 2024-01-29
  Tokio Kajitsuka; Issei Sato

• An Information-Theoretic Analysis of In-Context Learning [paper link] 2024-01-28
  Hong Jun Jeon; Jason D. Lee; Qi Lei; Benjamin Van Roy

• Anisotropy Is Inherent to Self-Attention in Transformers [paper link] 2024-01-24
  Nathan Godey; Éric de la Clergerie; Benoît Sagot

• Universal Neurons in GPT2 Language Models [paper link] 2024-01-22
  Wes Gurnee; Theo Horsley; Zifan Carl Guo; Tara Rezaei Kheirkhah; Qinyi Sun; Will Hathaway; Neel Nanda; Dimitris Bertsimas

• Transformers are Multi-State RNNs [paper link] 2024-01-11
  Matanel Oren; Michael Hassid; Yossi Adi; Roy Schwartz

• Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective [paper link] 2023-12-22
  Guhao Feng; Bohang Zhang; Yuntian Gu; Haotian Ye; Di He; Liwei Wang

• How Capable Can a Transformer Become? A Study on Synthetic, Interpretable Tasks [paper link] 2023-12-12
  Rahul Ramesh; Mikail Khona; Robert P. Dick; Hidenori Tanaka; Ekdeep Singh Lubana

• The Transient Nature of Emergent In-Context Learning in Transformers [paper link] 2023-12-11
  Aaditya K. Singh; Stephanie C. Y. Chan; Ted Moskovitz; Erin Grant; Andrew M. Saxe; Felix Hill

• Transformers learn through gradual rank increase [paper link] 2023-12-10
  Enric Boix-Adsera; Etai Littwin; Emmanuel Abbe; Samy Bengio; Joshua Susskind

• Interpretability Illusions in the Generalization of Simplified Models [paper link] 2023-12-06
  Dan Friedman; Andrew Lampinen; Lucas Dixon; Danqi Chen; Asma Ghandeharioun

• Transformers are uninterpretable with myopic methods: a case study with bounded Dyck grammars [paper link] 2023-12-03
  Kaiyue Wen; Yuchen Li; Bingbin Liu; Andrej Risteski

• An Investigation of LLMs' Inefficacy in Understanding Converse Relations [paper link] 2023-12-01
  Chengwen Qi; Bowen Li; Binyuan Hui; Bailin Wang; Jinyang Li; Jinwang Wu; Yuanjun Laili

• Calibrated Language Models Must Hallucinate [paper link] 2023-11-24
  Adam Tauman Kalai; Santosh S. Vempala

• White-Box Transformers via Sparse Rate Reduction: Compression Is All There Is? [paper link] 2023-11-22
  Yaodong Yu; Sam Buchanan; Druv Pai; Tianzhe Chu; Ziyang Wu; Shengbang Tong; Hao Bai; Yuexiang Zhai; Benjamin D. Haeffele; Yi Ma

• Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks [paper link] 2023-11-21
  Samyak Jain; Robert Kirk; Ekdeep Singh Lubana; Robert P. Dick; Hidenori Tanaka; Edward Grefenstette; Tim Rocktäschel; David Scott Krueger

• In-Context Learning Functions with Varying Number of Minima [paper link] 2023-11-21
  David Oniani; Yanshan Wang

• Exploring the Relationship between In-Context Learning and Instruction Tuning [paper link] 2023-11-17
  Hanyu Duan; Yixuan Tang; Yi Yang; Ahmed Abbasi; Kar Yan Tam

• Universality and Limitations of Prompt Tuning [paper link] 2023-11-16
  Yihan Wang; Jatin Chauhan; Wei Wang; Cho-Jui Hsieh

• Representational Strengths and Limitations of Transformers [paper link] 2023-11-16
  Clayton Sanford; Daniel Hsu; Matus Telgarsky

• When does In-context Learning Fall Short and Why? A Study on Specification-Heavy Tasks [paper link] 2023-11-15
  Hao Peng; Xiaozhi Wang; Jianhui Chen; Weikai Li; Yunjia Qi; Zimu Wang; Zhili Wu; Kaisheng Zeng; Bin Xu; Lei Hou; Juanzi Li

• Data Similarity is Not Enough to Explain Language Model Performance [paper link] 2023-11-15
  Gregory Yauney; Emily Reif; David Mimno

• Transformers can optimally learn regression mixture models [paper link] 2023-11-14
  Reese Pathak; Rajat Sen; Weihao Kong; Abhimanyu Das

• In-context Learning Generalizes, But Not Always Robustly: The Case of Syntax [paper link] 2023-11-13
  Aaron Mueller; Albert Webson; Jackson Petty; Tal Linzen

• Transformers learn to implement preconditioned gradient descent for in-context learning [paper link] 2023-11-09
  Kwangjun Ahn; Xiang Cheng; Hadi Daneshmand; Suvrit Sra

• Birth of a Transformer: A Memory Viewpoint [paper link] 2023-11-06
  Alberto Bietti; Vivien Cabannes; Diane Bouchacourt; Herve Jegou; Leon Bottou

• Simplifying Transformer Blocks [paper link] 2023-11-03
  Bobby He; Thomas Hofmann

• Connecting Pre-trained Language Model and Downstream Task via Properties of Representation [paper link] 2023-11-02
  Chenwei Wu; Holden Lee; Rong Ge

• Causal Interpretation of Self-Attention in Pre-Trained Transformers [paper link] 2023-10-31
  Raanan Y. Rohekar; Yaniv Gurwicz; Shami Nisimov

• The Impact of Depth and Width on Transformer Language Model Generalization [paper link] 2023-10-30
  Jackson Petty; Sjoerd van Steenkiste; Ishita Dasgupta; Fei Sha; Dan Garrette; Tal Linzen

• Transformers Learn Higher-Order Optimization Methods for In-Context Learning: A Study with Linear Models [paper link] 2023-10-26
  Deqing Fu; Tian-Qi Chen; Robin Jia; Vatsal Sharan

• The Expressive Power of Low-Rank Adaptation [paper link] 2023-10-26
  Yuchen Zeng; Kangwook Lee

• How do Language Models Bind Entities in Context? [paper link] 2023-10-26
  Jiahai Feng; Jacob Steinhardt

• Scaling Data-Constrained Language Models [paper link] 2023-10-25
  Niklas Muennighoff; Alexander M. Rush; Boaz Barak; Teven Le Scao; Aleksandra Piktus; Nouamane Tazi; Sampo Pyysalo; Thomas Wolf; Colin Raffel

• What Algorithms can Transformers Learn? A Study in Length Generalization [paper link] 2023-10-24
  Hattie Zhou; Arwen Bradley; Etai Littwin; Noam Razin; Omid Saremi; Josh Susskind; Samy Bengio; Preetum Nakkiran

• In-Context Learning Creates Task Vectors [paper link] 2023-10-24
  Roee Hendel; Mor Geva; Amir Globerson

• Implicit meta-learning may lead language models to trust more reliable sources [paper link] 2023-10-23
  Dmitrii Krasheninnikov; Egor Krasheninnikov; Bruno Mlodozeniec; Tegan Maharaj; David Krueger

• Function Vectors in Large Language Models [paper link] 2023-10-23
  Eric Todd; Millicent L. Li; Arnab Sen Sharma; Aaron Mueller; Byron C. Wallace; David Bau

• Why Can Large Language Models Generate Correct Chain-of-Thoughts? [paper link] 2023-10-20
  Rasul Tutunov; Antoine Grosnit; Juliusz Ziomek; Jun Wang; Haitham Bou-Ammar

• Trained Transformers Learn Linear Models In-Context [paper link] 2023-10-19
  Ruiqi Zhang; Spencer Frei; Peter L. Bartlett

• On the Optimization and Generalization of Multi-head Attention [paper link] 2023-10-19
  Puneesh Deora; Rouzbeh Ghaderi; Hossein Taheri; Christos Thrampoulidis

• In-context Learning with Transformer Is Really Equivalent to a Contrastive Learning Pattern [paper link] 2023-10-19
  Ruifeng Ren; Yong Liu

• The Curious Case of Hallucinatory Unanswerablity: Finding Truths in the Hidden States of Over-Confident Large Language Models [paper link] 2023-10-18
  Aviv Slobodkin; Omer Goldman; Avi Caciularu; Ido Dagan; Shauli Ravfogel

• How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representations [paper link] 2023-10-16
  Tianyu Guo; Wei Hu; Song Mei; Huan Wang; Caiming Xiong; Silvio Savarese; Yu Bai

• Understanding the Mechanics and Dynamics of Memorisation in Large Language Models: A Case Study with Random Strings [paper link] 2023-10-13
  Till Speicher; Aflah Mohammad Khan; Qinyuan Wu; Vedant Nanda; Soumi Das; Bishwamittra Ghosh; Krishna P. Gummadi; Evimaria Terzi

• Understanding prompt engineering may not require rethinking generalization [paper link] 2023-10-13
  Victor Akinwande; Yiding Jiang; Dylan Sam; J. Zico Kolter

• Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functions [paper link] 2023-10-13
  Satwik Bhattamishra; Arkil Patel; Phil Blunsom; Varun Kanade

• The Expressive Power of Transformers with Chain of Thought [paper link] 2023-10-13
  William Merrill; Ashish Sabharwal

• Large Language Models Cannot Self-Correct Reasoning Yet [paper link] 2023-10-13
  Jie Huang; Xinyun Chen; Swaroop Mishra; Huaixiu Steven Zheng; Adams Wei Yu; Xinying Song; Denny Zhou

• In-context Convergence of Transformers [paper link] 2023-10-13
  Yu Huang; Yuan Cheng; Yingbin Liang

• In-Context Learning through the Bayesian Prism [paper link] 2023-10-13
  Madhur Panwar; Kabir Ahuja; Navin Goyal

• In-Context Learning Learns Label Relationships but Is Not Conventional Learning [paper link] 2023-10-13
  Jannik Kossen; Yarin Gal; Tom Rainforth

• How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression? [paper link] 2023-10-13
  Jingfeng Wu; Difan Zou; Zixiang Chen; Vladimir Braverman; Quanquan Gu; Peter Bartlett

• How Large Language Models Implement Chain-of-Thought? [paper link] 2023-10-13
  Yiqun Wang; Sile Hu; Yonggang Zhang; Xiang Tian; Xuesong Liu; Yaowu Chen; Xu Shen; Jieping Ye

• Do pretrained Transformers Really Learn In-context by Gradient Descent? [paper link] 2023-10-12
  Lingfeng Shen; Aayush Mishra; Daniel Khashabi

• What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalization [paper link] 2023-10-10
  Yufeng Zhang; Fengzhuo Zhang; Zhuoran Yang; Zhaoran Wang

• How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition [paper link] 2023-10-09
  Guanting Dong; Hongyi Yuan; Keming Lu; Chengpeng Li; Mingfeng Xue; Dayiheng Liu; Wei Wang; Zheng Yuan; Chang Zhou; Jingren Zhou

• The Cost of Down-Scaling Language Models: Fact Recall Deteriorates before In-Context Learning [paper link] 2023-10-06
  Tian Jin; Nolan Clement; Xin Dong; Vaishnavh Nagarajan; Michael Carbin; Jonathan Ragan-Kelley; Gintare Karolina Dziugaite

• Explaining Emergent In-Context Learning as Kernel Regression [paper link] 2023-10-05
  Chi Han; Ziqi Wang; Han Zhao; Heng Ji

• Physics of Language Models: Part 3.2, Knowledge Manipulation [paper link] 2023-09-25
  Zeyuan Allen-Zhu; Yuanzhi Li

• Understanding Catastrophic Forgetting in Language Models via Implicit Inference [paper link] 2023-09-18
  Suhas Kotha; Jacob Mitchell Springer; Aditi Raghunathan

• Attention-Only Transformers and Implementing MLPs with Attention Heads [paper link] 2023-09-15
  Robert Huben; Valerie Morris

• Transformers as Support Vector Machines [paper link] 2023-09-07
  Davoud Ataee Tarzanagh; Yingcong Li; Christos Thrampoulidis; Samet Oymak

• CausalLM is not optimal for in-context learning [paper link] 2023-09-02
  Nan Ding; Tomer Levinboim; Jialin Wu; Sebastian Goodman; Radu Soricut

• Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth [paper link] 2023-08-01
  Yihe Dong; Jean-Baptiste Cordonnier; Andreas Loukas

• A Theory for Emergence of Complex Skills in Language Models [paper link] 2023-07-29
  Sanjeev Arora; Anirudh Goyal

• Can Neural Network Memorization Be Localized? [paper link] 2023-07-18
  Pratyush Maini; Michael C. Mozer; Hanie Sedghi; Zachary C. Lipton; J. Zico Kolter; Chiyuan Zhang

• The Expressive Power of Tuning Only the Normalization Layers [paper link] 2023-07-12
  Angeliki Giannou; Shashank Rajput; Dimitris Papailiopoulos

• One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attention [paper link] 2023-07-07
  Arvind Mahankali; Tatsunori B. Hashimoto; Tengyu Ma

• Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection [paper link] 2023-07-06
  Yu Bai; Fan Chen; Huan Wang; Caiming Xiong; Song Mei

• On the Power of Foundation Models [paper link] 2023-07-03
  Yang Yuan

• Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformer [paper link] 2023-07-02
  Yuandong Tian; Yiping Wang; Beidi Chen; Simon Du

• Limits for Learning with Language Models [paper link] 2023-06-21
  Nicholas Asher; Swarnadeep Bhar; Akshay Chaturvedi; Julie Hunter; Soumya Paul

• Transformers Learn In-Context by Gradient Descent [paper link] 2023-06-15
  Johannes Von Oswald; Eyvind Niklasson; Ettore Randazzo; Joao Sacramento; Alexander Mordvintsev; Andrey Zhmoginov; Max Vladymyrov

• Tighter Bounds on the Expressivity of Transformer Encoders [paper link] 2023-06-15
  David Chiang; Peter Cholak; Anand Pillay

• Task-Specific Skill Localization in Fine-tuned Language Models [paper link] 2023-06-15
  Abhishek Panigrahi; Nikunj Saunshi; Haoyu Zhao; Sanjeev Arora

• On the Role of Attention in Prompt-tuning [paper link] 2023-06-15
  Samet Oymak; Ankit Singh Rawat; Mahdi Soltanolkotabi; Christos Thrampoulidis

• How Do Transformers Learn Topic Structure: Towards a Mechanistic Understanding [paper link] 2023-06-15
  Yuchen Li; Yuanzhi Li; Andrej Risteski

• A Kernel-Based View of Language Model Fine-Tuning [paper link] 2023-06-15
  Sadhika Malladi; Alexander Wettig; Dingli Yu; Danqi Chen; Sanjeev Arora

• The Reversal Curse: Which Tokens You Predict Underlie the Factorization Curse and More [paper link] 2023-06-07
  Ouail Kitouni; Niklas Nolte; Diane Bouchacourt; Adina Williams; Mike Rabbat; Mark Ibrahim

• ResiDual: Transformer with Dual Residual Connections [paper link] 2023-04-28
  Shufang Xie; Huishuai Zhang; Junliang Guo; Xu Tan; Jiang Bian; Hany Hassan Awadalla; Arul Menezes; Tao Qin; Rui Yan

• Are Emergent Abilities of Large Language Models a Mirage? [paper link] 2023-04-28
  Rylan Schaeffer; Brando Miranda; Sanmi Koyejo

• The Closeness of In-Context Learning and Weight Shifting for Softmax Regression [paper link] 2023-04-26
  Shuai Li; Zhao Song; Yu Xia; Tong Yu; Tianyi Zhou

• The Learnability of In-Context Learning [paper link] 2023-03-14
  Noam Wies; Yoav Levine; Amnon Shashua

• A Theory of Emergent In-Context Learning as Implicit Structure Induction [paper link] 2023-03-14
  Michael Hahn; Navin Goyal

• Fast Attention Requires Bounded Entries [paper link] 2023-02-26
  Josh Alman; Zhao Song

• Towards Understanding Why Mask-Reconstruction Pretraining Helps in Downstream Tasks [paper link] 2023-02-11
  Jiachun Pan; Pan Zhou; Shuicheng Yan

• Transformers Learn Shortcuts to Automata [paper link] 2023-02-01
  Bingbin Liu; Jordan T. Ash; Surbhi Goel; Akshay Krishnamurthy; Cyril Zhang

• The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers [paper link] 2023-02-01
  Zonglin Li; Chong You; Srinadh Bhojanapalli; Daliang Li; Ankit Singh Rawat; Sashank J. Reddi; Ke Ye; Felix Chern; Felix Yu; Ruiqi Guo; Sanjiv Kumar

• A Stability Analysis of Fine-Tuning a Pre-Trained Model [paper link] 2023-01-24
  Zihao Fu; Anthony Man-Cho So; Nigel Collier

• What Can Transformers Learn In-Context? A Case Study of Simple Function Classes [paper link] 2023-01-14
  Shivam Garg; Dimitris Tsipras; Percy Liang; Gregory Valiant

• Transformer Vs. MLP-Mixer: Exponential Expressive Gap For NLP Problems [paper link] 2022-11-17
  Dan Navon; Alex M. Bronstein

• Your Transformer May Not be as Powerful as You Expect [paper link] 2022-10-31
  Shengjie Luo; Shanda Li; Shuxin Zheng; Tie-Yan Liu; Liwei Wang; Di He

• Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Models [paper link] 2022-10-25
  Hong Liu; Sang Michael Xie; Zhiyuan Li; Tengyu Ma

• The Devil in Linear Transformer [paper link] 2022-10-19
  Zhen Qin; XiaoDong Han; Weixuan Sun; Dongxu Li; Lingpeng Kong; Nick Barnes; Yiran Zhong

• Small Transformers Compute Universal Metric Embeddings [paper link] 2022-10-18
  Anastasis Kratsios; Valentin Debarnot; Ivan Dokmanić

• Transformers generalize differently from information stored in context vs in weights [paper link] 2022-10-13
  Stephanie C. Y. Chan; Ishita Dasgupta; Junkyung Kim; Dharshan Kumaran; Andrew K. Lampinen; Felix Hill

• In-Context Learning and Induction Heads [paper link] 2022-09-24
  Catherine Olsson; Nelson Elhage; Neel Nanda; Nicholas Joseph; Nova DasSarma; Tom Henighan; Ben Mann; Amanda Askell; Yuntao Bai; Anna Chen; Tom Conerly; Dawn Drain; Deep Ganguli; Zac Hatfield-Dodds; Danny Hernandez; Scott Johnston; Andy Jones; Jackson Kernion; Liane Lovitt; Kamal Ndousse; Dario Amodei; Tom Brown; Jack Clark; Jared Kaplan; Sam McCandlish; Chris Olah

• Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning [paper link] 2022-04-20
  Colin Wei; Sang Michael Xie; Tengyu Ma

• Training Compute-Optimal Large Language Models [paper link] 2022-03-29
  Jordan Hoffmann; Sebastian Borgeaud; Arthur Mensch; Elena Buchatskaya; Trevor Cai; Eliza Rutherford; Diego de Las Casas; Lisa Anne Hendricks; Johannes Welbl; Aidan Clark; Tom Hennigan; Eric Noland; Katie Millican; George van den Driessche; Bogdan Damoc; Aurelia Guy; Simon Osindero; Karen Simonyan; Erich Elsen; Jack W. Rae; Oriol Vinyals; Laurent Sifre

• DeepNet: Scaling Transformers to 1,000 Layers [paper link] 2022-03-01
  Hongyu Wang; Shuming Ma; Li Dong; Shaohan Huang; Dongdong Zhang; Furu Wei

• Quantifying Memorization Across Neural Language Models [paper link] 2022-02-15
  Nicholas Carlini; Daphne Ippolito; Matthew Jagielski; Katherine Lee; Florian Tramer; Chiyuan Zhang

• The Lipschitz Constant of Self-Attention [paper link] 2021-06-09
  Hyunjik Kim; George Papamakarios; Andriy Mnih

• A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks [paper link] 2021-04-14
  Nikunj Saunshi; Sadhika Malladi; Sanjeev Arora

• Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning [paper link] 2020-12-22
  Armen Aghajanyan; Luke Zettlemoyer; Sonal Gupta

• Why are Adaptive Methods Good for Attention Models? [paper link] 2020-10-23
  Jingzhao Zhang; Sai Praneeth Karimireddy; Andreas Veit; Seungyeon Kim; Sashank J. Reddi; Sanjiv Kumar; Suvrit Sra

• On the Ability and Limitations of Transformers to Recognize Formal Languages [paper link] 2020-09-23
  Satwik Bhattamishra; Kabir Ahuja; Navin Goyal

• On Layer Normalization in the Transformer Architecture [paper link] 2020-06-29
  Ruibin Xiong; Yunchang Yang; Di He; Kai Zheng; Shuxin Zheng; Chen Xing; Huishuai Zhang; Yanyan Lan; Liwei Wang; Tie-Yan Liu

• How fine can fine-tuning be? Learning efficient language models [paper link] 2020-06-03
  Evani Radiya-Dixit; Xin Wang

• On Identifiability in Transformers [paper link] 2020-02-07
  Gino Brunner; Yang Liu; Damián Pascual; Oliver Richter; Massimiliano Ciaramita; Roger Wattenhofer

• Scaling Laws for Neural Language Models [paper link] 2020-01-22
  Jared Kaplan; Sam McCandlish; Tom Henighan; Tom B. Brown; Benjamin Chess; Rewon Child; Scott Gray; Alec Radford; Jeffrey Wu; Dario Amodei

---

Contact

• Shiguang Wu, furyton AT outlook.com / shiguang.wu AT mail.sdu.edu.cn