Research map / Large language models / Language, safety and interpretability
Exploring Length Generalization in Large Language Models
Cem Anil, Yuhuai Wu, Anders Johan Andreassen, Aitor Lewkowycz, Vedant Misra, Vinay Venkatesh Ramasesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, Behnam Neyshabur
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| scale finetuned LaMDA from 244m to 64b | vanilla finetuning · Parity length generalization | OOD accuracy | No clear effect |
| add scratchpad to finetuning | finetuned LaMDA · Parity and variable assignment | OOD length accuracy | No clear effect |
| pad input and scratchpad to fix token count and position bias bins (ablation) | scratchpad finetuning · Parity | OOD length accuracy | Mixed |
| mask distractor tokens in input and scratchpad (ablation) | padded scratchpad finetuning · Parity | scratchpad step accuracy on OOD lengths | Improved |
| mask distractor tokens in input only vs scratchpad only (ablation) | padded scratchpad finetuning · Parity | OOD length accuracy | Improved |
| add few-shot scratchpad prompting without finetuning | LaMDA 128b · Coin-flip parity | accuracy on longer lengths | Improved |
| add few-shot prompting to scratchpad finetuning | LaMDA · Parity | OOD length accuracy | Improved |
| add few-shot prompting to scratchpad finetuning | LaMDA · Variable assignment | OOD length accuracy | No clear effect |
| use larger learning rates in finetuning | finetuned LaMDA · Parity | OOD length accuracy | Improved |
Closest papers by meaning
- Extrapolation by Association: Length Generalization Transfer In Transformers (NeurIPS 2025 · similarity 0.91)
- Beyond Single-Task: Robust Multi-Task Length Generalization for LLMs (NeurIPS 2025 · similarity 0.91)
- The Role of Sparsity for Length Generalization in LLMs (ICML 2025 · similarity 0.90)
- Quantifying Generalization Complexity for Large Language Models (ICLR 2025 · similarity 0.89)
- Universal Length Generalization with Turing Programs (ICML 2025 · similarity 0.89)
- Training Language Models to Reason Efficiently (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
