Research map / Large language models / Reasoning and chain of thought
Reasoning with Latent Thoughts: On the Power of Looped Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that looped transformers (k layers reused L times) match much deeper non-looped models on synthetic reasoning tasks and on reasoning-heavy downstream evaluations of 1B-scale LMs, with theory linking looping to latent-thought CoT simulation, and proposes a cosine-similarity looping regularizer.
Problem
Whether reasoning requires large depth or large parameter count; whether looped (weight-shared depth) models can deliver reasoning ability cheaply, and whether looping helps language modeling on reasoning benchmarks despite worse perplexity.
Approach
Treat depth as looping a shared k-layer block L times: at iso-FLOPs with a kL-layer model, looped models nearly match deeper non-looped models on reasoning while using far fewer parameters, show inductive bias toward reasoning at equal perplexity, and prove looped transformers can simulate CoT with one loop per step. A cosine-similarity regularizer pushes layer blocks toward being looped.
Key results
- On n-ary addition, a 1-layer model looped 12 times reaches 99.9% at n=8 versus 0.1% for the non-looped 1-layer model.
- On i-GSM, a 1-layer model looped 8 times reaches 73.2% accuracy, matching the 8-layer non-looped baseline (73.2%) versus 24.5% for the 1-layer baseline.
- For a 1B 24-layer LM on the Pile, the 12-layer model looped twice reaches 34.3 on math word problems versus 29.3 for the 24-layer baseline, despite worse perplexity (7.90 vs 7.40).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Loop a 1-layer model 12 times instead of using a 1-layer model | Transformer on addition · Addition, n=32 | accuracy: 0.0 -> 99.6 | Helped |
| Loop a 2-layer model 6 times instead of using a 2-layer model | Transformer on addition · Addition, n=32 | accuracy: 38.8 -> 99.5 | Helped |
| Loop a 2-layer model 3 times instead of using a 2-layer model | Transformer on p-hop · p-hop, p=32, n=256 | accuracy: 59.4 -> 99.8 | Helped |
| Loop a 1-layer model 4 times instead of using a 1-layer model | Transformer on i-GSM · i-GSM | accuracy: 24.5 -> 69.9 | Helped |
| Loop a 12-layer model twice instead of using the 24-layer baseline | 1B language model · Math Word Problems | accuracy: 29.3 -> 34.3 | Helped |
| Loop a 12-layer model twice instead of using the 24-layer baseline | 1B language model · Reasoning Primitives | accuracy: 47.5 -> 51.2 | Helped |
| Loop a 12-layer model twice instead of using the 24-layer baseline | 1B language model · Perplexity | perplexity: 7.40 -> 7.90 | Hurt |
| Apply cosine-similarity regularization to successive 4-layer blocks (k=4, λ=10) | 24-layer 1B Transformer · Math Word Problems | accuracy: 29.3 -> 36.4 | Helped |
| Apply cosine-similarity regularization to successive 4-layer blocks (k=4, λ=10) | 24-layer 1B Transformer · Reasoning Primitives | accuracy: 47.5 -> 57.2 | Helped |
| Apply cosine-similarity regularization to successive 4-layer blocks (k=4, λ=10) | 24-layer 1B Transformer · Perplexity | perplexity: 7.40 -> 7.38 | Helped |
| Use λ=1 instead of λ=10 for k=4 regularization (ablation) | 24-layer 1B Transformer · Reasoning Primitives | accuracy: 47.5 -> 42.5 | Hurt |
| Use block size k=6 instead of k=4 for λ=10 regularization (ablation) | 24-layer 1B Transformer · Math Word Problems | accuracy: 29.3 -> 31.0 | Helped |
Surprising findings
- Looped models outperform the iso-flop 24-layer baseline on reasoning primitives despite worse perplexity and fewer parameters.
- A cosine-similarity regularizer that encourages layer blocks to resemble each other improves reasoning benchmarks while keeping perplexity nearly unchanged (7.40 to 7.38).
Limitations
- Language-model experiments are limited to a 1B-scale model trained on the Pile, and reasoning is only tested on a subset of tasks.
- Perplexity and closed-book QA (memorization) remain worse than the iso-flop non-looped baseline.
- Theoretical results are constructions for existence, not learned behavior.
Experimental setup
Datasets: The Pile TriviaQA TydiQA Natural Questions Web Questions SquadV2 DROP QuAC CoQA SVAMP ASDiv MAWPS
Benchmarks: i-GSM n-ary addition p-hop induction TriviaQA TydiQA Natural Questions Web Questions SquadV2 DROP QuAC CoQA SVAMP
Metrics: accuracy validation perplexity % gap
Baselines: non-looped iso-param k-layer model non-looped iso-flop kL-layer model MidAS-style stacking
Builds on: chain-of-thought MidAS / gradual stacking CoTFormer i-GSM (Physics of LMs part 2.1) Universal Transformers
Closest papers by meaning
- LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation (ICLR 2026 · similarity 0.92)
- SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thought (NeurIPS 2025 · similarity 0.91)
- Long Chain-of-Thought Reasoning Across Languages (ICLR 2026 · similarity 0.91)
- A Formal Comparison Between Chain of Thought and Latent Thought (ICML 2026 · similarity 0.91)
- Finding the Minimal Parameter Budget for Implicit Reasoning: A Data Complexity Driven Scaling Law for Language Models (ICML 2026 · similarity 0.91)
- Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoning (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.