Research map / Large language models / Reasoning and chain of thought

LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Long CoT dynamics and efficiency

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LoopFormer, a looped Transformer conditioned on normalized time and step size with a shortcut-consistency loss, enabling elastic-depth (budget-conditioned) language modeling without retraining.

Problem

Looped Transformers are trained with a fixed number of loop iterations, so their representations degrade or stagnate when run at shorter or longer depths, preventing flexible compute budgets at inference.

Approach

Condition each loop on current normalized time t and step size Δt (AdaLN-style modulation of RMSNorm scales and residual gates), and train over sampled shortcut trajectories with a stop-gradient consistency loss aligning shorter-trajectory logits to the full-length trajectory, so any budget M≤L yields usable outputs.

Key results

• At 6x budget, LoopFormer (3⊗8 trained, 3⊗2 inference) reaches 40.36 avg accuracy and 14.3 Pile perplexity, versus 37.59 for TMLT-EE at the same budget.
• At 12x budget, LoopFormer (3⊗4) achieves 43.73 avg accuracy and 11.12 Pile perplexity, close to the Base 12-layer model (44.93 avg, 9.98 ppl).
• At 24x budget, LoopFormer (3⊗8) reaches 44.81 avg accuracy and 10.28 Pile perplexity, slightly above TMLT (44.69) and Base-Loop (42.88), but below the 24-layer Base (45.27, 9.49 ppl).

Surprising findings

• Naive early exiting in looped models yields stagnant, high-CKA representations across loop steps, while the shortcut-consistency training avoids this collapse.

Limitations

• Budgeting is global (sequence-level) rather than instance- or token-adaptive.
• Added training overhead (about 1.5x FLOPs) from the multi-trajectory consistency objective.
• Representation analyses are correlational rather than causal.

Experimental setup

Datasets: The Pile

Benchmarks: The Pile FineWeb-Edu OpenWebText COPA HellaSwag LAMBADA OpenBookQA PIQA RACE SciQ ARC SIQA

Metrics: perplexity zero-shot accuracy average accuracy curvature anisotropy prompt entropy CKA

Baselines: Base (non-looped Transformer) Base-Loop TMLT Base-Loop-EE Base-Loop-EE-Cons TMLT-EE Naive-Loop-EE

Builds on: Time-Modulated Looped Transformers Diffusion Transformers (adaLN) ALBERT Universal Transformer Mixture of Recursions

Closest papers by meaning

• Reasoning with Latent Thoughts: On the Power of Looped Transformers (ICLR 2025 · similarity 0.92)
• LoopRPT: Reinforcement Pre-Training for Looped Language Models (NeurIPS 2026 · similarity 0.89)
• Sparse Layers are Critical to Scaling Looped Language Models (NeurIPS 2026 · similarity 0.89)
• Stabilizing Recurrent Dynamics for Test-Time Scalable Latent Reasoning in Looped Language Models (ICML 2026 · similarity 0.89)
• Looped Diffusion Language Models (NeurIPS 2026 · similarity 0.88)
• Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes (ICLR 2026 · similarity 0.88)