Research map / Architectures and efficiency / RNNs and normalization

How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Models

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Neural scaling laws · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measures how many unique parameters one recurrence of a looped transformer is worth via an iso-depth scaling sweep and a joint law with recurrence-equivalence exponent φ (0.46 baseline; truncated BPTT lowers it to 0.38, hyperconnections raise it to 0.65).

Problem

Looped (depth-recurrent) transformers share parameters across recurrences, but how much unique capacity one recurrence is worth at matched compute has not been measured directly; prior scaling-law work fixes unique parameters and varies depth and inference cost together.

Approach

Fix effective depth (20 layers) across recurrence counts r in {1,2,4,8}, and extend the Chinchilla law with a recurrence-equivalence exponent φ so effective parameters are N_once + r^φ N_rec; φ quantifies per-recurrence value and serves as a diagnostic for training recipes and architectures.

Key results

• Joint scaling law fit gives recurrence-equivalence exponent phi = 0.459 (95% block-bootstrap CI [0.41, 0.53]) with R^2 0.9972 over 116 runs.
• Truncated BPTT lowers phi to 0.380 while lowering validation loss; hyperconnections raise phi to 0.646 (R^2 0.9900).
• At r=4 a 410M looped model is claimed to match a 580M non-looped model while incurring the training cost of a 1B non-looped model.

Surprising findings

• Truncated BPTT substantially lowers validation loss across runs yet reduces phi, indicating the loop mechanism is undertrained at r=2 and forward-only loops.
• Looped models trail the non-looped baseline by 0.03 to 0.12 nats on the compute-optimal frontier, with the gap growing with r.

Limitations

• Single architecture configuration (20 effective layers, prelude/coda 2/2) and single training recipe; phi may shift with other depth allocations.
• Validation loss only; reasoning downstream tasks show no above-noise architectural signal at the budgets tested.
• Joint law assumes each recurrence consistently helps or hurts relative to the non-looped baseline; the fit degrades otherwise.

Experimental setup

Datasets: FineWeb-Edu

Metrics: validation loss (nats) R^2 Huber loss RMSE bootstrap confidence interval on phi

Baselines: non-looped transformer (r=1) Chinchilla scaling law (phi=1 and phi=0 restricted fits) truncated BPTT variant

Builds on: Chinchilla scaling law prelude-recur-coda looped transformer hyperconnections truncated backpropagation through time nanochat

Closest papers by meaning

• How Long Does Infinite Width Last? Signal Propagation in Long-Range Linear Recurrences (NeurIPS 2026 · similarity 0.85)
• Sparse Layers are Critical to Scaling Looped Language Models (NeurIPS 2026 · similarity 0.83)
• Understanding and Improving Length Generalization in Recurrent Models (ICML 2025 · similarity 0.82)
• A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedules (ICLR 2025 · similarity 0.82)
• Universality of Linear Recurrences Followed by Non-linear Projections: Finite-Width Guarantees and Benefits of Complex Eigenvalues (ICML 2024 · similarity 0.82)
• Tokens-per-Parameter Coverage Is Critical for Robust LLM Scaling Law Extrapolation (NeurIPS 2026 · similarity 0.82)