Research map / Large language models / Reasoning and chain of thought

Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach

NeurIPS 2025 · Large language models › Reasoning and chain of thought · approach: Efficient and adaptive thinking · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Trains a 3.5B-parameter depth-recurrent transformer (Huginn) on 800B tokens with random recurrence counts so that test-time latent iterations improve reasoning benchmarks without verbalized chain-of-thought.

Problem

Test-time compute scaling usually relies on generating long chain-of-thought tokens, which needs specialized data, long contexts, and verbalization; the paper asks whether a language model can scale reasoning by iterating a recurrent block in latent space.

Approach

A decoder-only transformer split into prelude, a shared recurrent core block (input-injected from the embedded input with random state initialization), and coda; trained with a log-normal Poisson sampled recurrence count and truncated backpropagation through the last k=8 iterations, so depth can be unrolled arbitrarily at inference.

Key results

• At 0.8T tokens, Huginn with r=32 reaches 34.80/42.08 GSM8K CoT versus 0.00 with r=1 for the same checkpoint, showing recurrence drives reasoning gains.
• Huginn-0125 (3.5B) reaches 23.17 HumanEval pass@1 and 24.80 MBPP, outperforming general open models like OLMo-7B-0724 on HumanEval? No: it beats all general-purpose open models except OLMo-2 on math and beats them on code, while trailing StarCoder2.
• The recurrent model at 180B tokens substantially outperforms the non-recurrent baseline on GSM8K CoT (about 5x) and ARC-C, and the model's performance grows with iterations (ARC-E 69.91 at r=32 vs 34.89 at r=1).

Measured effects reported in the paper

Surprising findings

• The recurrent model with a single recurrence stops improving between early and final checkpoints, implying later gains are encoded entirely in the recurrent block.
• Earlier training runs with a learned adapter and standard pre-norm collapsed in token correlation or learned to ignore the incoming state, so the sandwich norm was required at scale.

Limitations

• Proof-of-concept single training run with a constant learning rate that is never cooled down and an untested data mixture.
• Results lag behind later OLMo models and dedicated code models such as StarCoder2; memorization of facts is weaker (closed-book QA).

Experimental setup

Datasets: smollm-fineweb-edu starcoderdata RedPajama Dolma The Stack v2 MathPile AutoMathText OpenMathInstruct MetaMathQA GSM8K Infinity-Instruct

Benchmarks: ARC-E ARC-C HellaSwag MMLU OpenBookQA PiQA SciQ WinoGrande GSM8K MATH (Minerva) MathQA MBPP

Metrics: accuracy normalized accuracy exact match pass@1 MT-Bench score

Baselines: Amber Pythia OLMo-7B OLMo-2-1124-7B StarCoder2 non-recurrent fixed-depth baseline

Builds on: looped transformers deep equilibrium models truncated backpropagation through time

Closest papers by meaning

• Stabilizing Recurrent Dynamics for Test-Time Scalable Latent Reasoning in Looped Language Models (ICML 2026 · similarity 0.91)
• Efficient Reasoning via Constrained Optimization in Latent Space (NeurIPS 2026 · similarity 0.90)
• Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens (ICML 2026 · similarity 0.89)
• ThreadWeaver: Adaptive Threading for Efficient Parallel Reasoning in Language Models (ICML 2026 · similarity 0.89)
• TrimR: Verifier-based Training-Free Thinking Trimming for Efficient Test-Time Scaling (ICLR 2026 · similarity 0.89)
• Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoning (NeurIPS 2025 · similarity 0.89)