Research map / Large language models / Reasoning and chain of thought

PonderLM-2: Pretraining LLM with Latent Thoughts in Continuous Space

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Search and self-correction in reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Pretrains LMs to generate a latent thought (last hidden state) before each next token, parallelized with Jacobi iteration, showing that one latent thought per token beats a 2x-parameter standard model at equal inference cost.

Problem

Improving per-token prediction quality in language model pretraining by scaling computation per token, as an alternative to vertical depth scaling and discrete chain-of-thought, at matched inference cost.

Approach

Feed each position's last hidden state back as an extra input token (interleaved with token embeddings) so the model performs a latent thinking step before predicting the next token; train in parallel using Jacobi fixed-point iteration with randomized iteration counts K in {2,3}, using only a standard pretraining objective on general corpora.

Key results

• PonderLM-2-Pythia-1.4B averages 58.5% zero-shot vs 57.3% for Pythia-2.8B and 55.4% for TinyLlama-1.1B on nine downstream tasks.
• PonderLM-2-LLaMA-1.4B reaches 52.3% zero-shot average vs 49.8% for LLaMA-2.8B trained from scratch, at 2x inference FLOPs.
• On LLaMA-3-3B continual pretraining, PonderLM-2 CPT raises 5-shot average from 66.4 to 67.7 over the original model, versus 66.2 for standard CPT.

Surprising findings

• Decoding the latent thought directly (Stage 0) often places the correct token at Top-1 with low probability, which the next step then sharpens.

Limitations

• Training FLOPs are roughly 8x the vanilla baseline, and inference KV-cache is doubled with one latent thought per token.
• Gains on GSM8K without CoT are modest in absolute terms (EM 12.59 vs 10.01).

Experimental setup

Datasets: The Pile SlimPajama Alpaca Wikitext LAMBADA GSM8K TruthfulQA

Benchmarks: Pile validation Wikitext LAMBADA LAMBADA-OpenAI ARC-Easy ARC-Challenge Winogrande PIQA HellaSwag SciQ RACE MT-Bench

Metrics: perplexity accuracy average accuracy exact match MT-Bench score throughput (tokens/s)

Baselines: Pythia PonderLM-Pythia OPT Bloom TinyLlama Looped Transformer Pause Token Pondering LLM LLaMA-2.8B

Builds on: PonderLM Coconut Pythia Jacobi iteration Chain-of-Thought

Closest papers by meaning

• Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens (ICML 2026 · similarity 0.90)
• Think Thrice Before You Act: Progressive Thought Refinement in Large Language Models (ICLR 2025 · similarity 0.90)
• Making Slow Thinking Faster: Compressing LLM Chain-of-Thought via Step Entropy (ICLR 2026 · similarity 0.89)
• Latent Thoughts Tuning: Bridging Context and Reasoning with Fused Information in Latent Tokens (ICML 2026 · similarity 0.89)
• Rethinking Thinking Tokens: LLMs as Improvement Operators (ICML 2026 · similarity 0.89)
• Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning (ICML 2025 · similarity 0.89)