Research map / Large language models / Reasoning and chain of thought

Reasoning via Test-Time Instance-Level Policy Gradient in Latent Space

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Search and self-correction in reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LatentSeek, which tests-time optimizes per-instance latent representations of an LLM via REINFORCE policy gradient guided by self-generated rewards, improving math reasoning without parameter updates.

Problem

Improving LLM reasoning at test time without updating parameters, where prior latent-space methods require training and token-space test-time scaling is limited.

Approach

Treat the token-wise latent representations (transformer outputs before the LM head) of a CoT initialization as free variables and optimize them per instance with REINFORCE, using the model's own self-generated reward (or a perfect sharp reward in ideal experiments), then decode greedily and continue generation.

Key results

• LatentSeek (Self) averages 83.29 on GSM8K Prompt 1 vs 73.43 for CoT and 80.47 for BoN across five backbones.
• With LLaMA3.1-8B-Instruct, LatentSeek reaches 83.7 on GSM8K and 54.6 on MATH-500 (average 69.2), above Genius (56.5) and SimpleRL-Zoo (51.1) without training.
• With a perfect sharp reward model, Qwen2.5-1.5B-Instruct reaches 82.8 on MATH-500 with 61.8 average iterations, trailing o1-preview (85.5) by 2.7 points.

Surprising findings

• Generated reasoning often contains incoherent tokens yet still reaches correct answers, suggesting latent optimization does not need human-readable traces.
• A constant negative reward (no guidance) drops average accuracy by 14.21 points, while a sharp all-or-nothing reward still improves results.

Limitations

• Relies on self-rewarding, bounded by the base model's own judging ability; public ORMs were found not robust enough.
• Experiments limited to models up to 14B parameters and to math benchmarks.
• Theoretical MIP analysis is heuristic and does not account for gradient-induced changes.

Experimental setup

Datasets: GSM8K MATH-500 AIME2024 Magpie

Benchmarks: GSM8K MATH-500 AIME2024

Metrics: accuracy

Baselines: Chain-of-Thought Few-Shot CoT Best-of-N Self-Reflection Self-Consistency Self-Rewarding ScPO CoH Genius SPIN iCoT SimpleRL-Zoo

Builds on: REINFORCE chain-of-thought Best-of-N Self-Consistency

Closest papers by meaning

• Thinking on the Fly: Test-Time Reasoning Enhancement via Latent Thought Policy Optimization (ICLR 2026 · similarity 0.94)
• Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning (ICML 2025 · similarity 0.92)
• Hybrid Latent Reasoning via Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
• Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes (ICLR 2026 · similarity 0.90)
• Efficient Reasoning via Constrained Optimization in Latent Space (NeurIPS 2026 · similarity 0.90)
• Training Language Models to Reason Efficiently (NeurIPS 2025 · similarity 0.90)