Research map / Large language models / Reasoning and chain of thought

Representation-Based Exploration for Language Models: From Test-Time to Post-Training

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes representation-based elliptical exploration bonuses from LM hidden states, applied both to inference-time coreset selection (RepExp) and as a reward bonus in GRPO post-training, to improve verifier efficiency, pass@k, and move beyond sharpening.

Problem

RL post-training of language models may only sharpen behaviors already in the base model rather than discover novel ones, and it is unclear which exploration technique scales to LMs with combinatorially large response spaces.

Approach

Adapt elliptical (leverage-score) bonuses from linear bandits to LMs using mean-pooled last-layer hidden states as features; use them to select diverse candidate responses at inference time and as sequence-level reward bonuses in GRPO, validating exploration first in the inference-time selection setting.

Key results

• On Qwen-2.5-14B-Instruct, RepExp obtains roughly 50% improvement in samples-to-correct (verifier efficiency) over random sampling on MATH, GSM8K, MBPP+, and Game-of-24.
• On AIME 2024, post-trained Qwen-2.5-7B-Instruct pass@80 matches GRPO's pass@256, a claimed 3x test-time sample efficiency gain.
• RepExp post-training preserves or improves pass@k at large k where standard GRPO degrades below the base model; responses are less likely under the base model than GRPO's.

Surprising findings

• Representation-based exploration hurts or gives no benefit for weaker models such as Qwen-2.5-0.5B, with benefits growing with model strength.
• Combining the exploration bonus with high-temperature sampling (tau=1.5) does not improve verifier efficiency, which the authors attribute to incoherent responses looking novel in representation space.

Limitations

• Post-training experiments are limited to Qwen-2.5-7B-Instruct on MATH, GSM8K, and AIME 2024.
• Token-level autoregressive exploration is computationally expensive, requiring one extra forward pass per vocabulary token per step, and was only tested on MATH with 200 hard questions.
• RepExp adds about 1.37x wall-clock time per step versus GRPO in post-training.

Experimental setup

Datasets: MATH GSM8K MBPP+ Game-of-24 AIME 2025 AIME 2024 DAPO-Math-17K MATH-500

Benchmarks: MATH GSM8K MBPP+ Game-of-24 AIME 2025 AIME 2024

Metrics: pass@k samples-to-correct verifier efficiency pass@1

Baselines: GRPO Unlikeliness Base Model random sampling min-p sampling nucleus sampling

Builds on: GRPO elliptical bonus / linear bandits unlikeliness reward verl

Closest papers by meaning

• Behavior Injection: Preparing Language Models for Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
• Reasoning with Exploration: An Entropy Perspective (AAAI 2026 · similarity 0.90)
• Can large language models explore in-context? (NeurIPS 2024 · similarity 0.90)
• TTRL: Test-Time Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
• On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models (ICML 2026 · similarity 0.90)
• Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.90)