Research map / Large language models / Reasoning and chain of thought

Reasoning with Sampling: Your Base Model is Smarter Than You Think

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes training-free power-distribution sampling via autoregressive MCMC over base-model likelihoods, matching or beating GRPO single-shot while preserving pass@k diversity.

Problem

Whether RL-posttraining's single-shot reasoning gains can be elicited from base LLMs at inference time by sampling, without training, data, or a verifier, given that RL reduces output diversity.

Approach

Sample from the power distribution p^alpha of the base model (instead of low-temperature sampling) using a block-wise autoregressive Metropolis-Hastings scheme with random-resampling proposals; acceptance depends only on base-model likelihoods.

Key results

• On MATH500 with Qwen2.5-Math-7B, power sampling reaches 0.748 vs 0.496 base and 0.785 for GRPO.
• On HumanEval with Phi-3.5-mini-instruct, power sampling reaches 0.732 vs 0.134 for GRPO and 0.213 base.
• Power sampling keeps pass@k above GRPO for k>1 and approaches base-model upper bound at large k.

Surprising findings

• Low-temperature sampling is not equivalent to sampling from the power distribution; the paper proves this formally.
• Power sampling yields response lengths similar to GRPO (679 vs 671 tokens) without being rewarded for length.

Limitations

• Inference cost scales quadratically with sequence length (about 8.84x tokens at experimental settings).
• Hyperparameters alpha, block size B and MCMC steps are tuned empirically; results are sensitive to these choices.
• Evaluation is limited to three base models and a small set of benchmarks.

Experimental setup

Datasets: MATH MATH500 HumanEval GPQA Diamond AlpacaEval 2.0

Benchmarks: MATH500 HumanEval GPQA Diamond AlpacaEval 2.0

Metrics: accuracy win rate (length-controlled) pass@k average response length

Baselines: GRPO low-temperature sampling base model

Builds on: GRPO Metropolis-Hastings low-temperature sampling MCMC for LLM sampling (Zhao et al. 2024)

Closest papers by meaning

• Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpening (ICML 2026 · similarity 0.91)
• T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling (ICML 2025 · similarity 0.91)
• Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Models (ICLR 2026 · similarity 0.91)
• On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models (ICML 2026 · similarity 0.90)
• Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement (ICML 2026 · similarity 0.90)
• Tina: Tiny Reasoning Models via LoRA (ICLR 2026 · similarity 0.90)