Research map / Large language models / Reasoning and chain of thought

Power Distribution Bridges Sampling, Self-Reward RL, and Self-Distillation

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows the power distribution links power sampling, self-reward KL-regularized RL, and self-distillation, and proposes power self-distillation, an offline SFT surrogate that amortizes power sampling into a single model.

Problem

Power sampling (sharpening the base model toward pi^alpha) improves reasoning but is expensive at inference; its relation to RL and distillation is unclear, and cheap local approximations to it are not well understood.

Approach

Identifies the power distribution as the closed-form optimum of KL-regularized RL with the model's own sequence log-likelihood as reward, then replaces the reverse KL with a forward KL so the target can be learned by offline MLE on teacher samples drawn from the power distribution (power self-distillation). Also proves per-token temperature and one-step SIS proposals cannot match the sequence-leve

Key results

• On MATH500 with Qwen2.5-Math-7B, power sampling raises accuracy from 0.508 (standard) to 0.714 and the power-distilled model with temperature decoding reaches 0.722 accuracy at lower inference cost.
• Power self-distillation with temperature decoding attains the highest self-reward (-0.043) and accuracy (0.722) among the reported settings on MATH500.
• Self-reward Best-of-N gives the largest true-reward gains across models (base standard 0.508 to 0.680), while the randomly initialized model shows no accuracy gain, consistent with the covariance condition.

Surprising findings

• Token-wise temperature scaling with the same exponent recovers much of the power sampling gain on the base model (0.683 vs 0.714 accuracy), and the distilled model with plain temperature decoding matches or exceeds power sampling.

Limitations

• Gains are bounded by the base model's capability; small gains when the base is weak.
• Analysis is limited to autoregressive models over finite horizons.
• Sharpening bound relies on a finite model class with size M and a large-alpha limit that is not practical.

Experimental setup

Datasets: MATH MBPP GPQA HumanEval

Benchmarks: MATH500 HumanEval GPQA-Diamond

Metrics: accuracy self-reward (average completion log-likelihood)

Baselines: Standard autoregressive sampling Temperature (token-wise) Power sampling (MH)

Builds on: power sampling (Metropolis-Hastings) distribution sharpening reward-augmented maximum likelihood KL-regularized RL closed form LoRA

Closest papers by meaning

• Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpening (ICML 2026 · similarity 0.92)
• Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Models (ICLR 2026 · similarity 0.89)
• Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.89)
• Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward (ICLR 2026 · similarity 0.89)
• Reasoning with Sampling: Your Base Model is Smarter Than You Think (ICLR 2026 · similarity 0.89)
• T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling (ICML 2025 · similarity 0.89)