Research map / Large language models / Reasoning and chain of thought

PowerFlow: Unlocking the Dual Nature of LLMs via Principled Distribution Matching

ICML 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PowerFlow, a GFlowNet-based length-aware trajectory-balance method that trains LLMs toward alpha-power distributions of their base model, sharpening for reasoning and flattening for creative writing without external rewards.

Problem

Heuristic intrinsic rewards in unsupervised RL from internal feedback (RLIF) lack a principled optimization target and suffer length collapse/explosion and overconfidence; over-sharpened aligned models also lose creative diversity.

Approach

Reformulates unsupervised fine-tuning as distribution matching to the alpha-power (escort) distribution of the base model, trained with a length-aware trajectory-balance objective that replaces the partition function with a per-token normalized energy term (log Z' raised to length) so gradients are scale-invariant to response length.

Key results

• On Qwen2.5-Math-1.5B, PowerFlow averages 34.30 across six reasoning benchmarks vs. GRPO 32.75 and EMPO 32.45 (avg@16).
• On Qwen2.5-Math-7B, PowerFlow averages 42.17 vs. GRPO 42.38 and TTRL 41.18, roughly matching supervised GRPO.
• For creative writing with alpha=0.5, PowerFlow is reported to shift the quality-diversity Pareto frontier, improving both quality and semantic diversity over Instruct models.

Surprising findings

• Pure alpha-power distribution matching (TB-traj) collapses response length and drives accuracy toward zero during training, so the length correction is essential.
• Token-level normalized variants (TB-token, RL-token) show initial gains followed by decay due to exploitation of repetitive tokens.

Limitations

• Alpha is set heuristically per model family (alpha=4 for base, 2 for instruct) without automated per-model tuning.
• RLIF baselines use released official checkpoints rather than retrained models, complicating fair comparison.
• Length-aware objective does not preserve the full global ranking of the target distribution; only within-length-shell ranking is exactly preserved.

Experimental setup

Datasets: NuminaMath-CoT PoemHunter prompts BookMIA Reddit r/DadJokes

Benchmarks: MATH500 OlympiadBench AIME24 AIME25 AMC23 GPQA

Metrics: avg@16 accuracy semantic diversity ROUGE-L lexical redundancy LLM-judged quality solution diversity score

Baselines: Intuitor EMPO TTRL PowerSampling One-shot EM GRPO Low-temp Format-only Instruct High-temp VS-Standard

Builds on: GFlowNet Trajectory Balance GRPO DAPO EMPO Intuitor TTRL FlowRL

Closest papers by meaning

• Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpening (ICML 2026 · similarity 0.90)
• Amortizing intractable inference in large language models (ICLR 2024 · similarity 0.89)
• Power Distribution Bridges Sampling, Self-Reward RL, and Self-Distillation (NeurIPS 2026 · similarity 0.88)
• Pre-Training and Fine-Tuning Generative Flow Networks (ICLR 2024 · similarity 0.86)
• SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoning (ICML 2026 · similarity 0.86)
• Random Policy Evaluation Uncovers Policies of Generative Flow Networks (ICML 2025 · similarity 0.86)