Research map / Large language models / Reasoning and chain of thought
Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EMPO, a fully unsupervised RL method that minimizes semantic entropy over clustered sampled answers on unlabeled questions to incentivize reasoning in base LLMs.
Problem
Enhancing LLM reasoning typically requires external supervision (labeled traces, golden answers, or reward models); the paper asks how to incentivize reasoning in a fully unsupervised manner.
Approach
Use the semantic entropy of a group of sampled responses (clustered by meaning, via regex for math or a small verifier model for free-form tasks) as the RL reward, i.e., reward each output by the probability of its meaning cluster, with entropy thresholding to avoid reward hacking and KL removed.
Key results
- Qwen2.5-Math-7B average math accuracy rises from 30.7% to 48.1% with EMPO without supervision, close to Qwen2.5-Math-Instruct (49.4%) and above GRPO (46.8%).
- Qwen2.5-7B MMLU-Pro accuracy improves from 32.1% to 50.1% with EMPO, versus 54.5% for GRPO with golden answers.
- On GPQA, Qwen2.5-7B improves from 15.9% to 28.8% with EMPO, while 14B MMLU-Pro goes from 32.7% to 58.8%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply EMPO to Qwen2.5-Math 1.5B Base | Qwen2.5-Math 1.5B Base · Math benchmarks (Avg.) | accuracy: 28.1% to 42.1% | Helped |
| apply EMPO to Qwen2.5-Math 7B Base | Qwen2.5-Math 7B Base · Math benchmarks (Avg.) | accuracy: 30.7% to 48.1% | Helped |
| apply EMPO instead of GRPO | Qwen2.5-Math 7B Base · Math benchmarks (Avg.) | accuracy: 46.8 -> 48.1 | Helped |
| apply EMPO to Qwen2.5-7B Base | Qwen2.5-7B Base · MMLU-Pro (Avg.) | accuracy: 32.1% to 50.1% | Helped |
| apply EMPO to Qwen2.5-14B Base | Qwen2.5-14B Base · MMLU-Pro (Avg.) | accuracy: 32.7% to 58.8% | Helped |
| apply EMPO to Qwen2.5-7B Base | Qwen2.5-7B Base · GPQA | accuracy: 15.9% to 28.8% | Helped |
| apply EMPO to Qwen2.5-14B Base | Qwen2.5-14B Base · GPQA | accuracy: 30.6% to 35.3% | Helped |
| apply EMPO instead of GRPO | Qwen2.5-7B Base · MMLU-Pro (Avg.) | accuracy: 54.5 -> 50.1 | Hurt |
| apply EMPO instead of GRPO | Qwen2.5-7B Base · GPQA | accuracy: 33.8 -> 28.8 | Hurt |
| apply EMPO instead of GRPO | Qwen2.5-14B Base · MMLU-Pro (Avg.) | accuracy: 59.6 -> 58.8 | Hurt |
| apply EMPO instead of GRPO | Qwen2.5-14B Base · GPQA | accuracy: 35.6 -> 35.3 | Hurt |
| use EMPO with DeBERTa semantic clustering instead of General-Verifier | EMPO on free-form natural reasoning · Free-form natural reasoning | training stability | Hurt |
Surprising findings
- SFT on Natural Reasoning reference responses often hurts performance (e.g., Qwen2.5-7B MMLU-Pro drops to 25.6%), attributed to noisy references.
- Base models catch up with or exceed RL-trained models in pass@k at large k, suggesting RL mainly improves sampling efficiency.
Limitations
- Free-form clustering depends on a 1.5B verifier model; weaker clustering (DeBERTa) led to reward hacking.
- Llama3 base models could not be trained fully unsupervised and required an instruct-tuned starting point.
- Gains are reported mainly on Qwen2.5 base models; the unsupervised approach relies on the base model already having consistent reasoning behavior.
Experimental setup
Datasets: NuminaMath-CoT Natural Reasoning
Benchmarks: MATH Minerva Math OlympiadBench AIME24 AMC23 MMLU-Pro GPQA
Metrics: pass@1 accuracy (greedy) pass@k
Baselines: GRPO SFT Online-DPO Qwen2.5-Math-Instruct Qwen2.5-Instruct Llama-3.1-70B-Instruct Eurus-2-7B-PRIME
Builds on: GRPO semantic entropy Tent Open-R1 DeepSeek-R1-Zero
Closest papers by meaning
- The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning (NeurIPS 2025 · similarity 0.92)
- General-Reasoner: Advancing LLM Reasoning Across All Domains (NeurIPS 2025 · similarity 0.90)
- Reinforcing General Reasoning Without Verifiers (ICLR 2026 · similarity 0.90)
- LogicReward: Incentivizing LLM Reasoning via Step-Wise Logical Supervision (ICLR 2026 · similarity 0.89)
- $\textbf{Re}^{2}$: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving (ICLR 2026 · similarity 0.89)
- Rectifying LLM Thought from Lens of Optimization (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.