Research map / Large language models / Reasoning and chain of thought
Escaping the Mode: Multi-Answer Reinforcement Learning in LMs
Isha Puri, Mehul Damani, Idan Shenfeld, Marzyeh Ghassemi, Jacob Andreas, Yoon Kim
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes multi-answer reinforcement learning so that language models keep a distribution over multiple valid answers instead of collapsing onto one dominant mode.
Problem
Post-training of language models often collapses the implicit answer distribution onto a single dominant mode, which is harmful for tasks with multiple valid answers or irreducible uncertainty (e.g., medical diagnosis, scientific hypothesis generation).
Approach
Uses reinforcement learning to train the model to produce multiple distinct valid answers rather than one dominant answer, so the output distribution over answers is preserved after post-training. Abstract-only information; specific mechanism not stated.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with multi-answer RLVR reward instead of single-answer RLVR | RLVR-Single · DDXPlus (k=3) | average correct per set: 0.62 -> 0.79 | Improved |
| train with multi-answer RLVR reward instead of single-answer RLVR | RLVR-Single · MBPP (k=3) | average correct per set: 0.98 -> 1.35 | Improved |
| train with multi-answer RLVR reward instead of single-answer RLVR | RLVR-Single · MBPP (k=3) | average tokens: 511.73 -> 235.49 | Improved |
| train with multi-answer RLVR reward instead of single-answer RLVR | RLVR-Single · MBPP (k=3) | Top-1 accuracy: 0.29 -> 0.49 | Improved |
| train with multi-answer RLVR reward instead of single-answer RLVR | RLVR-Single · DDXPlus (k=3) | Top-1 accuracy: 0.50 -> 0.42 | Hurt |
| train with multi-answer RLVR reward instead of single-answer RLVR | RLVR-Single · HotPotQA-Modified (k=3) | average correct per set: 0.17 -> 0.27 | Improved |
| prompt a single-answer-trained model for multiple answers instead of training for multi-answer (ablation) | RLVR-Single · DDXPlus (k=3) | average correct per set: 0.62 -> 0.67 (Multi-Answer RLVR 0.79) | Mixed |
| prompt RLCR-Single model with multi prompt (ablation) | RLCR-Single · DDXPlus (k=3) | average correct per set: 0.65 -> 0.49 | Hurt |
| add calibration reward (RLCR) to multi-answer RL (ablation) | Multi-Answer RLVR · DDXPlus (k=3) | set ECE: 0.16 -> 0.01 | Improved |
| add calibration reward (RLCR) to multi-answer RL (ablation) | Multi-Answer RLVR · MBPP (k=3) | set ECE: 0.54 -> 0.37 | Improved |
| multi-answer RLCR versus single-answer RLCR calibration on HotPotQA | RLCR · HotPotQA-Hard (k=3) | set ECE: 0.13 -> 0.31 | Hurt |
| multi-answer training does not degrade top-answer calibration | Multi-Answer RL · DDXPlus (k=3) | Top-1 ECE: RLCR-S 0.02 vs RLCR-M 0.02 | No clear effect |
Closest papers by meaning
- Answer, Assemble, Ace: Understanding How LMs Answer Multiple Choice Questions (ICLR 2025 · similarity 0.89)
- Thinker: Learning to Think Fast and Slow (NeurIPS 2025 · similarity 0.89)
- Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty (ICLR 2026 · similarity 0.89)
- Representation Consistency for Accurate and Coherent LLM Answer Aggregation (NeurIPS 2025 · similarity 0.88)
- Incentivizing LLMs to Self-Verify Their Answers (NeurIPS 2025 · similarity 0.88)
- Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
