AI Paper Atlas

Research map / Large language models / Reasoning and chain of thought

Escaping the Mode: Multi-Answer Reinforcement Learning in LMs

Isha Puri, Mehul Damani, Idan Shenfeld, Marzyeh Ghassemi, Jacob Andreas, Yoon Kim

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Routing, pruning and distillation for reasoning

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes multi-answer reinforcement learning so that language models keep a distribution over multiple valid answers instead of collapsing onto one dominant mode.

Problem

Post-training of language models often collapses the implicit answer distribution onto a single dominant mode, which is harmful for tasks with multiple valid answers or irreducible uncertainty (e.g., medical diagnosis, scientific hypothesis generation).

Approach

Uses reinforcement learning to train the model to produce multiple distinct valid answers rather than one dominant answer, so the output distribution over answers is preserved after post-training. Abstract-only information; specific mechanism not stated.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
train with multi-answer RLVR reward instead of single-answer RLVRRLVR-Single · DDXPlus (k=3)average correct per set: 0.62 -> 0.79Improved
train with multi-answer RLVR reward instead of single-answer RLVRRLVR-Single · MBPP (k=3)average correct per set: 0.98 -> 1.35Improved
train with multi-answer RLVR reward instead of single-answer RLVRRLVR-Single · MBPP (k=3)average tokens: 511.73 -> 235.49Improved
train with multi-answer RLVR reward instead of single-answer RLVRRLVR-Single · MBPP (k=3)Top-1 accuracy: 0.29 -> 0.49Improved
train with multi-answer RLVR reward instead of single-answer RLVRRLVR-Single · DDXPlus (k=3)Top-1 accuracy: 0.50 -> 0.42Hurt
train with multi-answer RLVR reward instead of single-answer RLVRRLVR-Single · HotPotQA-Modified (k=3)average correct per set: 0.17 -> 0.27Improved
prompt a single-answer-trained model for multiple answers instead of training for multi-answer (ablation)RLVR-Single · DDXPlus (k=3)average correct per set: 0.62 -> 0.67 (Multi-Answer RLVR 0.79)Mixed
prompt RLCR-Single model with multi prompt (ablation)RLCR-Single · DDXPlus (k=3)average correct per set: 0.65 -> 0.49Hurt
add calibration reward (RLCR) to multi-answer RL (ablation)Multi-Answer RLVR · DDXPlus (k=3)set ECE: 0.16 -> 0.01Improved
add calibration reward (RLCR) to multi-answer RL (ablation)Multi-Answer RLVR · MBPP (k=3)set ECE: 0.54 -> 0.37Improved
multi-answer RLCR versus single-answer RLCR calibration on HotPotQARLCR · HotPotQA-Hard (k=3)set ECE: 0.13 -> 0.31Hurt
multi-answer training does not degrade top-answer calibrationMulti-Answer RL · DDXPlus (k=3)Top-1 ECE: RLCR-S 0.02 vs RLCR-M 0.02No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy