Research map / Large language models / Reasoning and chain of thought

Diversity-Enhanced Reasoning for Subjective Questions

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MultiRole-R1, which SFTs LRMs on self-synthesized multi-role reasoning chains and then runs GRPO with a token-level diversity reward shaping term, trained only on subjective questions, to improve subjective reasoning and diversity.

Problem

RLVR degrades generation diversity in large reasoning models, and existing diversity-enhanced training targets objective tasks with a single ground truth, leaving subjective questions with multiple role-dependent valid answers under-addressed.

Approach

Combines perspective diversity (self-synthesized multi-role reasoning chains used for SFT) with token-level diversity (a composite lexical/structural/discourse diversity score used as a shaping reward alongside the role-aware accuracy reward in GRPO).

Key results

• MultiRole-R1 raises in-domain accuracy by 14.1% on average and OOD accuracy by 7.64% versus zero-shot CoT, trained only on subjective data.
• On AIME 2024, MultiRole-R1 reaches 63.2% on R1-Distill-Qwen-7B (vs 55.5% zero-shot) and 80.1% on Qwen3-8B (vs 76.0%).
• Per-task correlation between diversity and accuracy averages 0.736 versus 0.554 for length and accuracy.

Surprising findings

• Accuracy gains coincide with shorter responses: MultiRole-R1 averages 657.8 words versus 1572.9 for SFT, suggesting diversity rather than verbosity drives accuracy.
• Standard entropy regularization in GRPO led to entropy collapse around 70 steps, while reward shaping was stable.

Limitations

• Evaluation restricted to multiple-choice subjective benchmarks; free-form creative tasks are not covered.
• Role-aware accuracy depends on existing role-dependent ground truths available only for a few datasets.
• Diversity metric is a hand-designed equal-weight composite, and its weighting is not validated beyond human alignment on a small set.

Experimental setup

Datasets: BBQ GlobalOpinionQA ETHICS CALI CSQA GSM8K AIME 2024

Benchmarks: BBQ GLOQA ETHICS CALI CSQA GSM8K AIME 2024

Metrics: pass@1 accuracy length-normalized diversity score pass@k Pearson correlation between accuracy and diversity human alignment score

Baselines: Zero-shot CoT Self-Refine Role-Play More think SelfConsis SFT SelfConsis SFT+DPO SelfConsis SFT+GRPO

Builds on: GRPO DeepSeek-R1 budget forcing self-consistency Llama-Factory

Closest papers by meaning

• Diversity-Aware Policy Optimization for Large Language Model Reasoning (NeurIPS 2025 · similarity 0.90)
• Diversity-Incentivized Exploration for Versatile Reasoning (ICLR 2026 · similarity 0.90)
• Whatever Remains Must Be True: Filtering Drives Reasoning in LLMs, Shaping Diversity (ICLR 2026 · similarity 0.90)
• R-Diverse: Mitigating Diversity Illusion in Self-Play LLM Training (ICML 2026 · similarity 0.89)
• SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoning (ICML 2026 · similarity 0.89)
• Native Reasoning Models: Training Language Models to Reason on Unverifiable Data (ICLR 2026 · similarity 0.88)