Research map / Large language models / Reasoning and chain of thought

Diversity-Incentivized Exploration for Versatile Reasoning

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DIVER, an RLVR method that adds a potential-based intrinsic reward based on global sequence-level diversity (BLEU-based textual and formula-based equational diversity) with conditional and clipped shaping to avoid reward hacking, improving math and out-of-domain reasoning on Qwen2.5-Math-7B.

Problem

Existing RLVR exploration methods mostly add token-level entropy or uncertainty bonuses, leaving sample inefficiency and weak exploration of diverse global reasoning paths in LLM reasoning under sparse rewards.

Approach

Treat group-level sequence diversity (BLEU-based textual dissimilarity and formula-based equational diversity) as an intrinsic reward, shaped via potential-based reward shaping so that the sequence-level bonus equals the final response's diversity while preserving optimal policy invariance; mitigate hacking by clipping, decaying the shaping ratio, and rewarding only correct responses.

Key results

• DIVER-MIX averages 43.1 on six in-domain math benchmarks and 58.8 on out-of-domain tasks with Qwen2.5-Math-7B, versus 40.7/55.2 for GRPO w/ Clip-higher.
• DIVER achieves 50.0 Pass@32 on AIME25 (+6.7 over Entropy-RL) and 68.9 on OlympiadBench (+2.1 over GRPO w/ Clip-higher).
• Under an LLM judge (DeepSeek-V3.2-Exp), DIVER solutions rank best in 76.4% of cases, with average rank 1.52 versus 2.57–3.19 for baselines.

Surprising findings

• Rewarding diversity on incorrect or all responses severely degrades test accuracy and causes response length explosion, so conditional shaping on correct responses is necessary.
• Standard GRPO-style diversity measures (TD) can be higher for the GRPO baseline than for DIVER in the correlation table, indicating the diversity metric and reasoning quality are not monotonically aligned.

Limitations

• Only single-turn math RLVR is studied; multi-turn agent settings are left for future work.
• Equation extraction has an estimated ~8.4% failure rate, and the diversity metrics are lexical heuristics rather than semantic.
• Gains on in-domain benchmarks are modest (about 1–2 points) and many results are within reported standard deviations.

Experimental setup

Datasets: OpenR1-Math-220k NuminaMath 1.5

Benchmarks: AIME24 AIME25 AMC MATH-500 Minerva OlympiadBench ARC-c GPQA-diamond MMLU-Pro

Metrics: Avg@32 Pass@1 Pass@k

Baselines: SimpleRL-Zoo OpenReasoner-Zero PRIME-Zero GRPO w/ Clip-higher Entropy-RL Pass@k Training

Builds on: GRPO potential-based reward shaping LUFFY Pass@k Training

Closest papers by meaning

• Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward (ICLR 2026 · similarity 0.92)
• Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs (ICLR 2026 · similarity 0.92)
• SAGE: Shaping Anchors for Guided Exploration in RLVR of LLMs (ICML 2026 · similarity 0.91)
• DeepSearch: Overcome the Bottleneck of Reinforcement Learning with Verifiable Rewards via Monte Carlo Tree Search (ICLR 2026 · similarity 0.91)
• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.90)
• Whatever Remains Must Be True: Filtering Drives Reasoning in LLMs, Shaping Diversity (ICLR 2026 · similarity 0.90)