Research map / Large language models / Reasoning and chain of thought

R-Diverse: Mitigating Diversity Illusion in Self-Play LLM Training

ICML 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Identifies "Diversity Illusion" in R-Zero-style self-play and proposes R-Diverse, which adds a persistent memory-bank penalty (MAP) and code-embedding-based skill-aware similarity (SAM) to sustain gains over iterations on Qwen3 base models.

Problem

Self-play LLM reasoning frameworks like R-Zero show non-sustained improvement: gains degrade after a few Challenger-Solver iterations because question diversity is enforced only within-batch and measured by surface lexical similarity.

Approach

Penalize Challenger questions against a persistent cross-iteration memory bank (max and mean similarity, plus experience replay for the Solver), and measure repetition in a skill space by canonicalizing questions into Python solver code and comparing code embeddings instead of BLEU.

Key results

• On Qwen3-4B-Base, R-Diverse reaches 52.59 Math AVG vs 49.07 for R-Zero and 42.58 for the base model.
• On Qwen3-8B-Base, R-Diverse reaches 56.46 Math AVG and 40.75 Overall AVG, beating R-Zero (54.69 / 38.73) and Socratic-Zero (39.15 Overall AVG).
• R-Zero's LLM-judge duplicate ratio rises from 71% to 84% across iterations, while R-Diverse lowers it from 59% to 53%.

Surprising findings

• R-Diverse with 3 iterations (R-Diverse*) is outperformed on some general benchmarks (e.g., BBEH on 4B) by the R-Zero baseline only marginally, and AIME25 scores fluctuate non-monotonically across iterations (e.g., 4B Iter 2 drops to 2.19).
• Ablating MAP causes the largest drop on the 4B model (52.59 to 49.62), larger than removing SAM (to 50.50).

Limitations

• Skill-aware measurement relies on Python code as a semantic bottleneck, which may not generalize to domains that are hard to formalize.
• Gains on general reasoning benchmarks are smaller than on math, since the curriculum is generated only in the math domain.
• Some AIME results are noisy and non-monotonic across iterations.

Experimental setup

Datasets: self-generated Challenger questions AMC MATH GSM8K Olympiad-Bench Minerva AIME24 AIME25 SuperGPQA MMLU-Pro BBEH

Benchmarks: AMC Minerva MATH GSM8K Olympiad-Bench AIME25 AIME24 SuperGPQA MMLU-Pro BBEH

Metrics: pass@1 (greedy) mean@32 Math AVG Overall AVG repetition ratio challenger entropy

Baselines: R-Zero Absolute Zero SPIRAL Socratic-Zero Base Model

Builds on: R-Zero GRPO BLEU EasyR1

Closest papers by meaning

• Whatever Remains Must Be True: Filtering Drives Reasoning in LLMs, Shaping Diversity (ICLR 2026 · similarity 0.90)
• Diversity-Enhanced Reasoning for Subjective Questions (ICLR 2026 · similarity 0.89)
• Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examples (ICML 2025 · similarity 0.89)
• Beyond Pass@ 1: Self-Play with Variational Problem Synthesis Sustains RLVR (ICLR 2026 · similarity 0.89)
• Diversity-Aware Policy Optimization for Large Language Model Reasoning (NeurIPS 2025 · similarity 0.89)
• Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability (ICML 2026 · similarity 0.89)