Research map / Reinforcement learning / RL from feedback

Jointly Reinforcing Diversity and Quality in Language Model Generations

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: RL for LLM reasoning · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DARLING, an online RL method that multiplies a learned semantic-equivalence-based diversity score with a quality reward so GRPO updates favor responses that are both high-quality and semantically distinct from other rollouts.

Problem

Post-training of LMs sharpens output distributions and collapses diversity, reducing usefulness for creative and exploratory tasks, while improving quality.

Approach

Measure diversity at the semantic level with a learned equivalence classifier over rollouts from the same prompt, and multiply the normalized diversity with the quality reward to form the advantage in GRPO, so high-quality, distinct responses get amplified gradient updates.

Key results

• On Llama-3.1-8B-Instruct, DARLING reaches 55.2% AlpacaEval 2.0 LCWR vs 48.7% for GRPO and 5.49 distinct responses on NoveltyBench vs 2.08 for GRPO.
• On Qwen3-14B-Base math, DARLING improves average pass@1 by +1.90% over GRPO and pass@128 by +10.16%, with the largest gains on HMMT.
• On Llama-3.3-70B-Instruct, DARLING reaches EQ-Bench ELO 1531 vs 1261 for GRPO.

Surprising findings

• Lexical 4-gram diversity reward leads the policy to hack the reward by generating post-answer self-reflections, and it underperforms GRPO on math pass@1.
• Removing the std normalization in GRPO improved both quality and semantic diversity in the non-verifiable setting with dense noisy reward-model scores.

Limitations

• Diversity classifier relies on human annotations and is trained on small data (1000 NoveltyBench annotations for non-verifiable; math classifier from Llama-3.3-70B annotations), limiting generality.
• Quality reward from a reward model (Athene-RM-8B) may be vulnerable to reward hacking, as the authors note for Skywork RM in NoveltyBench.
• Experiments limited to Llama-3 and Qwen3 models on two task types with a single seed-level comparison; few baselines.

Experimental setup

Datasets: WildChat DeepScaleR NoveltyBench annotations

Benchmarks: AlpacaEval 2.0 ArenaHard v2.0 ArenaHard v1.0 EQ-Bench NoveltyBench AIME25 OlympiadBench HMMT 2025 BRUMO 2025

Metrics: length-controlled win rate win rate ELO distinct count distinct-4 pass@1 pass@k

Baselines: GRPO DivPO GRPO-Unlikeliness 4gram diversity reward

Builds on: GRPO DivPO NoveltyBench verl DeepScaleR

Closest papers by meaning

• Diversity-Aware Policy Optimization for Large Language Model Reasoning (NeurIPS 2025 · similarity 0.90)
• Representation-Based Exploration for Language Models: From Test-Time to Post-Training (ICLR 2026 · similarity 0.90)
• Diversity as a Reward: Fine-Tuning LLMs on a Mixture of Domain-Undetermined Data (NeurIPS 2025 · similarity 0.88)
• SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoning (ICML 2026 · similarity 0.88)
• Diversity-Incentivized Exploration for Versatile Reasoning (ICLR 2026 · similarity 0.88)
• Quality Diversity through Human Feedback: Towards Open-Ended Diversity-Driven Optimization (ICML 2024 · similarity 0.88)