Research map / Large language models / Reasoning and chain of thought

Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for Open-Ended LLM Reasoning

ICML 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RuscaRL, a GRPO-based RL framework for open-ended reasoning that uses rubric criteria as decaying in-prompt scaffolding during rollouts (intra-group differentiation, sigmoid decay) and as LLM-judged rubric rewards for training.

Problem

RLVR is hard to apply to open-ended tasks without verifiable answers, and exploration is bounded by the base model's capabilities, so policies collapse onto a narrow set of reasoning trajectories.

Approach

Inject subsets of checklist-style rubric criteria into the prompt during rollout generation, with per-sample scaffolding levels varying within each GRPO group and a sigmoid schedule gradually removing scaffolding over training; training log-probs are computed without the scaffold, while rewards are normalized rubric scores graded by an LLM judge.

Key results

• On HealthBench-500, Qwen3-30B-A3B-Instruct rises from 46.9 to 61.1 with RuscaRL, above OpenAI-o3's 59.8.
• With Qwen2.5-7B-Instruct, RuscaRL reaches 56.4 on HealthBench-500 versus 52.0 for rubric-based RL and 36.8 for MeRF.
• Llama-3.1-8B base improves from 0.0 to 25.8 on HealthBench-500 with RuscaRL.

Surprising findings

• Constant full scaffolding performs much worse than decayed scaffolding, suggesting persistent in-prompt guidance is not the source of gains.
• Importance-ratio analysis uses a training log-prob that omits the scaffold, and this choice outperformed the theoretically correct importance-sampling ratio in experiments.

Limitations

• Medical-domain results dominate the main analysis, which uses Qwen2.5-7B-Instruct and HealthBench only.
• Closed-ended benchmarks (MedQA, MedMCQA) show marginal or negative gains.
• Relies on a GPT-4.1-generated rubric dataset and LLM-judge grading, so rubric quality and judge bias are not controlled.

Experimental setup

Datasets: HealthBench LongWriter-6k LongWriter-Zero-RLData IF-multi-constraints-upto5 SCP-116K MATH

Benchmarks: HealthBench-500 LLMEval-Med MedQA MedMCQA WritingBench Creative Writing v3 IFEVAL IFBench

Metrics: accuracy best-of-N novelty (importance ratio) self-BLEU semantic distance

Baselines: Rubric-based RL MeRF RL-Plus EA-RL SFT SFT + Rubric-based RL

Builds on: GRPO MeRF RL-Plus

Closest papers by meaning

• Step-wise Rubric Rewards for LLM Reasoning (NeurIPS 2026 · similarity 0.91)
• RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs (ICLR 2026 · similarity 0.89)
• DeepSearch: Overcome the Bottleneck of Reinforcement Learning with Verifiable Rewards via Monte Carlo Tree Search (ICLR 2026 · similarity 0.89)
• Rectifying LLM Thought from Lens of Optimization (ICLR 2026 · similarity 0.89)
• Beyond Accuracy: Dissecting Mathematical Reasoning for LLMs Under Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
• Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration (ICML 2026 · similarity 0.89)