Research map / Multi-agent systems and games / LLM agents

DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Reinforcement Learning with Evolving Rubrics (RLER), training an open 8B deep research agent (DR Tulu-8B, from Qwen3-8B) with search-grounded rubrics that co-evolve with the policy during GRPO, achieving strong long-form deep research results.

Problem

Open deep research agents are trained with RLVR on short-form QA, which does not extend to long-form, open-ended deep research where reliable reward signals are hard to define.

Approach

Rubrics are initialized from web-search-grounded documents and then continuously regenerated during RL by contrasting current policy rollouts (positive and negative rubrics), with a variance-based buffer that keeps the most discriminative rubrics, so the verifier co-evolves with the policy.

Key results

• DR Tulu-8B (RL) averages 65.6 across four long-form benchmarks, 15.6 points above the best open baseline Tongyi DR-30B (50.0) and 0.7 above OpenAI Deep Research (64.9).
• RLER adds 6.4-16.0 points over the SFT checkpoint across the four long-form benchmarks.
• DR Tulu-8B costs about $0.0019 per query on SQAv2 versus about $1.80 for OpenAI Deep Research.

Surprising findings

• Higher training reward did not necessarily correspond to higher downstream benchmark performance (train-test mismatch).
• A single general rubric shared across samples underperforms the SFT baseline when used as static reward.

Limitations

• Relies on GPT-4.1 and GPT-4.1-mini as judge and rubric generator; open Qwen3-8B judge underperforms by about 1.3 points.
• Train-test mismatch: training reward gains do not always transfer to external benchmark scores.
• GeneticDiseasesQA is small (47 questions) and evaluated on a limited set of baselines.

Experimental setup

Datasets: SearchArena OpenScholar RaR HotpotQA TaskCraft WebWalker-Silver MegaScience PopQA-inspired synthetic prompts HealthBench ResearchQA AstaBench-ScholarQA-CS2 DeepResearchBench

Benchmarks: AstaBench-ScholarQA-CS2 HealthBench ResearchQA DeepResearchBench GeneticDiseasesQA SimpleQA WebWalkerQA 2Wiki

Metrics: rubric score / average score citation precision citation recall Pass@1 accuracy

Baselines: Tongyi Deep Research-30B WebThinker-32B Search-R1-7B ASearcher-Web-7B WebExplorer-8B Ai2 ScholarQA OpenAI Deep Research Gemini 3 Pro + Search GPT-5 + Search Perplexity Deep Research Claude-Sonnet Search QwQ-32B

Builds on: GRPO Search-R1 Rubrics as Rewards (RaR) Open-Instruct Qwen3

Closest papers by meaning

• DeepSearch: Overcome the Bottleneck of Reinforcement Learning with Verifiable Rewards via Monte Carlo Tree Search (ICLR 2026 · similarity 0.86)
• Deep Research as Rubric (NeurIPS 2026 · similarity 0.86)
• Quest: Training Frontier Deep Research Agents with Fully Synthetic Tasks (NeurIPS 2026 · similarity 0.86)
• Reward Hacking in Rubric-Based Reinforcement Learning (NeurIPS 2026 · similarity 0.84)
• Search More, Think Less: Rethinking Long-Horizon Agentic Search for Efficiency and Generalization (NeurIPS 2026 · similarity 0.84)
• MAD-TD: Model-Augmented Data stabilizes High Update Ratio RL (ICLR 2025 · similarity 0.84)