Research map / Large language models / Agents, code and math
ResearchRubrics: A Benchmark of Prompts and Rubrics For Deep Research Agents
Manasi Sharma, Chen Bo Calvin Zhang, Chaithanya Bandi, Clinton Wang, Ankit Aich, Huy Nghiem, Tahseen Rabbani, Ye Htet, Brian Jang, Sumana Basu, Aishwarya Balwani, Denis Peskoff and 4 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use binary instead of ternary grading for LLM judges (ablation) | LLM-as-judge rubric grading · ResearchRubrics human agreement | Macro F1: ternary 0.567 -> binary 0.760 (Gemini DR, Gemini-2.5-Pro) | Improved |
| add concrete inline examples to rubric criteria (ablation) | rubric criteria · ResearchRubrics judge alignment | Macro F1 (binary): 0.733 -> 0.760 (Gemini DR) | Improved |
| LLM-based augmentation of rubric text (ablation) | rubric criteria · ResearchRubrics judge alignment | Macro F1 (binary): 0.760 -> 0.564 (Gemini DR) | Hurt |
| increase logical nesting depth of the task | deep research agents · ResearchRubrics | rubric compliance | Hurt |
| increase conceptual breadth of the task | deep research agents · ResearchRubrics | rubric compliance | Hurt |
| longer responses | Gemini DR and ChatGPT DR · ResearchRubrics | rubric compliance: r of about 0.24-0.28 | Improved |
| more citations (Gemini DR 111 vs Perplexity 31) | deep research agents · ResearchRubrics citation analysis | citation accuracy: 90% (31 citations) -> 81% (111 citations) | Hurt |
Closest papers by meaning
- From Intent to Evidence: A Categorical Approach for Structural Evaluation of Deep Research Agents (NeurIPS 2026 · similarity 0.90)
- MiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcome (NeurIPS 2026 · similarity 0.90)
- DRBench: A Realistic Benchmark for Enterprise Deep Research (ICLR 2026 · similarity 0.89)
- Fathom-DeepResearch: Unlocking Long Horizon Information Retrieval and Synthesis for SLMs (ICLR 2026 · similarity 0.89)
- Quest: Training Frontier Deep Research Agents with Fully Synthetic Tasks (NeurIPS 2026 · similarity 0.89)
- Deep Research Arena: The First Exam of LLMs’ Research Abilities via Seminar-Grounded Tasks (AAAI 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
