Research map / Large language models / Agents, code and math
DEER: A Benchmark for Evaluating Deep Research Agents on Expert Report Generation
Janghoon Han, Heegyu Kim, Changho Lee, Dahm Lee, Min Hyung Park, Hosung Song, Stanley Jungkyu Choi, Moontae Lee, Honglak Lee
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add reasoning (think) to general LLMs | deep research benchmark systems · DEER overall mean | mean score: 4.88 -> 5.89 (GPT-5.2 fast -> think) | Improved |
| add web search to reasoning LLMs | GPT-5.2 · DEER overall mean | mean score: 5.89 -> 6.62 | Improved |
| add web search or deep research orchestration to reasoning models | report-quality dimensions excluding information scores · DEER | report-quality scores | Hurt |
| add dimensions to the vanilla LLM judge (ablation) | LLM-as-judge · human correlation | Pearson r: 0.63 -> 0.66 | No clear effect |
| add 101 granular rubric items (ablation) | LLM-as-judge with dimensions · human correlation | Pearson r: 0.66 -> 0.64 | Hurt |
| add task-specific expert guidance (ablation) | LLM-as-judge with granular rubrics · human correlation | Pearson r: 0.64 -> 0.73 | Improved |
| add expert guidance (ablation) | LLM-as-judge · inter-evaluator reliability | Krippendorff alpha: 0.33 -> 0.55 | Improved |
| add dimensions to vanilla judge (ablation) | LLM-as-judge · inter-evaluator reliability | Krippendorff alpha: 0.46 -> 0.32 | Hurt |
| use 10-point scale instead of 5-point scale (ablation) | report quality judge · human correlation | Pearson r: 0.694 -> 0.734 | Improved |
| add 5-band score anchors (ablation) | report quality judge · human correlation | Pearson r: 0.722 -> 0.734 | Improved |
| semantic back-tracking vs sliding window for citation recovery (ablation) | information verification · B/C claims | Jaccard: 0.6822 -> 0.7070 | Improved |
| grouped verification by evidence URL (ablation) | claim verification (GPT-5-mini) · claim verification | F1: 87.25 -> 90.91; cost 12.84 -> 3.65 | Improved |
Later papers that compared with this method
15 later accepted papers report DEER as a baseline in their results tables; 14 of them report a better number on most of the results they share with it.
- LYNX: Learning Dynamic Exits for Confidence-Controlled Reasoning (NeurIPS 2026 · better on 8 of 10 shared results)
- Unveiling the Entropy Dynamics of Chain-of-Thought Reasoning (ICML 2026 · better on 9 of 9 shared results)
- Statistical Early Stopping for Reasoning Models (ICML 2026 · better on 7 of 8 shared results)
- Mitigating Overthinking in Large Reasoning Language Models via Reasoning Path Deviation Monitoring (NeurIPS 2026 · better on 8 of 8 shared results)
- Test-time Prompt Intervention (AAAI 2026 · better on 8 of 8 shared results)
- Answering the Unanswerable Is to Err Knowingly: Analyzing and Mitigating Abstention Failures in Large Reasoning Models (AAAI 2026 · better on 5 of 5 shared results)
- Intervene When It Doubts: Conjunction-Guided Interactive Reasoning (ICML 2026 · better on 2 of 2 shared results)
- DiffAdapt: Difficulty-Adaptive Reasoning for Token-Efficient LLM Inference (ICLR 2026 · better on 2 of 2 shared results)
7 more not listed.
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- From Intent to Evidence: A Categorical Approach for Structural Evaluation of Deep Research Agents (NeurIPS 2026 · similarity 0.88)
- DeepTRACE: Auditing Deep Research AI Systems for Tracking Reliability Across Citations and Evidence (ICLR 2026 · similarity 0.87)
- MiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcome (NeurIPS 2026 · similarity 0.87)
- WebWeaver: Structuring Web-Scale Evidence with Dynamic Outlines for Open-Ended Deep Research (ICLR 2026 · similarity 0.87)
- Deep Research Arena: The First Exam of LLMs’ Research Abilities via Seminar-Grounded Tasks (AAAI 2026 · similarity 0.86)
- CycleResearcher: Improving Automated Research via Automated Review (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
