Research map / Large language models / Agents, code and math
A Benchmark for Deep Information Synthesis
Debjit Paul, Daniel Murphy, Milan Gritta, Ronald Cardenas, Victor Prokhorov, Lena Sophia Bolliger, Aysim Toker, Roy Miles, Andreea-Maria Oncescu, Jasivan Alex Sivakumar, Philipp Borchert, Ismail Elezi and 5 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add search toolkit to OWL (ablation) | OWL (GPT-4.1) · DeepSynth | F1: 3.60 -> 5.41 | Improved |
| add web browsing toolkit to OWL (ablation) | OWL (GPT-4.1) · DeepSynth | F1: 4.80 -> 5.41 | Improved |
| add document processing toolkit to OWL (ablation) | OWL (GPT-4.1) · DeepSynth | F1: 4.90 -> 5.41 | Improved |
| add code execution toolkit to OWL (ablation) | OWL (GPT-4.1) · DeepSynth | F1: 4.82 -> 5.41 | Improved |
| provide gold intermediate reasoning steps (ablation) | GPT-4.1 · DeepSynth | F1: 3.46 -> 9.36 | Improved |
| provide gold intermediate reasoning steps (ablation) | Smolagent (GPT-4.1) · DeepSynth | F1: 3.75 -> 10.50 | Improved |
| provide gold intermediate reasoning steps (ablation) | o3 · DeepSynth | F1: 3.29 -> 12.87 | Improved |
| add web search and code interpreter (deep research agent) to o3 | o3 · DeepSynth | F1: 3.29 -> 8.97 | Improved |
| wrap GPT-4.1 in smolagents framework | GPT-4.1 · DeepSynth | F1: 3.46 -> 3.75 | Improved |
| use reasoning models instead of general-purpose LLMs | LLMs · DeepSynth | F1 | No clear effect |
| majority voting (N=5) instead of Best@5 | Smolagents (GPT-4.1) · DeepSynth-Dev | LLM judge accuracy: 27.5 -> 5 | Hurt |
Closest papers by meaning
- From Intent to Evidence: A Categorical Approach for Structural Evaluation of Deep Research Agents (NeurIPS 2026 · similarity 0.87)
- MiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcome (NeurIPS 2026 · similarity 0.87)
- Fathom-DeepResearch: Unlocking Long Horizon Information Retrieval and Synthesis for SLMs (ICLR 2026 · similarity 0.87)
- \$OneMillion-Bench: How Far are Language Agents from Human Experts? (NeurIPS 2026 · similarity 0.86)
- WideSearch: Benchmarking Agentic Broad Info-Seeking (ICLR 2026 · similarity 0.85)
- Can AI Agents Synthesize Scientific Conclusions? (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
