Research map / Large language models / AI and society

MiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcome

NeurIPS 2026 · Large language models › AI and society · approach: Data commons and human baselines · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MiroEval, a live-refreshable benchmark of 100 real-user and trend-generated deep research tasks (70 text-only, 30 multimodal) with adaptive synthesis rubrics, agentic factuality verification, and process-centric evaluation, applied to 13 deep research systems.

Problem

Existing deep research benchmarks grade only final reports with fixed rubrics, ignore the research process, offer limited multimodal coverage, use synthetic tasks, and become stale over time.

Approach

Combines a dual-path (user-derived and trend-grounded) refreshable query construction with three evaluation layers: task-adaptive rubric synthesis, agentic factuality checking over web and attachments with a four-way label including CONFLICT, and process audits measuring intrinsic quality and report-process alignment.

Key results

• MiroThinker-H1 ranks highest overall with 77.5 on text-only and 74.5 on multimodal tasks, the smallest multimodal drop (3.0 points).
• Process score correlates with combined outcome score at Pearson 0.88 across systems.
• Human ranking of 10 systems on 5 queries agrees with MiroEval ranking (Kendall's tau = 0.91), with top three matching exactly.

Surprising findings

• Report-to-process traceability is much lower than findings-to-report coverage across systems (e.g., 63.3 vs 87.0 for MiroThinker-H1), indicating reports contain content not traceable to the research process.
• Factuality ratios are nearly unchanged between text-only and multimodal settings (drop of only 0.2 points on average) despite large synthesis drops.

Limitations

• Process evaluation requires systems to expose intermediate reasoning traces, excluding fully closed systems.
• The CONFLICT label flags cross-source disagreements without resolving which source is correct.
• Human study is small (5 annotators, 5 queries) and the human verification uses three annotators.

Experimental setup

Datasets: internal MiroMind user testing queries web trend headlines via Serper API

Benchmarks: MiroEval

Metrics: synthesis score factuality ratio (RIGHT rate) process score overall score Kendall's tau Spearman's rho Fleiss' kappa precision

Baselines: OpenAI Deep Research Gemini-3.1-Pro Deep Research Grok Deep Research Claude-Opus-4.6 Research Manus-1.6-Max Wide Research Doubao Deep Research ChatGLM Agent Kimi-K2.5 Deep Research Qwen-3.5-Plus Deep Research MiniMax-M2.5 Research

Builds on: MiroFlow DeepResearchEval

Closest papers by meaning

• AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite (ICLR 2026 · similarity 0.89)
• From Intent to Evidence: A Categorical Approach for Structural Evaluation of Deep Research Agents (NeurIPS 2026 · similarity 0.88)
• SciResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery (NeurIPS 2026 · similarity 0.88)
• InnovatorBench: Evaluating Agents’ Ability to Conduct Innovative AI Research (ICLR 2026 · similarity 0.88)
• MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research (NeurIPS 2025 · similarity 0.87)
• WorldMemArena: Evaluating Multimodal Agent Memory Through Action–World Interaction (NeurIPS 2026 · similarity 0.87)