Research map / Vision-language and multimodal / Multimodal LLMs
Visual Document Understanding and Reasoning: A Multi-Agent Collaboration Framework with Agent-Wise Adaptive Test-Time Scaling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MACT, a four-agent (planning, execution, judgment, answer) framework with agent-wise adaptive test-time scaling and mixed process/outcome rewards for visual document understanding, built on Qwen2.5-VL, MiMo-VL, and InternVL3 bases.
Problem
Monolithic scaling of VLMs yields diminishing returns on document understanding and reasoning, which needs procedural reasoning, specialized skills, and error self-correction.
Approach
Decompose document VQA into four specialized agents with a separate judgment agent that routes errors back to planning or execution for correction, and allocate test-time compute per agent (parallel plans, step-wise candidate execution scored by a PRM, budget forcing for judgment) with a mixed agent-specific and global reward for RL.
Key results
- MACT-MiMo-VL-Series-28B reaches 75.3 average versus the base MiMo-VL-7B-SFT gain of 9.9%; MACT variants average gains of 10.3%, 9.9%, and 11.5% over base models.
- MACT-MiMo-VL-Series-28B reports 5.6% average gain over the best open-source model and 5.9% over the best closed-source model (per the paper's table, the 28B variant listed is 77.2 overall vs 74.8 for Qwen-24B in the same table, so the reported averages differ by variant).
- Ablation: judgment agent addition raises average by 5.5%; agent-wise adaptive scaling beats the best fixed scaling baseline (internal scaling) by at least 1.8% on average.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace multi-agent system with a monolithic model (with scaling and reward retained) (ablation) | Qwen2.5-VL-Series-24B · MMLong / TabBen / MVision ablation | Avg.: 74.8 -> 66.2 | Hurt |
| add multi-agent collaboration (four agents vs single unified agent) (ablation) | Qwen2.5-VL-Series-24B · MMLong / TabBen / MVision ablation | Avg.: 58.6 -> 74.8 | Helped |
| add agent-wise adaptive test-time scaling (ablation) | Qwen2.5-VL-Series-24B · MMLong / TabBen / MVision ablation | Avg.: 71.1 -> 74.8 | Helped |
| add mixed reward modeling (ablation) | Qwen2.5-VL-Series-24B · MMLong / TabBen / MVision ablation | Avg.: 71.4 -> 74.8 | Helped |
| add judgment agent to planning+execution pipeline (ablation) | multi-agent pipeline · MMLong / TabBen / MVision ablation | Avg.: 68.4 -> 73.9 | Helped |
| add answer agent to planning+execution pipeline (without judgment agent) (ablation) | multi-agent pipeline · MMLong / TabBen / MVision ablation | Avg.: 68.4 -> 68.8 | Helped |
| agent-wise adaptive scaling vs parallel scaling (ablation) | test-time scaling · MMLong / TabBen / MVision ablation | Avg.: 72.0 -> 74.8 | Helped |
| agent-wise adaptive scaling vs sequential scaling (ablation) | test-time scaling · MMLong / TabBen / MVision ablation | Avg.: 72.4 -> 74.8 | Helped |
| agent-wise adaptive scaling vs hybrid scaling (ablation) | test-time scaling · MMLong / TabBen / MVision ablation | Avg.: 73.0 -> 74.8 | Helped |
| agent-wise adaptive scaling vs internal budget-forcing scaling (ablation) | test-time scaling · MMLong / TabBen / MVision ablation | Avg.: 72.3 -> 74.8 | Helped |
| use agent-specific rewards only instead of mixed reward (ablation) | reward modeling · MMLong / TabBen / MVision ablation | Avg.: 72.7 -> 74.8 | Helped |
| use global reward only instead of mixed reward (ablation) | reward modeling · MMLong / TabBen / MVision ablation | Avg.: 74.8 -> 70.2 | Hurt |
Surprising findings
- Integrating all four agent functions into a single model (w/o multi-agent collaboration) performs worse than the base model, which the authors attribute to monolithic limitations.
- Global reward alone yields lower performance than no reward on the ablation benchmarks (70.2 vs 71.4 average).
Limitations
- Multi-agent pipeline with multiple correction loops and N_p/N_e sampling increases inference cost, which is not quantified in the main text.
- Judgment labels rely on GPT-4o and rule-based verification, and reward models (VisualPRM, Skywork-VL-Reward) are external dependencies.
- Results on some benchmarks (e.g., CharXiv for MiMo variant, MMLongBench-Doc for stronger closed models) are not uniformly best; MMLongBench-Doc remains low at 43-47.
Experimental setup
Datasets: DocVQA DUDE SlideVQA MMLongBench-Doc VisualMRC InfographicVQA ChartQA CharXiv TableVQA-Bench TableBench ScienceQA RealWorldQA
Benchmarks: DocVQA DUDE SlideVQA MMLongBench-Doc VisualMRC InfographicVQA ChartQA CharXiv TableVQA-Bench TableBench ScienceQA RealWorldQA
Metrics: accuracy (GPT-4o judged) ANLS F1 average accuracy
Baselines: GPT-4o-latest Claude-3.7-Sonnet Gemini-2.0-Pro Qwen2.5-VL-72B-Instruct InternVL3-78B-Instruct MDocAgent M3DocRAG MiMo-VL-7B-SFT
Builds on: GRPO budget forcing analogical prompting VisualPRM Skywork-VL-Reward
Closest papers by meaning
- Resolving Evidence Sparsity: Agentic Context Engineering for Long-Document Understanding (CVPR 2026 · similarity 0.90)
- Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models (CVPR 2025 · similarity 0.88)
- PixelCraft: A Multi-Agent system for High-Fidelity Visual Reasoning on Structured Images (ICLR 2026 · similarity 0.88)
- ATLAS: Agentic or Latent Visual Reasoning? One Word is Enough for Both (NeurIPS 2026 · similarity 0.87)
- Agent-X: Evaluating Deep Multimodal Reasoning in Vision-Centric Agentic Tasks (ICLR 2026 · similarity 0.87)
- SLVR: Structured Latent Visual Reasoning via Human-like Reasoning Flows (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.