Research map / Vision-language and multimodal / Multimodal LLMs
Visual Agents as Fast and Slow Thinkers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces FaST, a visual agent that switches between fast (System 1) and slow (System 2) thinking via a switch adapter, building a chain of evidence with proposal and segmentation adapters on LLaVA-v1.5.
Problem
Visual agents built on MLLMs give overconfident, inaccurate answers on complex or real-world queries involving uncertain or tiny objects, lacking explicit fast/slow cognitive modeling.
Approach
A switch adapter decides per query whether to answer directly (System 1) or invoke a slow pipeline that identifies missing objects and context clues, proposes regions with a proposal adapter, and refines them into masks with a SAM-based seg adapter, forming a 'chain of evidence' before the final answer.
Key results
- FaST reaches 80.8% on VQA-v2 vs 78.5% for LLaVA-v1.5 and achieves 1517.4 MME perception score.
- On ReasonSeg, FaST gets 47.6 CIoU and 48.7 GIoU, exceeding LISA-7B's 44.4 CIoU and 46.0 GIoU.
- FaST runs 31% faster than System 2-only on MME (2023ms vs 2938ms) with comparable accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use FaST instead of LLaVA-v1.5 | visual question answering · VQA-v2 | accuracy: +2.3 | Helped |
| use FaST instead of LLaVA-v1.5 | visual question answering · GQA | accuracy: +1.8 | Helped |
| use FaST instead of LLaVA-v1.5 | visual question answering · TextVQA | accuracy: + 2.5 | Mixed |
| use FaST instead of LLaVA-v1.5 | multimodal evaluation · MME | total score: +6.7 | Helped |
| use FaST instead of LLaVA-v1.5 | multimodal evaluation · SEED | score: + 1.5 | Helped |
| use FaST instead of LLaVA-v1.5 | multimodal evaluation · MM-Vet | score: + 0.5 | Helped |
| use FaST instead of LISA-7B | reasoning segmentation · ReasonSeg | CIoU: 3.2% CIoU | Helped |
| use FaST instead of LISA-7B | reasoning segmentation · ReasonSeg | GIoU: 2.7% GIoU | Helped |
| add proposal adapter (ablation) | FaST without proposal or seg adapters · GQA | accuracy: 62.1 -> 63.2 | Helped |
| add seg adapter (ablation) | FaST without proposal or seg adapters · MME | total score: 1509.2 -> 1514.4 | Helped |
| add proposal and seg adapters (ablation) | FaST without proposal or seg adapters · POPE | F1 score: 85.7 -> 86.2 | Helped |
| add missing objects (ablation) | FaST switch adapter · MME | total score: 1511.8 -> 1513.4 | Helped |
Surprising findings
- System 1-only configuration achieves 1508.7 MME vs 1518.6 for System 2-only, so the slow mode yields little gain on MME.
Limitations
- Reliance on a predefined negative dataset may not generalize to novel real-world scenarios.
- Pixel-level mask decoder struggles with highly textured or patterned images and small or occluded objects.
- Generalizability across domains needs further validation.
Experimental setup
Datasets: V* PixelLM LLaVA-v1.5 665k instruction tuning GQA VAW LLaVA-80K LAION-CC-SBU 558K ADE20K COCO-Stuff LVIS-PACO refCOCO refCOCO+
Benchmarks: VQA-v2 GQA TextVQA ScienceQA-IMG POPE MME SEED MM-Vet refCOCO refCOCO+ refCOCOg ReasonSeg
Metrics: accuracy F1 score MME total score CIoU GIoU GPT-4 score
Baselines: LLaVA-v1.5 BLIP-2 InstructBLIP Qwen-VL-Chat mPLUG-Owl2 Monkey Chain of Spot V* Visual CoT LISA-7B LAVT OVSeg
Builds on: LLaVA-v1.5 LISA Chain of Spot V* SAM
Closest papers by meaning
- Visual Graph Arena: Evaluating Visual Conceptualization of Vision and Multimodal Large Language Models (ICML 2025 · similarity 0.87)
- MATA: A Trainable Hierarchical Automaton System for Multi-Agent Visual Reasoning (ICLR 2026 · similarity 0.87)
- CodeV: Code with Images for Faithful Visual Reasoning via Tool-Aware Policy Optimization (CVPR 2026 · similarity 0.87)
- SPARC: Separating Perception And Reasoning Circuits for Test-time Scaling of VLMs (ICML 2026 · similarity 0.87)
- Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models (CVPR 2025 · similarity 0.87)
- DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learning (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.