Research map / Vision-language and multimodal / Multimodal LLMs
Seeing as Experts Do: A Knowledge-Augmented Agent for Open-Set Fine-Grained Visual Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes KFRA, a tool-augmented agent that coordinates open-vocabulary detection, web retrieval, knowledge-guided region localisation and LMM reasoning for open-set fine-grained understanding, and releases the FGExpertBench benchmark.
Problem
Fine-grained visual understanding is limited by closed-set taxonomies and single-label prediction, degrading under open-set or context-dependent conditions that require justified, knowledge-grounded reasoning.
Approach
A closed three-stage loop (candidate generation via detection and web retrieval, knowledge-guided global-to-local region localisation with super-resolution refinement, and evidence-conditioned LMM inference) that couples retrieved textual knowledge with spatial grounding, without task-specific retraining.
Key results
- KFRA built on GLM-4.5V reaches 74.81% average accuracy on FGExpertBench, outperforming Gemini-2.5-Flash by 4.83 points.
- KFRA with Qwen2.5-VL-7B improves over its standalone base by 19.14 points on FGExpertBench (48.64% to 67.78%).
- KFRA (GLM-4.5V-12B) reaches 90.24% average on six FGIC datasets, versus 89.10% for Gemini-2.5-Flash and 83.21% for DeepPerception-7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add KFRA agent tooling (retrieval, detection, grounding, SR) on top of Qwen2.5-VL-7B | Qwen2.5-VL-7B · FGExpertBench | average accuracy (%): +19.14 | Helped |
| remove knowledge reference (KR) tool from full KFRA (ablation) | KFRA (Qwen2.5-VL-7B) · FGExpertBench | average accuracy (%): -4.36 (67.78 to 63.42 is removing SR; KR removal gives 50.21 vs 67.78 is -17.57) | Hurt |
| remove super-resolution (SR) from full KFRA (ablation) | KFRA (Qwen2.5-VL-7B) · FGExpertBench | average accuracy (%): 63.42 vs 67.78 (-4.36) | Hurt |
| remove visual search (VS) from full KFRA (ablation) | KFRA (Qwen2.5-VL-7B) · FGExpertBench | average accuracy (%): 52.83 vs 67.78 (-14.95) | Hurt |
| remove open-vocabulary detection (OD) from full KFRA (ablation) | KFRA (Qwen2.5-VL-7B) · FGExpertBench | average accuracy (%): 57.15 vs 67.78 (-10.63) | Hurt |
| remove global-to-local focusing (GF) from full KFRA (ablation) | KFRA (Qwen2.5-VL-7B) · FGExpertBench | average accuracy (%): 50.21 vs 67.78 (-17.57) | Hurt |
| enable only perception tools (VS, OD, GF, SR) without knowledge reference (ablation) | Qwen2.5-VL-7B · FGExpertBench | average accuracy (%): +0.92 | No clear effect |
| add KFRA agent on GLM-4.5V-12B backbone | GLM-4.5V-12B · FGExpertBench | average accuracy (%): 74.81 vs 65.41 (+9.40) | Helped |
| add KFRA agent on Qwen3-VL-8B backbone | Qwen3-VL-8B · FGExpertBench | average accuracy (%): 71.85 vs 59.54 (+12.31) | Helped |
| add KFRA agent on Qwen2.5-VL-7B backbone for FGIC datasets | Qwen2.5-VL-7B · FGIC avg (6 datasets) | accuracy (%): +23.61 | Helped |
| add KFRA agent on GLM-4.5V-12B backbone for FGIC datasets | GLM-4.5V-12B · FGIC avg (6 datasets) | accuracy (%): 90.24 vs 87.92 (+2.32) | Helped |
Surprising findings
- Knowledge reference (KR) contributes the largest single gain in the ablation, while the perception-only tools yield only a small improvement (48.64% to 49.56%).
Limitations
- Relies on external APIs for retrieval and knowledge access, which may introduce latency or retrieval errors.
- Multi-step design increases computational cost relative to single-pass models.
- FGExpertBench has limited domain coverage.
Experimental setup
Datasets: CUB-200-2011 FGVC-Aircraft Stanford Dogs Stanford Cars Oxford 102 Flowers Oxford-IIIT Pets
Benchmarks: FGExpertBench CUB-200-2011 Stanford Cars Stanford Dogs Oxford 102 Flowers FGVC-Aircraft Oxford-IIIT Pets
Metrics: accuracy
Baselines: VSA MMSearch GPT-4o Gemini-2.5-Flash Doubao-1.5-Vision-Pro Qwen2.5-VL-7B Qwen3-VL-8B GLM-4.5V Finedefics-8B DeepPerception-7B
Builds on: Grounding-DINO VisionReasoner OseDiff Qwen3-VL
Closest papers by meaning
- Diversity Over Frequency: Rethinking Tool Use in Visual Chain-of-Thought Agents (ICML 2026 · similarity 0.89)
- Look on Demand: A Cognitive Scheduling Framework for Visual Evidence Acquisition in Multimodal Reasoning (ICML 2026 · similarity 0.88)
- Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models (CVPR 2025 · similarity 0.88)
- The Perceptual Bandwidth Bottleneck in Vision-Language Models: Active Visual Reasoning via Sequential Experimental Design (ICML 2026 · similarity 0.88)
- Multi-step Visual Reasoning with Visual Tokens Scaling and Verification (NeurIPS 2025 · similarity 0.88)
- VisuRiddles: Fine-grained Perception is a Primary Bottleneck for Multimodal Large Language Models in Abstract Visual Reasoning (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.