Research map / Vision-language and multimodal / Multimodal LLMs
Argus: Vision-Centric Reasoning with Grounded Chain-of-Thought
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Argus adds explicit language-guided RoI grounding as visual chain-of-thought to an 8B MLLM (Eagle-style MoVE encoders with Llama3-8B), re-engaging the grounded regions via re-encoding or re-sampling of visual tokens.
Problem
Existing MLLMs underperform on vision-centric reasoning because language-directed visual attention is only implicit through LLM self-attention, leaving precise focus on task-relevant regions unaddressed.
Approach
Predict text-format bounding boxes for the query-relevant region as visual CoT, then explicitly re-engage those regions either by re-encoding crops through the vision experts or by re-sampling cached patch tokens overlapping the box, and compare four attention-guidance strategies.
Key results
- Argus-X3-8B reaches 65.3 average on vision-centric tasks, with V-Star 78.5 and CV-Bench 2D 68.5, versus 59.6 for Eagle-X3-8B.
- On RefCOCO val, Argus-X3-8B scores 89.8 and RefCOCO+ val 84.7, leading generalist MLLMs of comparable scale.
- Re-sampling reaches 26 extra visual tokens and 492 ms inference versus 1024 tokens and 827 ms for re-encoding.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add bounding-box CoT guidance (ablation) | Eagle-X3 · V-Star | score: 58.6 -> 63.9 | Helped |
| Explicitly re-encode RoI crops (ablation) | Eagle-X3 with box guidance · V-Star | score: 63.9 -> 68.1 | Helped |
| Explicitly re-sample cached RoI patch tokens (ablation) | Eagle-X3 with box guidance · TextVQA | score: 71.6 -> 73.9 | Helped |
| Use re-encoding instead of re-sampling (ablation) | Argus explicit RoI engagement · V-Star | score: 67.0 -> 68.1 | Helped |
| Replace MoVE with fewer vision encoders (ablation) | Argus re-encoding · V-Star | score: --4.2 | Hurt |
| Replace MoVE with fewer vision encoders (ablation) | Argus re-sampling · V-Star | score: --11.5 | Hurt |
| Expand RoI to square context (ablation) | Argus re-encoding · V-Star | score: +1.6 | Helped |
| Expand RoI to square context (ablation) | Argus re-sampling · TextVQA | score: --3.1 | Hurt |
| Use separate MLP layers for RoI re-engagement (ablation) | Argus re-sampling · CV-Bench 2D | score: +1.7 | Helped |
| Add CoT signals and the re-engagement mechanism during SFT (ablation) | Eagle-X3 · TextVQA | score: 66.3 -> 71.1 | Helped |
| Add grounding datasets during SFT (ablation) | Eagle-X3 with CoT signals · TextVQA | score: 71.1 -> 73.9 | Helped |
| Expand predicted RoI by 20% (ablation) | Argus re-encoding · ChartQA | score: 73.1 -> 73.3 | Helped |
Surprising findings
- Re-encoding outperforms re-sampling on V-Star (68.1 vs 67.0), the one benchmark where the usual re-sampling advantage reverses.
- Argus is reported to surpass some proprietary MLLMs on the vision-centric and text benchmark averages.
Limitations
- Evaluated only with an 8B Llama3 backbone, so scaling findings are untested.
- Visual CoT training data is limited in diversity and scale.
- Multi-RoI extension is a simple multi-step prompting scheme and open-world detection is not covered.
Experimental setup
Datasets: LLaVA-595K Eagle-1.8M VCoT GRIT Shikra RefCOCO RefCOCO+ RefCOCOg V-Star CV-Bench MMVP RealworldQA
Benchmarks: V-Star CV-Bench MMVP RealworldQA ChartQA OCRBench TextVQA DocVQA MMMU MMBench SEED-Bench GQA
Metrics: accuracy Acc@0.5 GMACs inference time
Baselines: Eagle-X3-8B Visual-CoT-7B Mini-Gemini-HD-8B LLaVA-NeXT-8B InternVL QwenVL-7B Grounding-DINO-L Shikra-7B Ferret-7B MiniGPT-v2
Builds on: Eagle Visual CoT LLaVA Shikra
Closest papers by meaning
- VGR: Visual Grounded Reasoning (ICLR 2026 · similarity 0.91)
- VisuRiddles: Fine-grained Perception is a Primary Bottleneck for Multimodal Large Language Models in Abstract Visual Reasoning (ICLR 2026 · similarity 0.91)
- Look-Back: Implicit Visual Re-focusing in MLLM Reasoning (AAAI 2026 · similarity 0.90)
- AStar: Boosting Multimodal Reasoning with Automated Structured Thinking (AAAI 2026 · similarity 0.90)
- Grounded Chain-of-Thought for Multimodal Large Language Models (CVPR 2026 · similarity 0.90)
- MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.