Research map / Vision-language and multimodal / Multimodal LLMs
Visual-O1: Understanding Ambiguous Instructions via Multi-modal Multi-turn Chain-of-thoughts Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Visual-O1, a multi-modal multi-turn chain-of-thought reasoning framework that builds instance-level (for high-intelligence models) or dataset-level empirical (for general models) disambiguation experience, and evaluates on ambiguous and general RIS and VQA splits.
Problem
Multi-modal models perform poorly on ambiguous language instructions that require visual context or common sense to resolve, and existing CoT or training-based fixes are costly or require strong reasoning ability.
Approach
Iterative reasoning-reflection loops over the ambiguous instruction and image produce a disambiguation experience; high-intelligence models use per-sample instantial experience at inference, while general models use empirical experience learned once from a few examples and then rewrite the instruction before answering.
Key results
- On ambiguous RIS, Visual-O1 (SoM) reaches 0.1722 cIoU versus 0.1158 for SoM, and Visual-O1 (LISA) reaches 0.1088 gIoU versus 0.0237 for LISA.
- On VizWiz VQA ambiguous split, Visual-O1 (GPT-4o) reaches 22.20 accuracy versus 14.10 for GPT-4o; Visual-O1 (LLaVa) reaches 21.35 versus 8.58.
- Visual-O1 (GPT-4o) improves general RIS gIoU to 0.5777 from 0.5728 reported for GPT-4o, and Visual-O1 (LLaVa) improves general VQA accuracy from 54.19 to 57.38 on the reported table.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Visual-O1 to LISA | LISA · RIS ambiguous instructions | gIoU: 0.0237 -> 0.1088 | Helped |
| Apply Visual-O1 to SoM | SoM · RIS ambiguous instructions | cIoU: 0.1158 -> 0.1722 | Helped |
| Apply Visual-O1 to LISA | LISA · RIS general instructions | gIoU: 0.4654 -> 0.4794 | Helped |
| Apply Visual-O1 to GPT-4o | GPT-4o · VQA ambiguous instructions | accuracy: 14.10 -> 22.20 | Helped |
| Apply Visual-O1 to LLaVA | LLaVA · VQA ambiguous instructions | accuracy: 8.58 -> 21.35 | Helped |
| Apply Visual-O1 to GPT-4o | GPT-4o · VQA general instructions | BLEU-1: 0.8192 -> 0.8339 | Helped |
| Apply Visual-O1 to LLaVA | LLaVA · VQA general instructions | accuracy: 54.19 -> 57.38 | Helped |
| Add response synthesis (ablation) | Visual-O1 (LLaVA) · VQA general instructions | accuracy: 47.59 -> 57.38 | Helped |
| Add response synthesis (ablation) | Visual-O1 (GPT-4o) · VQA general instructions | BLEU-1: 0.8254 -> 0.8339 | Helped |
| Add reasoning and reflection (ablation) | Visual-O1 (LLaVA) · VQA general instructions | accuracy: 54.25 -> 57.38 | Helped |
| Add reasoning and reflection (ablation) | Visual-O1 (GPT-4o) · VQA general instructions | BLEU-1: 0.7943 -> 0.8339 | Helped |
| Use multiple examples during optimization (ablation) | Visual-O1 (LLaVA) · VQA general instructions | accuracy: 55.75 -> 57.38 | Helped |
Surprising findings
- Traditional CoT and FuDD degrade performance on general VQA for LLaVA, suggesting complex reasoning impairs general-intelligence models.
- Manually designed disambiguation experience performs worse than automatically reasoned empirical experience.
Limitations
- One-time optimization still requires a high-intelligence model for reflection.
- Ambiguous evaluation subsets are small (150 RIS, 106 VQA) and manually curated.
- Subjectivity category has only 3.3% of the data.
Experimental setup
Datasets: RefCOCO+ VizWiz
Benchmarks: RefCOCO+ VizWiz Room-to-Room
Metrics: gIoU cIoU accuracy BLEU-1 SR SPL Navi Error
Baselines: LISA SoM Chain-of-thoughts FuDD LLaVA GPT-4o
Builds on: chain-of-thought prompting FuDD SoM
Closest papers by meaning
- Incentivizing Reasoning for Advanced Instruction-Following of Large Language Models (NeurIPS 2025 · similarity 0.88)
- CoMT: A Novel Benchmark for Chain of Multi-modal Thought on Large Vision-Language Models (AAAI 2025 · similarity 0.88)
- Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models (CVPR 2025 · similarity 0.88)
- Eliciting Reasoning in Language Models with Cognitive Tools (NeurIPS 2025 · similarity 0.88)
- When Visualizing is the First Step to Reasoning: MIRA, a Benchmark for Visual Chain-of-Thought (CVPR 2026 · similarity 0.88)
- ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoning (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.