Research map / Vision-language and multimodal / Multimodal LLMs
Visual Chain-of-Thought Prompting for Knowledge-Based Visual Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Visual Chain-of-Thought Prompting (VCTP), a training-free see-think-confirm loop that alternates visual concept grounding, captioning and LLM prompting with CLIP-verified rationales for knowledge-based visual QA.
Problem
Knowledge-based visual reasoning requires interleaving visual perception with open-world knowledge and step-by-step reasoning, but prior few-shot methods treat vision and language as independent modules and produce black-box answers.
Approach
An iterative see-think-confirm framework: an LLM attends to detected visual concepts, a captioner converts them to text for answer prediction, and the LLM generates rationales that are verified by a CLIP cross-modality classifier and fed back into the prompt until answers stabilize.
Key results
- VCTP reaches 46.0 accuracy on the A-OKVQA test split, versus 43.8 for PICa and 43.7 for CoT with OPT-66B.
- With Llama-2-70B, the method reaches 54.4 on A-OKVQA test and 54.9 on OK-VQA; with BLIP2 and Codex it reaches 56.2 on OK-VQA.
- Under cost-aligned ensembling (k=14), PICa and CoT remain below VCTP (42.84/38.15 vs 46.41/44.62 on A-OKVQA val / OK-VQA).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VCTP instead of PICa | OPT-66B visual question answering · A-OKVQA val | accuracy: 42.4 -> 46.4 | Helped |
| Use VCTP instead of PICa | OPT-66B visual question answering · A-OKVQA test | accuracy: 43.8 -> 46.0 | Helped |
| Use VCTP instead of PICa | OPT-66B visual question answering · OK-VQA test | accuracy: 42.9 -> 44.6 | Helped |
| Use Llama-2-70B instead of OPT-66B as the language model | VCTP · A-OKVQA test | accuracy: 46.0 -> 54.4 | Helped |
| Remove attend-describe components (ablation) | VCTP · A-OKVQA val | accuracy: 46.41 -> 43.48 | Hurt |
| Add rationale generation (ablation) | VCTP · A-OKVQA val | accuracy: 45.54 -> 46.41 | Helped |
| Add CLIP verification (ablation) | VCTP · A-OKVQA val | accuracy: 45.82 -> 46.41 | Helped |
| Attend to a random region (ablation) | VCTP · A-OKVQA val | accuracy: 46.41 -> 43.21 | Hurt |
| Replace the global caption with a random training-set caption (ablation) | VCTP · A-OKVQA val | accuracy: 46.41 -> 38.98 | Hurt |
| Select a random rationale from the training set (ablation) | VCTP · A-OKVQA val | accuracy: 46.41 -> 44.53 | Hurt |
| Set the maximum number of iterations to 1 (ablation) | VCTP · A-OKVQA val | accuracy: 44.68 -> 46.41 | Hurt |
| Set the maximum number of iterations to 4 (ablation) | VCTP · A-OKVQA val | accuracy: 46.41 -> 46.42 | No clear effect |
Surprising findings
- Removing the CLIP verification module gave a slightly higher accuracy on A-OKVQA val (45.82) than full model but lower concept recall, which the authors attribute to rejecting misleading rationales.
Limitations
- Relies on multiple pretrained components (detector, captioner, LLM, CLIP) and multiple LLM queries per sample (mean about 13.6).
- Evaluated only on OK-VQA and A-OKVQA; confirm module removed for OK-VQA since no rationales exist.
Experimental setup
Datasets: A-OKVQA OK-VQA Visual Genome
Benchmarks: A-OKVQA OK-VQA
Metrics: accuracy BLEU CLIP sentence similarity recall of key concepts
Baselines: PICa CoT GPV-2 KAT-GPT-3 BLIP2 KRISP LXMERT ViLBERT
Builds on: chain-of-thought prompting PICa BLIP CLIP neural module networks
Closest papers by meaning
- VGR: Visual Grounded Reasoning (ICLR 2026 · similarity 0.91)
- Interleaved-Modal Chain-of-Thought (CVPR 2025 · similarity 0.91)
- TVI-CoT: Text-Visual Interleaved Chain-of-Thought Reasoning for Multimodal Understanding (ICML 2026 · similarity 0.90)
- Multi-step Visual Reasoning with Visual Tokens Scaling and Verification (NeurIPS 2025 · similarity 0.90)
- SLVR: Structured Latent Visual Reasoning via Human-like Reasoning Flows (NeurIPS 2026 · similarity 0.90)
- VLM-R³: Region Recognition, Reasoning, and Refinement for Enhanced Multimodal Chain-of-Thought (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.