Research map / Vision-language and multimodal / Multimodal LLMs
CogCoM: A Visual Language Model with Chain-of-Manipulations Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Chain of Manipulations (CoM), where a VLM solves visual problems step-by-step by emitting grounding/OCR/zoom-in manipulations with intermediate visual results, trained on 70K automatically generated and 7K human-annotated CoM samples, yielding the 17B CogCoM.
Problem
Conclusive instruction-tuning of VLMs teaches them to answer directly, skipping intermediate visual reasoning, which causes failures on detail-heavy visual tasks, unfaithful answers, and hallucinations.
Approach
Define six atomic visual manipulations (OCR, grounding, counting, calculate, crop-and-zoom-in, line drawing) that the VLM emits as part of its reasoning chain with the intermediate image/box results, produced via a cascade of GPT-4 (linguistic steps) and GroundingDINO/PaddleOCR (visual results) with DFS path filtering, and a memory-based multi-turn multi-image architecture built on CogVLM.
Key results
- CogCoM reaches 71.1 on TextVQA and 70.0 on ST-VQA, versus 69.7 and 61.0 for CogVLM, with up to 9.0-point gains on detailed recognition benchmarks.
- On MM-Vet CogCoM scores 46.1 vs 45.5 for CogVLM, and on POPE adversarial 87.8 vs 87.2.
- Ablation removing the 70K CoM data drops TextVQA from 71.1 to 64.5 and MathVista from 35.7 to 34.8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| include 70K CoM training data (ablation) | CogCoM-chat · TextVQA | VQA Score: 64.5 -> 71.1 | Helped |
| include 70K CoM training data (ablation) | CogCoM-chat · MM-Vet | score: 45.9 -> 46.1 | Helped |
| include 70K CoM training data (ablation) | CogCoM-chat · MathVista | score: 34.8 -> 35.7 | Helped |
| add Chain of Manipulations reasoning and training | CogVLM · GQA test-balanced | accuracy: 65.2 -> 71.7 | Helped |
| add Chain of Manipulations reasoning and training | CogVLM · TallyVQA simple | accuracy: 79.8 -> 84.0 | Helped |
| add Chain of Manipulations reasoning and training | CogVLM · TallyVQA complex | accuracy: 68.0 -> 70.1 | Helped |
| add Chain of Manipulations reasoning and training | CogVLM · TextVQA test | VQA Score: 69.7 -> 71.1 | Helped |
| add Chain of Manipulations reasoning and training | CogVLM · ST-VQA test | VQA Score: 61.0 -> 70.0 | Helped |
| add Chain of Manipulations reasoning and training | CogVLM · RefCOCO val | accuracy: 92.51 -> 92.34 | Hurt |
| add Chain of Manipulations reasoning and training | CogVLM · RefCOCO test-A | accuracy: 93.95 -> 94.57 | Helped |
| add Chain of Manipulations reasoning and training | CogVLM · RefCOCOg val | accuracy: 89.46 -> 89.32 | Hurt |
| add Chain of Manipulations reasoning and training | CogVLM · MM-Vet | score: 45.5 -> 46.1 | Helped |
Surprising findings
- Only about 35.55% of GPT-4 generated reasoning chains reached the golden answer, so most candidate paths were discarded.
- Gains over CogVLM on general benchmarks are small (MM-Vet +0.6, POPE adversarial +0.6), while detailed VQA gains are large.
Limitations
- Many generated negative paths due to inaccurate visual tools (coarse boxes, OCR errors), limiting data diversity.
- Re-inputting manipulated images slows inference and adds time overhead.
- Linguistic reasoning steps from GPT-4 lack diversity.
Experimental setup
Datasets: TextVQA ST-VQA TDIUC MathVista ChartQA MultiInstruct LLaVAR ShareGPT4V LAION-2B COYO-700M LAION-115M GQA
Benchmarks: GQA TextVQA ST-VQA TallyVQA RefCOCO RefCOCO+ RefCOCOg MM-Vet POPE CoM-test MathVista
Metrics: accuracy VQA score IoU@0.5 keypoints-aware reasoning score (Levenshtein + BLEU)
Baselines: CogVLM Qwen-VL LLaVA-1.5 Monkey Shikra InstructBLIP ViperGPT V*
Builds on: CogVLM GPT-4 GroundingDINO PaddleOCR Vicuna
Closest papers by meaning
- CoFFT: Chain of Foresight-Focus Thought for Visual Language Models (NeurIPS 2025 · similarity 0.90)
- CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models (CVPR 2025 · similarity 0.89)
- VGR: Visual Grounded Reasoning (ICLR 2026 · similarity 0.89)
- Zebra-CoT: A Dataset for Interleaved Vision-Language Reasoning (ICLR 2026 · similarity 0.89)
- TVI-CoT: Text-Visual Interleaved Chain-of-Thought Reasoning for Multimodal Understanding (ICML 2026 · similarity 0.89)
- Visual Chain-of-Thought Prompting for Knowledge-Based Visual Reasoning (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.