Research map / Vision-language and multimodal / Multimodal LLMs
CoFFT: Chain of Foresight-Focus Thought for Visual Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoFFT, a training-free iterative reasoning method that alternates diverse sample generation, dual foresight decoding (attention-based focus score plus reasoning-progression score), and attention-guided visual cropping to improve VLM visual reasoning.
Problem
VLMs are distracted by large amounts of visually salient but task-irrelevant image content, producing redundant reasoning and hallucinations, because they cannot locate and process the regions needed during multi-step reasoning.
Approach
Interleaves multi-step reasoning with dynamic visual focus: candidate multi-step reasoning samples are scored jointly by cross-attention alignment with the question (relative to a descriptive-prompt baseline) and by mean log-probability improvement; the first step of the best sample is kept, and a sliding window crops and magnifies the region that is relevant to both the question and the future re
Key results
- CoFFT on Qwen2.5-VL-7B reaches 48.19 average vs 42.72 baseline, with a 52.96 average for Qwen2.5-VL-32B vs 47.05 baseline.
- Average gains of 3.1-5.8% across models, with CoFFT best on all seven benchmarks in the main table.
- CoFFT costs about 2.38e+14 FLOPS on Qwen2.5-VL-7B versus 4.05e+14 for MCTS and 8.35e+12 for the baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the visual-focus score E_att to DFD (ablation) | CoFFT with Qwen2.5-VL-7B-Instruct · MathVista | Pass@1 accuracy: 68.5 -> 70.4 | Helped |
| Add the visual-focus score E_att to DFD (ablation) | CoFFT with Qwen2.5-VL-7B-Instruct · SeekWorld-China | Pass@1 accuracy: 28.42 -> 35.12 | Helped |
| Add adaptive image cropping through VFA (ablation) | CoFFT with Qwen2.5-VL-7B-Instruct · MathVista | Pass@1 accuracy: 69.3 -> 70.4 | Helped |
| Add adaptive image cropping through VFA (ablation) | CoFFT with Qwen2.5-VL-7B-Instruct · SeekWorld-China | Pass@1 accuracy: 27.08 -> 35.12 | Helped |
| Increase foresight length from l=3 to l=7 (ablation) | CoFFT with Qwen2.5-VL-7B-Instruct · MathVista | Pass@1 accuracy: 68.8 -> 70.7 | Helped |
| Increase sampling number from k=2 to k=10 (ablation) | CoFFT with Qwen2.5-VL-7B-Instruct · MathVista | Pass@1 accuracy: 68.8 -> 72.2 | Helped |
| Apply CoFFT rather than the strongest listed baseline | Qwen2.5-VL-7B-Instruct · MathVista | Pass@1 accuracy: 69.9 -> 70.4 | Helped |
| Apply CoFFT rather than the strongest listed baseline | Qwen2.5-VL-7B-Instruct · MathVision | Pass@1 accuracy: 19.73 -> 23.36 | Helped |
| Apply CoFFT rather than the strongest listed baseline | Qwen2.5-VL-7B-Instruct · MMStar | Pass@1 accuracy: 66.2 -> 69.4 | Helped |
| Apply CoFFT rather than the strongest listed baseline | Qwen2.5-VL-7B-Instruct · M3CoT | Pass@1 accuracy: 61.34 -> 62.47 | Helped |
| Apply CoFFT rather than the strongest listed baseline | Qwen2.5-VL-7B-Instruct · Charxiv | Pass@1 accuracy: 45.3 -> 47.2 | Helped |
| Apply CoFFT rather than the strongest listed baseline | Qwen2.5-VL-7B-Instruct · SeekWorld-China | Pass@1 accuracy: 32.44 -> 35.12 | Helped |
Surprising findings
- Simply combining existing visual search (DyFo) and reasoning (Predictive Decoding) methods often yields suboptimal results, due to design incompatibilities.
- CoFFT can introduce new errors on problems the base VLM already solves correctly.
Limitations
- CoFFT can introduce errors on problems that VLMs alone solve correctly.
- Higher computational overhead than direct inference, scaling with samples k and foresight length l.
- Gap to closed-source models like GPT-4o and Gemini-2.0-Flash on SeekWorld-Global, attributed to regional knowledge differences.
Experimental setup
Datasets: MathVista MathVision MMStar M3CoT CharXiv SeekWorld-China SeekWorld-Global
Benchmarks: MathVista MathVision MMStar M3CoT CharXiv SeekWorld-China SeekWorld-Global
Metrics: Pass@1 accuracy FLOPS
Baselines: MCTS Predictive Decoding ICoT DyFo GPT-4o Claude-3.5-sonnet Gemini-2.0-Flash
Builds on: Predictive Decoding DyFo Monte Carlo Tree Search Interleaved chain-of-thought
Closest papers by meaning
- When to Think and When to Look: Uncertainty-Guided Lookback (CVPR 2026 · similarity 0.92)
- From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models (ICML 2026 · similarity 0.92)
- See Further, Think Deeper: Advancing VLM's Reasoning Ability with Low-level Visual Cues and Reflection (CVPR 2026 · similarity 0.91)
- Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.91)
- Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas (ICML 2025 · similarity 0.91)
- Look-Back: Implicit Visual Re-focusing in MLLM Reasoning (AAAI 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.