Research map / Vision-language and multimodal / Multimodal LLMs
ReFocus: Visual Editing as a Chain of Thought for Structured Image Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ReFocus, which lets multimodal LLMs write Python code that visually edits table/chart images (draw boxes, highlight, mask) as visual chain-of-thought, and distills such traces into Phi-3.5-vision via SFT on a 14k ChartQA-based dataset.
Problem
Multimodal LLMs struggle with multi-hop selective attention over structured images such as tables and charts, limiting intermediate reasoning to text.
Approach
Equip MLLMs with image-editing tools (draw box, highlight, mask) invoked through generated Python code so the edited image becomes a visual thought fed back as new input, iteratively refocusing attention without adding external information.
Key results
- ReFocus with GPT-4o (2024-08-06) improves VWTQ from 66.4 to 77.2 and VWTQ_syn from 70.4 to 82.8.
- ReFocus with GPT-4o (2024-05-13) improves CharXiv from 49.0 to 57.3 and Horizontal Bar from 78.2 to 85.4.
- SFT of Phi-3.5-vision on ReFocus VCoT data reaches 71.4 average on bar charts vs 63.4 for SFT on QA data (+8.0) and 68.8 for CoT data (+2.6).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ReFocus visual editing | GPT-4o-2024-05-13 · VWTQ | accuracy: 66.5 -> 76.9 (+10.4) | Helped |
| Add ReFocus visual editing | GPT-4o-2024-05-13 · VWTQ_syn | accuracy: 73.2 -> 79.6 (+3.4) | Helped |
| Add ReFocus visual editing | GPT-4o-2024-05-13 · VTabFact | accuracy: 89.6 -> 89.6 (+0.0) | No clear effect |
| Add ReFocus visual editing | GPT-4o-2024-05-13 · CharXiv | accuracy: 49.0 -> 57.3 (+8.3) | Helped |
| Add ReFocus visual editing | GPT-4o-2024-08-06 · CharXiv | accuracy: 48.9 -> 46.2 (-2.7) | Hurt |
| Add ReFocus visual editing | GPT-4o-2024-08-06 · VWTQ_syn | accuracy: 70.4 -> 82.8 (+12.4) | Helped |
| Use mask out editing (ablation) | ReFocus on GPT-4o · VWTQ_syn | accuracy: 70.4 -> 82.4 | Helped |
| Use draw box editing (ablation) | ReFocus on GPT-4o · VWTQ | accuracy: 66.4 -> 77.6 | Helped |
| Use highlight editing (ablation) | ReFocus on GPT-4o · VWTQ | accuracy: 66.4 -> 74.8 | Helped |
| Train with ReFocus visual CoT including bounding boxes rather than QA pairs (ablation) | Phi-3.5-vision SFT · Horizontal Bar | accuracy: 60.6 -> 71.0 | Helped |
| Train with ReFocus visual CoT including bounding boxes rather than QA pairs (ablation) | Phi-3.5-vision SFT · Vertical Bar | accuracy: 66.8 -> 72.2 | Helped |
| Remove refocus bounding box coordinates from ReFocus VCoT training data (ablation) | Phi-3.5-vision SFT · Horizontal Bar | accuracy: 71.0 -> 67.1 | Hurt |
Surprising findings
- Visual edits improve GPT-4o even though no additional information is introduced.
- SFT on plain QA pairs degraded Phi-3.5-vision's visual CoT prompting accuracy (68.2 to 63.4 on bar charts).
Limitations
- Visual editing tools are hand-designed per task (tables and bar charts), limiting generality.
- Evaluation uses GPT-4 as judge and closed GPT-4o backbone, with modest test subsets (e.g., 143 CharXiv items).
Experimental setup
Datasets: VWTQ VWTQ_syn VTabFact CharXiv ChartQA horizontal bar ChartQA vertical bar
Benchmarks: VWTQ VWTQ_syn VTabFact CharXiv ChartQA horizontal bar ChartQA vertical bar
Metrics: accuracy
Baselines: GPT-4o LLaVA-NeXT Phi 3 vision Gemini-Pro 1.5 VisProg Phi-3.5-vision
Builds on: Visual Sketchpad VisProg chain-of-thought prompting Set-of-Mark
Closest papers by meaning
- TVI-CoT: Text-Visual Interleaved Chain-of-Thought Reasoning for Multimodal Understanding (ICML 2026 · similarity 0.88)
- Focus, Don't Prune: Identifying Instruction-Relevant Regions for Information-Rich Image Understanding (CVPR 2026 · similarity 0.87)
- Thinking with Programming Vision: Towards a Unified View for Thinking with Images (CVPR 2026 · similarity 0.87)
- Scene Graph Thinking: Reinforcing Structured Visual Reasoning for Multimodal Large Language Models (ICML 2026 · similarity 0.87)
- Factuality Matters: When Image Generation and Editing Meet Structured Visuals (ICLR 2026 · similarity 0.87)
- WEAVE: Unleashing and Benchmarking the In-context Interleaved Comprehension and Generation (CVPR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.