Research map / Vision-language and multimodal / Multimodal LLMs
Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces synthetic visual algorithmic reasoning tasks (TableRead, GridNav, VisualAnalogy) with SIMPLE/HARD splits and shows that mixing text, image, and image-via-text supervision plus hard-text data transfers simple-to-hard generalization from text to image in VLMs.
Problem
VLMs underperform on multi-step visual reasoning compared with the same tasks in text form (modality imbalance); it is unclear how to make simple-to-hard generalization on images match text.
Approach
Controlled synthetic tasks with text and image equivalents let the authors study modality gaps under simple-to-hard generalization; explicit image-to-text conversion in chain-of-thought, mixed with text supervision and a hard-text warm-up/alignment phase, transfers reasoning from text to image, and gradient alignment scores explain why.
Key results
- Text supervision reaches about 80% on HARD-text TableRead while image supervision reaches only about 20% on HARD-image, a modality gap.
- Mix+ supervision yields 64%, 92%, and 35% HARD-image accuracy on TableRead, GridNav, and VisualAnalogy respectively.
- Align-Mix+ raises HARD-image accuracy to 76%, 96%, and 56% on the three tasks while keeping inference cost low.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with image-to-text conversion supervision instead of image-and-text mixing alone | Consecutive Table Read · hard-image | accuracy | Helped |
| Combine image-to-text supervision with text and image supervision (Mix) | Consecutive Table Read · medium-image | accuracy | Helped |
| Add hard-text supervision to Mix training (Mix+) | Table Read, GridNav, and Visual Analogy · hard-image | S2H accuracy: 64%; 92%; 35% | Helped |
| Add a simple text and image-to-text alignment phase before Mix+ | Table Read, GridNav, and Visual Analogy · hard-image | S2H accuracy: 76%; 96%; 56% | Helped |
| Prompt Mix+ with the inference-time token “Convert” | Mix+ models · hard-image reasoning | S2H accuracy | Helped |
| Continue training an intermediate Mix checkpoint with Mix+ supervision (ablation) | Table Read · hard-image solution loss | loss | Helped |
| Add hard-text examples to simple-example Mix training | Table Read · gradient alignment between simple- and hard-image examples | gradient alignment score | Helped |
| Train with Align-Mix+ rather than Mix+ | Table Read · early training steps | gradient alignment score | Helped |
| Use multi-task training with an equal mix of 3 tasks (ablation) | Mix, Mix+, and Align-Mix+ · Table Read and GridNav versus Visual Analogy | performance | Mixed |
| Remove chain-of-thought or progressively internalize it (ablation) | the proposed training strategies · hard-image | S2H accuracy | Hurt |
| Use verbose CoT that explains cell coordinates and path steps rather than a trace listing only path numbers (ablation) | Table Read · image-input task performance | performance | Helped |
| Use a verbose DFS trace that explains valid and invalid actions rather than a minimal trace (ablation) | GridNav · simple-to-hard generalization | S2H accuracy | Helped |
Surprising findings
- Models trained with image-via-text supervision learn to internalize conversion, producing correct solutions without explicitly writing the text conversion at inference.
- Minor changes to the chain-of-thought format, or the color used to highlight cells, substantially change image generalization (e.g., 30% to 70%).
Limitations
- Tasks are synthetic with a limited image distribution and small set of prompts, so transfer to real-world benchmarks is not established.
- Results are sensitive to chain-of-thought design and training epochs/data mixture choices.
- Relies on a text-warm-up or alignment phase; the approach may not be optimal where text generalization already holds.
Experimental setup
Datasets: synthetic TableRead data synthetic GridNav data synthetic VisualAnalogy data (PGM-based)
Benchmarks: TableRead (synthetic) GridNav (synthetic) VisualAnalogy (synthetic)
Metrics: exact match accuracy
Baselines: Text supervision Image supervision Image-text mixed supervision Image-via-text supervision
Builds on: LLaVA Eagle Procedurally Generated Matrices
Closest papers by meaning
- Probing Visual Language Priors in VLMs (ICML 2025 · similarity 0.91)
- Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs (NeurIPS 2025 · similarity 0.91)
- VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMs (CVPR 2026 · similarity 0.91)
- Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs (NeurIPS 2024 · similarity 0.91)
- Modality-Specialized Synergizers for Interleaved Vision-Language Generalists (ICLR 2025 · similarity 0.91)
- Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models (CVPR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.