Research map / Vision-language and multimodal / Multimodal LLMs
Downscaling Intelligence: Exploring Perception and Reasoning Bottlenecks in Small Multimodal Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies how downscaling the LLM in multimodal models hurts perception and reasoning, then proposes Extract+Think (visual extraction tuning for the perception module plus step-by-step reasoning with Qwen3 thinking mode in the reasoning module), which beats PrismCaptioner and some end-to-end small MLLMs with far less visual data.
Problem
Small multimodal models lose disproportionate visual capability when the backbone LLM is downscaled; it is unclear whether the loss comes from visual reasoning or from perception.
Approach
Decouple perception (a small VLM extracting question-relevant visual details) from reasoning (an LLM answering with step-by-step thinking), and train the perception module with visual extraction tuning, which converts instruction data into fine-grained visual extraction targets generated by Qwen3-VL-8B.
Key results
- Extract+Think (1.7B perception, 4B reasoning) reaches 85.3 average in-domain and 52.6 on MMStar, versus PrismCaptioner (7B/70B) at 78.3 and 45.7.
- The smaller Extract+Think (0.6B perception, 1.7B reasoning) outperforms LLaVA-OneVision-0.5B by 12.9% on in-domain data and 19.5% on MMStar while using 73% fewer visual samples.
- Visual extraction tuning raises in-domain performance by 5.2 points at the 0.6B perception LLM over the captioning baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Visual extraction tuning added to the perception module (vs. captioning baseline) (ablation) | Perception module at 0.6B LLM · In-domain multiple-choice | average accuracy: +5.2 | Helped |
| Visual extraction tuning added to the perception module (vs. captioning baseline) (ablation) | Perception module at 1.7B LLM · In-domain multiple-choice | average accuracy: +4.1 | Helped |
| Visual extraction tuning added to the perception module (vs. captioning baseline) (ablation) | Perception module at 0.6B and 1.7B LLM · MMStar out-of-domain | average accuracy: +3.6 (0.6B), +4.6 (1.7B) | Helped |
| Captioning post-training of the perception module (ALLaVA-4V) (ablation) | Perception module under LLM downscaling · Decoupled analysis, in-domain and out-of-domain | average accuracy | Helped |
| Step-by-step CoT reasoning in the reasoning module (Qwen3 thinking mode) (ablation) | Reasoning module, out-of-domain MMStar · MMStar | average accuracy | Helped |
| Step-by-step CoT reasoning in the reasoning module (ablation) | Reasoning module, in-domain tasks · In-domain multiple-choice | average accuracy | Mixed |
| Downscaling the perception module LLM from 8B to 0.6B (ablation) | Decoupled perception module · In-domain multiple-choice | average accuracy drop: 0.15 drop | Hurt |
| Downscaling the perception module LLM from 8B to 0.6B (ablation) | Decoupled perception module · MMStar out-of-domain | average accuracy drop: 0.07 drop | Hurt |
| Visual extraction tuning data (2.4M vis data, Extract+Think) vs. 0.4M from-scratch configuration | Extract+Think 0.6B/1.7B · In-domain multiple-choice | average accuracy: 80.3 vs 78.0 | Helped |
| Extract+Think (1.7B/4B) vs. PrismCaptioner 7.0B/70B | Two-stage decoupled framework with smaller modules · In-domain and MMStar | average accuracy: 85.3 vs 78.3 in-domain; 52.6 vs 45.7 MMStar | Helped |
| Visual extraction tuning from scratch (0.4M visual data) vs LLaVA-OneVision-0.5B (8.8M) | Extract+Think-dagger 0.6B/1.7B · In-domain multiple-choice | average accuracy: 78.0 vs 71.1 | Helped |
Surprising findings
- Downscaling the perception module reduces performance about as much as downscaling the reasoning module, even on visually-framed reasoning tasks.
- Tasks most sensitive to LLM downscaling (Grounding, NIGHTS) are the vision-centric ones rather than knowledge-heavy tasks like ScienceQA.
Limitations
- Two-stage design is an information bottleneck: the text extraction may omit critical visual details needed for answers.
- Only evaluated with small LLM sizes and Qwen3-based reasoning; findings may not extend to larger LLMs.
- CoT gains do not fully close the reasoning gap for small models.
Experimental setup
Datasets: VQAv2 OCR-VQA TextVQA ScienceQA GQA VizWiz ImageNet RefCOCO M4-Instruct subsets PieAPP ALLaVA-4V BLIP558K
Benchmarks: MMStar VMCBench
Metrics: accuracy average accuracy drop latency
Baselines: LLaVA-OneVision-0.5B InternVL2.5-0.5B SmolVLM-1.7B PrismCaptioner Caption+Think
Builds on: PrismCaptioner Prism framework LLaVA-OneVision Qwen3 NoWait
Closest papers by meaning
- Compose and Fuse: Revisiting the Foundational Bottlenecks in Multimodal Reasoning (ICLR 2026 · similarity 0.91)
- Don't Show Pixels, Show Cues: Unlocking Visual Tool Reasoning in Language Models via Perception Programs (CVPR 2026 · similarity 0.90)
- Thinking Diffusion: Penalize and Guide Visual-Grounded Reasoning in Diffusion Multimodal Language Models (CVPR 2026 · similarity 0.90)
- LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.90)
- Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models (CVPR 2026 · similarity 0.90)
- Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.