Research map / Vision-language and multimodal / Cross-modal learning
Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes circuits for analogous visual and textual tasks in three VLMs, finds modality-specific data-processing components, and proposes training-free back-patching of late visual token representations into earlier layers to close about a third of the visual-textual accuracy gap.
Problem
VLMs answer analogous tasks more accurately on text than on images (e.g., counting, arithmetic), and the mechanistic source of this multimodal performance gap is unclear.
Approach
Decompose task circuits into data, query, and generation sub-circuits; show query and generation components are functionally interchangeable across modalities while data components are not; then back-patch later-layer visual token activations (more textually aligned) into earlier layers at test time.
Key results
- Back-patching improves visual-task accuracy by an average of 4.6% absolute across tasks and models, closing about 32% of the visual-textual gap.
- Vision and language circuits share on average only about 18% of components; normalized IoU is near zero for data positions and about 12% for query positions.
- Interchanging query and generation sub-circuits across modalities preserves faithfulness, while interchanging data sub-circuits drops faithfulness to the random baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Patch later-layer visual token representations into earlier layers | VLM visual task prompts · Counting, Arithmetic, Color Ordering, Factual Recall, Sentiment Analysis | accuracy: Qwen2-7B-VL: +4.2% ± 1.9%, +9.4% ± 1.3%, +0.6% ± 0.6%, +1.6% ± 2.2%, +2.2% ± 1.4%; Pixtral-12B: +2.2% ± 1.1%, +6.7% ± 1. | Helped |
| Patch later-layer visual token representations into earlier layers | VLM visual prompts · VQAv2 | accuracy: 65.0% -> 68.2% ± 0.9%; 62.2% -> 66.2% ± 1.1%; 65.7% -> 70.3% ± 1.3% | Helped |
| Patch later-layer visual token representations into earlier layers | VLM visual prompts · RealWorldQA | accuracy: 58.8% -> 61.6% ± 1.0%; 58.3% -> 60.1% ± 1.2%; 59.1% -> 60.0% ± 0.8% | Helped |
| Apply back-patching repeatedly instead of once (ablation) | Back-patching method · Highest-performing model/task settings | accuracy | Hurt |
| Back-patch textual prompts as a control rather than visual prompts (ablation) | Back-patching method · Textual variants of five tasks | accuracy | Mixed |
| Swap the query or generation sub-circuit across modalities (ablation) | Textual and visual task circuits · Five analogous tasks | faithfulness | No clear effect |
| Swap the data-processing sub-circuit across modalities (ablation) | Textual and visual task circuits · Five analogous tasks | faithfulness | Hurt |
Surprising findings
- Back-patching on textual prompts yields smaller gains than on visual prompts in roughly 79% of cases, supporting that gains come from textual alignment rather than extra compute.
- Iterative back-patching beyond the first application generally degrades accuracy.
Limitations
- Restricted to single-image QA with single-token answers due to circuit discovery constraints.
- Only adapter-based VLMs analyzed; findings may not transfer to other fusion approaches.
- Remaining performance gap (about two-thirds) is not explained.
Experimental setup
Datasets: CLEVR VQAv2 RealWorldQA Wikimedia Commons entity images
Benchmarks: VQAv2 RealWorldQA Analogous VLM task dataset
Metrics: accuracy circuit faithfulness normalized IoU logit difference bootstrap standard deviation
Baselines: Qwen2-7B-VL-Instruct baseline accuracy Pixtral-12B baseline accuracy Gemma-3-12B-it baseline accuracy control back-patching on textual prompts
Builds on: attribution patching integrated gradients path patching TransformerLens
Closest papers by meaning
- Vision-Language Models Create Cross-Modal Task Representations (ICML 2025 · similarity 0.92)
- Words or Vision: Do Vision-Language Models Have Blind Faith in Text? (CVPR 2025 · similarity 0.91)
- Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs? (ICML 2025 · similarity 0.91)
- Caption This, Reason That: VLMs Caught in the Middle (NeurIPS 2025 · similarity 0.91)
- Aligning What Vision-Language Models See and Perceive with Adaptive Information Flow (CVPR 2026 · similarity 0.90)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.