Research map / Vision-language and multimodal / Multimodal LLMs
Bring Reason to Vision: Understanding Perception and Reasoning through Model Merging
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Merges the language backbone of VLMs (LLaVA-Next, Idefics2, InternVL2) with math/reasoning LLM task vectors via linear weight interpolation, training-free, improving math benchmarks, and uses knockout/masking to localize perception vs. reasoning across layers.
Problem
VLMs lag behind LLMs in multimodal reasoning, and it is unclear how perception and reasoning abilities are combined and localized within VLM parameters.
Approach
Model merging across modalities: linearly interpolate the VLM's language-model weights with a reasoning-specialized text LLM's task vector (no training), and use layer-wise masking/replacement of merged parameters as an interpretability tool to locate perception and reasoning abilities.
Key results
- Merging LLaVA-Next-LLaMA3-8B with Dart-Prop (lambda=0.9) raises MathVista Math subset from 25.4 to 28.9 and MathVerse Text-Dominant from 25.9 to 30.7.
- Merging Idefics2-8B with MAmmoTH-1 gives the best average gain (about 1.0 point), while most task vectors give no benefit for Idefics2.
- Masking analysis suggests perception is concentrated in early layers, while math reasoning relies on the later layers (roughly the last five or more); after merging, reasoning sensitivity spreads across nearly all layers.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Linear merge of Dart-Prop math task vector into LLaVA-Next language model (λ=0.9) | LLaVA-Next-LLaMA3-8B · MathVista Math subset | accuracy: +3.5 | Helped |
| Linear merge of Dart-Prop math task vector into LLaVA-Next | LLaVA-Next-LLaMA3-8B · MathVerse Text-Dominant | accuracy: +4.8 | Helped |
| Linear merge of Dart-Prop math task vector into LLaVA-Next | LLaVA-Next-LLaMA3-8B · MathVision | accuracy: +1.0 | Helped |
| Linear merge of general-purpose reasoning task vectors (MAmmoTH-2, Magpie-v0.3) into LLaVA-Next | LLaVA-Next-LLaMA3-8B · MathVista All | accuracy: +0.3 to +0.5 (not significant) | No clear effect |
| Merge with math task vectors (Dart) into LLaVA-Next | LLaVA-Next-LLaMA3-8B · MathVista General VQA / MathVerse Vision-Dominant | accuracy: General 51.7 -> 48.7 for Dart-prop | Hurt |
| Merge with Dart-Prop/Uniform/MAmmoTH variants into Idefics2-8B (Mistral-based) | Idefics2-8B · MathVista All | accuracy: MAmmoTH-1 +1.2 All; Dart-Prop -0.2 All; most vectors give no benefit | Mixed |
| Merge Dart-Uniform task vector into InternVL2-76B | InternVL2-LLaMA3-76B · MathVerse Overall | accuracy: +1.2 | Helped |
| Merge with logical reasoning (LogiCoT) task vector into LLaVA-Next (ablation) | LLaVA-Next-LLaMA3-8B · MathVerse Overall | accuracy: +3.4 | Helped |
| Merge Qwen2-Math into Qwen2-VL-7B (ablation) | Qwen2-VL-7B · MathVista All / MMStar | accuracy: MathVista All 61.2 -> 60.2 | Hurt |
| Replace linear merging with Dare-TIES, Dare-Linear, or Swap5Layers, or TIES (ablation) | LLaVA-Next-LLaMA3-8B merged model · MathVerse-Text-Dominant and MathVision | accuracy: TIES 31.4 vs linear 30.7 on T-D; Dare-TIES 22.7 on T-D but 17.8 on MathVision | Mixed |
Surprising findings
- Merging with general reasoning models (MAmmoTH-2, Magpie) gives no statistically significant gains, while math-specialized Dart-Prop does.
- Merging gives little or negative change on vision-dominant and general VQA tasks, and VLMs already pretrained on math text benefit less.
Limitations
- Gains are modest (often 1-4 points) and mostly on math benchmarks; the paper reports degradation on general and vision-dominant tasks.
- Hyperparameter lambda is tuned on MathVista/Dart-Prop, and results are reported for a small set of VLMs.
- Interpretability conclusions rely on masking/replacement heuristics that inject noise rather than causal interventions.
Experimental setup
Datasets: MathVista MathVerse MathVision DynaMath MMStar MM-Math LogiCoT
Benchmarks: MathVista MathVerse MathVision DynaMath MMStar MM-Math
Metrics: accuracy absolute accuracy change statistical significance (p<0.05)
Baselines: TIES merging Dare-TIES Dare-Linear Layer Swapping MAmmoTH-1 MAmmoTH-2 Magpie-v0.3 DeepSeek-R1-Distill
Builds on: task arithmetic TIES-merging DARE Layer Swapping
Closest papers by meaning
- From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models (ICML 2026 · similarity 0.93)
- Don't Show Pixels, Show Cues: Unlocking Visual Tool Reasoning in Language Models via Perception Programs (CVPR 2026 · similarity 0.91)
- VisuRiddles: Fine-grained Perception is a Primary Bottleneck for Multimodal Large Language Models in Abstract Visual Reasoning (ICLR 2026 · similarity 0.91)
- Perception Tokens Enhance Visual Reasoning in Multimodal Language Models (CVPR 2025 · similarity 0.91)
- SPARC: Separating Perception And Reasoning Circuits for Test-time Scaling of VLMs (ICML 2026 · similarity 0.90)
- Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.