Research map / Vision-language and multimodal / Multimodal LLMs
Multi-Layer Visual Feature Fusion in Multimodal LLMs: Methods, Analysis, and Best Practices
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically studies which visual encoder layers to fuse into MLLMs and where/how to fuse them, finding that external direct fusion of multi-layer features from distinct representational stages is the most stable and effective recipe.
Problem
Multi-layer visual features are used in MLLMs with arbitrary layer choices and fusion designs, and there is no systematic guidance on which visual layers to select or how to fuse them with the LLM.
Approach
Defines similarity-based and proportion-based layer selection criteria and a 2x2 taxonomy of fusion position (internal vs external) and pattern (modular vs direct), then compares all combinations on a lightweight Mini-LLaVA testbed, finding external direct fusion most robust.
Key results
- External direct fusion with the Former or All layer sets reached average 49.88 (All) on Mini-LLaVA, versus a best internal fusion average of 48.74 (Double, pre-cross).
- Pre-cross modular fusion with the All layer set failed to converge, with training loss plateauing around 6 after starting near 11.
- Scaling components, external direct fusion rose from 49.18 (baseline) to 51.76 with SigLIP and 52.63 with MobileLLaMA 2.7B, and to 58.62 with Vicuna 1.5 7B using Triple layers.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fuse all visual layers directly at the input rather than using the Mini-LLaVA baseline | Mini-LLaVA · Overall benchmark average | average score: 48.51 -> 49.88 | Helped |
| Use external direct fusion rather than internal direct fusion with the same Triple layer set (ablation) | Mini-LLaVA with layers {3, 18, 23} · Overall benchmark average | average score: 48.54 -> 49.18 | Helped |
| Use all layers instead of former-stage features in external direct fusion (ablation) | Mini-LLaVA · Overall benchmark average | average score: 48.95 -> 49.88 | Helped |
| Add beginning-stage layer 3 to middle-stage layer 18 in internal modular pre-cross fusion (ablation) | Mini-LLaVA · Overall benchmark average | average score: 48.60 -> 48.74 | Helped |
| Add ending-stage layer 23 to layers {3, 18} in internal modular pre-cross fusion (ablation) | Mini-LLaVA · Overall benchmark average | average score: 48.74 -> 46.91 | Hurt |
| Use former-stage rather than latter-stage features in internal modular pre-cross fusion (ablation) | Mini-LLaVA · Overall benchmark average | average score: 37.19 -> 45.60 | Helped |
| Replace pre-cross with post-cross modular fusion using the Double layer set (ablation) | Mini-LLaVA internal modular fusion · Overall benchmark average | average score: 48.74 -> 49.24 | Helped |
| Replace pre-cross with parallel modular fusion using the Double layer set (ablation) | Mini-LLaVA internal modular fusion · Overall benchmark average | average score: 48.74 -> 48.43 | Hurt |
| Add layers from the same encoder stage rather than selecting representative features across stages (ablation) | Multi-layer visual feature fusion · Multiple benchmarks | benchmark performance | Hurt |
| Increase SFT data from 332k to 665k for internal fusion (ablation) | Internal direct and modular fusion · Overall benchmark average | average score: 48.39 -> 48.54; 45.24 -> 46.91 | Helped |
| Replace CLIP ViT-L with SigLIP (ablation) | External direct fusion · Overall benchmark average | average score: 49.18 -> 51.76 | Helped |
| Replace CLIP ViT-L with SigLIP (ablation) | Internal modular fusion · Overall benchmark average | average score: 46.91 -> 43.27 | Hurt |
Surprising findings
- Adding more layers from the same stage, including the 23rd layer already used as visual tokens, typically degrades performance.
- Internal modular fusion with the SigLIP encoder dropped sharply to an average of 43.27, below the CLIP baseline.
Limitations
- LLM scale limited to 7B and dataset scale to about 1M samples.
- Main experiments run on a lightweight 1.4B Mini-LLaVA testbed, so conclusions may not transfer fully to large models.
- Modular fusion results are sensitive to layer choice and sometimes fail to converge.
Experimental setup
Datasets: LAION-CC-SBU 558K (BLIP captions) LLaVA-1.5 665K instruction tuning Cambrian-1 737K
Benchmarks: GQA MMBench MME TextVQA OCRBench CV-Bench POPE
Metrics: accuracy score average score across benchmarks
Baselines: Mini-LLaVA Dense Connector DeepStack MMFuser EVLM
Builds on: LLaVA-1.5 DeepStack Dense Connector MMFuser Flamingo-style gated cross-attention
Closest papers by meaning
- Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs (CVPR 2024 · similarity 0.91)
- Where Does Vision Meet Language? Understanding and Refining Visual Fusion in MLLMs via Contrastive Attention (CVPR 2026 · similarity 0.90)
- OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Merging (ICLR 2026 · similarity 0.90)
- Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models (CVPR 2026 · similarity 0.90)
- DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perception (NeurIPS 2024 · similarity 0.90)
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.