Research map / Vision-language and multimodal / Multimodal LLMs
Dense Connector for MLLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Dense Connector, a parameter-free vision-language connector that fuses multi-layer frozen CLIP/SigLIP ViT features into LLaVA-style MLLMs, with an efficient token-reduced variant and training-free image-to-video transfer.
Problem
Existing MLLMs feed only the final high-level features of a frozen visual encoder to the LLM, leaving intermediate visual layers unused and visual signal under-exploited.
Approach
Integrate offline features from multiple ViT layers into the connector without adding parameters, via three instantiations: sparse token integration (concatenating pooled multi-layer tokens), sparse channel integration (concatenating layers along channels), and dense channel integration (grouped averaging of all layers before channel concatenation); plus 2D interpolation to cut visual tokens and F
Key results
- Dense Connector with Vicuna-13B (LLaVA-1.5 data) reaches 65.4 GQA, 73.0 SQA, 71.4 MMB, versus LLaVA-1.5 13B at 63.3, 71.6, 67.6.
- Efficient Dense Connector with 144 visual tokens (4x fewer) reaches GQA 62.8, VQAv2 79.4, MMB 67.6, outperforming TokenPacker by 0.9, 1.5, 2.5 points on GQA, VQAv2, MMB.
- Training-free video extension with 34B LLM reaches 77.4% MSVD-QA, 62.1% MSRVTT-QA, 55.8% ActivityNet-QA accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Integrate features from layers 8 and 16 with DCI grouped into two groups (ablation) | LLaVA-1.5 with Vicuna-7B · SQA | accuracy: 66.8 -> 69.5 | Helped |
| Use two DCI groups rather than three (ablation) | LLaVA-1.5 · SQA | accuracy: 67.8 -> 69.5 | Helped |
| Add layer 20 to STI layer selection (ablation) | LLaVA-1.5 · MMBench | accuracy: 67.2 -> 67.6 | Helped |
| Add layer 20 to SCI layer selection (ablation) | LLaVA-1.5 · SQA | accuracy: 68.9 -> 67.0 | Hurt |
| Remove layer 8 from SCI selection (ablation) | LLaVA-1.5 · GQA | accuracy: 63.7 -> 63.0 | Hurt |
| Replace STI average pooling with 1D convolution downsampling (ablation) | LLaVA-1.5 · GQA | accuracy: 63.3 -> 62.6 | Hurt |
| Add 2D convolution before SCI feature concatenation (ablation) | LLaVA-1.5 · MMBench | accuracy: 66.2 -> 66.0 | No clear effect |
| Add shared linear layer and layer normalization to DCI (ablation) | LLaVA-1.5 · SQA | accuracy: 69.5 -> 68.5 | Hurt |
| Fine-tune the CLIP-L vision encoder during instruction tuning (ablation) | Dense Connector with Vicuna-7B · MMBench | accuracy: 66.8 -> 68.6 | Helped |
| Fine-tune the CLIP-L vision encoder during instruction tuning (ablation) | Dense Connector with Vicuna-7B · SQA | accuracy: 69.5 -> 67.4 | Hurt |
| Use SigLIP-SO instead of CLIP-L as the visual encoder | Dense Connector with Vicuna-7B · MMBench | accuracy: 66.8 -> 68.4 | Helped |
| Increase training data from LLaVA data to Mini-Gemini data (ablation) | Dense Connector with Vicuna-13B · MathVista | accuracy: 29.6 -> 36.5 | Helped |
Surprising findings
- Adding extra learnable parameters (1D conv, 2D conv, linear layers) to the connector did not improve performance over parameter-free instantiations.
- Fine-tuning the ViT gave gains on MMBench and TextVQA but a slight drop on ScienceQA.
Limitations
- Three instantiations add no parameters; the authors could not find an effective way to add parameters to the connector.
- Video capability is training-free and trained only on images, so the model sometimes misreads videos as images.
- 70B model results use LoRA fine-tuning and lack initial-stage pretraining data, so its full potential is not reached.
Experimental setup
Datasets: LLaVA-1.5 pretraining data LLaVA-1.5 instruction tuning data Mini-Gemini data
Benchmarks: GQA VQAv2 ScienceQA-IMG TextVQA POPE MathVista MMBench MM-Vet MMMU LLaVA-Bench-In-the-Wild MME MSVD-QA
Metrics: accuracy score MME perception score
Baselines: LLaVA-1.5 LLaVA-NeXT Mini-Gemini TokenPacker Qwen-VL-Chat TinyLLaVA LLaMA-VID Video-ChatGPT FreeVA
Builds on: LLaVA-1.5 LLaVA-NeXT Mini-Gemini FreeVA FPN
Closest papers by meaning
- DenseMLLM: Standard Multimodal LLMs for Dense Prediction (ICML 2026 · similarity 0.91)
- Visual Anchors Are Strong Information Aggregators For Multimodal Large Language Model (NeurIPS 2024 · similarity 0.91)
- DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perception (NeurIPS 2024 · similarity 0.90)
- FastVLM: Efficient Vision Encoding for Vision Language Models (CVPR 2025 · similarity 0.90)
- iLLaVA: An Image is Worth Fewer Than 1/3 Input Tokens in Large Multimodal Models (ICLR 2026 · similarity 0.89)
- Rethinking the Mixture of Vision Encoders Paradigm for Enhanced Visual Understanding in Multimodal LLMs (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.