Research map / Vision-language and multimodal / Multimodal LLMs
Where to Connect? Boosting MLLMs via Dynamic Gated Pathways across ALL ViT and LLM Layers
Yingying Yan, Jiaqi Tang, Wei Wei, Qianzhou Wang, Jianmin Chen, Yuyang Xia, Botong Geng, Jinjian Wu, Lei Zhang, Qifeng Chen
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes dynamic gated pathways that connect multiple ViT layers to all LLM layers in an MLLM, letting each LLM layer select visual granularity instead of using a single fixed visual layer.
Problem
Mainstream MLLMs inject a fixed single-layer visual representation into the LLM and treat all LLM layers as one consumer, ignoring that different LLM layers may need different visual granularities.
Approach
Connects all hierarchical ViT layers to all LLM layers through dynamic gated pathways, so each LLM layer adaptively weights and fuses visual features from different encoder depths rather than receiving one fixed-layer representation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add dynamic gated all-to-all ViT-LLM routing | LLaVA-1.5-7B · SQA-IMG | accuracy: 66.8 -> 71.0 | Improved |
| add dynamic gated all-to-all ViT-LLM routing | LLaVA-1.5-7B · MM-Vet | score: 30.5 -> 33.8 | Improved |
| add dynamic gated all-to-all ViT-LLM routing | LLaVA-1.5-13B · LLaVA-Wild | score: 70.7 -> 70.0 | Hurt |
| add DGP to other MLLM frameworks | Qwen3-VL-2B and LLaVA-NeXT · GQA, SQA-IMG, MMBench, MM-Vet | accuracy: LLaVA-NeXT SQA-IMG 69.5 -> 72.3 | Improved |
| replace gated routing with uniform average over 24 ViT layers (ablation) | LLaVA-1.5-7B · most benchmarks | accuracy: GQA 62.0 -> 60.2; POPE 85.9 -> 77.2 | Hurt |
| add gated routing (ablation) | LLaVA-1.5-7B · most benchmarks | accuracy: GQA 62.0 -> 64.1; SQA-IMG 66.8 -> 70.5 | Improved |
| add entropy regularizer on top of gated routing (ablation) | DGP · 11 benchmarks | accuracy: GQA 64.1 -> 64.6; MM-Vet 32.4 -> 33.8 | Improved |
| unfreeze vision encoder during training (ablation) | DGP-7B · GQA, VQAv2, VizWiz, SEED-I, POPE | accuracy: GQA 64.6 -> 64.1; POPE 87.0 -> 84.8 | Hurt |
| unfreeze vision encoder during training (ablation) | DGP-7B · MMBench | accuracy: 66.7 -> 68.3 | Improved |
| vary gate initialization 0.1, 0.3, 0.5 (ablation) | DGP-7B · 11 benchmarks | accuracy: GQA 64.4 / 64.6 / 64.3 | No clear effect |
Closest papers by meaning
- Dense Connector for MLLMs (NeurIPS 2024 · similarity 0.90)
- Multi-Layer Visual Feature Fusion in Multimodal LLMs: Methods, Analysis, and Best Practices (CVPR 2025 · similarity 0.90)
- Lifting the Veil on Visual Information Flow in MLLMs: Unlocking Pathways to Faster Inference (CVPR 2025 · similarity 0.90)
- Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Models (CVPR 2026 · similarity 0.90)
- Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference (AAAI 2025 · similarity 0.90)
- Understanding Language Prior of LVLMs by Contrasting Chain-of-Embedding (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
