AI Paper Atlas

Research map / Vision-language and multimodal / Multimodal LLMs

Where to Connect? Boosting MLLMs via Dynamic Gated Pathways across ALL ViT and LLM Layers

Yingying Yan, Jiaqi Tang, Wei Wei, Qianzhou Wang, Jianmin Chen, Yuyang Xia, Botong Geng, Jinjian Wu, Lei Zhang, Qifeng Chen

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Training-free token reduction for MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes dynamic gated pathways that connect multiple ViT layers to all LLM layers in an MLLM, letting each LLM layer select visual granularity instead of using a single fixed visual layer.

Problem

Mainstream MLLMs inject a fixed single-layer visual representation into the LLM and treat all LLM layers as one consumer, ignoring that different LLM layers may need different visual granularities.

Approach

Connects all hierarchical ViT layers to all LLM layers through dynamic gated pathways, so each LLM layer adaptively weights and fuses visual features from different encoder depths rather than receiving one fixed-layer representation.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add dynamic gated all-to-all ViT-LLM routingLLaVA-1.5-7B · SQA-IMGaccuracy: 66.8 -> 71.0Improved
add dynamic gated all-to-all ViT-LLM routingLLaVA-1.5-7B · MM-Vetscore: 30.5 -> 33.8Improved
add dynamic gated all-to-all ViT-LLM routingLLaVA-1.5-13B · LLaVA-Wildscore: 70.7 -> 70.0Hurt
add DGP to other MLLM frameworksQwen3-VL-2B and LLaVA-NeXT · GQA, SQA-IMG, MMBench, MM-Vetaccuracy: LLaVA-NeXT SQA-IMG 69.5 -> 72.3Improved
replace gated routing with uniform average over 24 ViT layers (ablation)LLaVA-1.5-7B · most benchmarksaccuracy: GQA 62.0 -> 60.2; POPE 85.9 -> 77.2Hurt
add gated routing (ablation)LLaVA-1.5-7B · most benchmarksaccuracy: GQA 62.0 -> 64.1; SQA-IMG 66.8 -> 70.5Improved
add entropy regularizer on top of gated routing (ablation)DGP · 11 benchmarksaccuracy: GQA 64.1 -> 64.6; MM-Vet 32.4 -> 33.8Improved
unfreeze vision encoder during training (ablation)DGP-7B · GQA, VQAv2, VizWiz, SEED-I, POPEaccuracy: GQA 64.6 -> 64.1; POPE 87.0 -> 84.8Hurt
unfreeze vision encoder during training (ablation)DGP-7B · MMBenchaccuracy: 66.7 -> 68.3Improved
vary gate initialization 0.1, 0.3, 0.5 (ablation)DGP-7B · 11 benchmarksaccuracy: GQA 64.4 / 64.6 / 64.3No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy