Research map / Vision-language and multimodal / Multimodal LLMs
Vision Function Layer in Multimodal LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies narrow Vision Function Layers in MLLMs via visual token swapping and dropping, and uses them to guide LoRA target-layer selection and function-based data selection.
Problem
Internal mechanisms of how MLLMs process vision tokens remain a black box; it is unclear which decoder layers implement specific visual functions such as counting, grounding, OCR, and recognition.
Approach
Visual Token Swapping replaces vision-token KV entries at a chosen layer with those of another image during decoding, and Vision Token Dropping removes vision tokens from a layer onward; together they localize visual functions to 2-3 layers that appear in a consistent order across MLLMs. These layer localizations are then used to select LoRA layers and to classify training data by function.
Key results
- VFL-Select with 20% of LLaVA-665k reaches 99.5% of full-data performance on shallow-layer benchmarks and 97.4% on deep-layer benchmarks, above COINCIDE (99.0%, 94.0%) and Random.
- VFL-LoRA uses about 155M tunable parameters versus 309M for standard LoRA, with out-of-domain average 75.0% vs 74.3% for standard LoRA.
- On Qwen2.5-VL-7B token dropping, count-related layers are 14-16 and OCR layers 22-24 per the paper's text; MMMU peaks when some vision layers are dropped (up to +1.8%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply LoRA to count-function layers instead of uniformly across adaptable layers | Qwen2.5-VL · CV-Count | accuracy: 70.9% vs. 72.6% | Helped |
| Apply LoRA to count-function layers instead of uniformly across adaptable layers | Qwen2.5-VL · CV-Bench spatial tasks | average accuracy: 84.4% vs. 85.0% | Mixed |
| Apply LoRA to count-function layers instead of uniformly across adaptable layers | Qwen2.5-VL · Out-of-domain benchmarks | average performance: 75.0% vs. 74.3% | Helped |
| Select training data using VFL rather than random selection | LLaVA · SQA-I, 665k samples | accuracy: 60.4 -> 86.0 | Helped |
| Select training data using VFL rather than expert selection | LLaVA · SQA-I, 665k samples | accuracy: 66.0 -> 86.0 | Helped |
| Select a 20% LLaVA-665k subset using VFL instead of training on the full dataset | LLaVA · Shallow-layer tasks | relative performance: 99.5% | Helped |
| Select a 20% LLaVA-665k subset using VFL instead of training on the full dataset | LLaVA · Deep-layer tasks | relative performance: 97.4% | Helped |
| Drop 4 final vision layers (ablation) | Qwen2.5-VL-7B · MMMU | accuracy: 50.7 -> 50.8 | Helped |
| Drop 18 final vision layers (ablation) | Qwen2.5-VL-7B · MMMU | accuracy: 50.7 -> 45.8 | Hurt |
| Drop 4 final vision layers (ablation) | Qwen2.5-VL-7B · TextVQA | accuracy: 82.8 -> 74.1 | Hurt |
| Drop 8 final vision layers (ablation) | Qwen2.5-VL-7B · TextVQA | accuracy: 82.8 -> 15.3 | Hurt |
| Drop 20 final vision layers (ablation) | LLaVA-v1.5-7B · MMMU | accuracy: 34.7 -> 35.2 | Helped |
Surprising findings
- Dropping up to half of the vision-function layers can leave ScienceQA and MMMU performance unchanged or slightly improved.
- On LLaVA-665k, VFL-Select at 20% data reaches 97.4% of full-data performance on deep-layer benchmarks.
Limitations
- Layer localization is shown mainly on Qwen2.5-VL-7B; other models are reported with less detail.
- Gains from VFL-LoRA are modest on in-domain CV-Bench (84.4% vs 85.0%) and no error bars are reported.
- Code and data are promised only after acceptance.
Experimental setup
Datasets: CLEVR COCO arXiv word corpus SAT LLaVA-665k CV-Bench ChartQA OCRBench MMMU POPE
Benchmarks: ScienceQA-I MMMU MME POPE SEED CVBench TextVQA OCRBench ChartQA VQAv2 GQA AI2D
Metrics: accuracy result change rate relative performance to full data (Rel.%)
Baselines: Standard LoRA Reversed-VFL Random data selection Expert data selection D2-Pruning EL2N COINCIDE
Builds on: LoRA LISA D2-Pruning EL2N COINCIDE
Closest papers by meaning
- Accelerating Multimodal Large Language Models by Searching Optimal Vision Token Reduction (CVPR 2025 · similarity 0.86)
- Multi-Layer Visual Feature Fusion in Multimodal LLMs: Methods, Analysis, and Best Practices (CVPR 2025 · similarity 0.86)
- Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Models (CVPR 2026 · similarity 0.86)
- DenseMLLM: Standard Multimodal LLMs for Dense Prediction (ICML 2026 · similarity 0.85)
- Dense Connector for MLLMs (NeurIPS 2024 · similarity 0.85)
- Large Vision-Language Models Get Lost in Attention (ICML 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.