Research map / Vision-language and multimodal / Multimodal LLMs
Visual Perception by Large Language Model’s Weights
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VLoRA, which converts visual features into low-rank LoRA-style weight updates merged into the LLM instead of feeding visual tokens, cutting compute while staying near LLaVA-v1.5 on six MLLM benchmarks.
Problem
Existing MLLMs concatenate many visual tokens with text tokens, making LLM training and inference expensive because attention cost grows quadratically with input length.
Approach
Replace input-space alignment with parameter-space alignment: a perceptual weights generator (decoder-only with cross-attention over learnable queries) maps image features to low-rank matrices that are added to the LLM's weight matrices, so no visual tokens enter the LLM input.
Key results
- VLoRA uses 0 visual tokens in the LLM and about 619 GFLOPs versus 8027 for LLaVA-v1.5 at text length 32 (about 8%).
- VLoRA scores 63.4 on MMBench vs 64.3 for LLaVA-v1.5, and 36.0 on MMMU vs 35.7 for LLaVA-v1.5.
- VLoRA scores 1311.3 on MME vs 1510.7 for LLaVA-v1.5, a notable gap the authors attribute to random initialization of the generator and limited pre-training data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Align image features with LLM weights and omit visual tokens | VLoRA · MMBench | score: 63.4 vs 64.3 | No clear effect |
| Align image features with LLM weights and omit visual tokens | VLoRA · MME | score: 1311.3 vs 1510.7 | Hurt |
| Align image features with LLM weights and omit visual tokens | VLoRA · ScienceQA | score: 66.4 vs 66.8 | No clear effect |
| Align image features with LLM weights and omit visual tokens | VLoRA · HallusionBench | score: 26.4 vs 36.8 | Hurt |
| Align image features with LLM weights and omit visual tokens | VLoRA · MMMU | score: 36.0 vs 37.0 | Hurt |
| Align image features with LLM weights and omit visual tokens | VLoRA · CCBench | score: 28.6 vs 41.2 | Hurt |
| Replace input-space visual tokens with parameter-space perceptual weights | VLoRA vs LLaVA-v1.5 · LLM overhead (text length 32) | GFLOPs: 619 vs 8027 | Helped |
| Add perceptual weights for all five weight types rather than a subset (ablation) | VLoRA · MMBench | score: 59.6 -> 63.4; 59.4 -> 63.4; 57.2 -> 63.4; 53.3 -> 63.4 | Helped |
| Add value-weight perceptual weights to qk (ablation) | VLoRA · MMBench | score: 53.3 -> 59.4 | Helped |
| Increase perceptual-weight rank from 16 to 64 (ablation) | VLoRA · MMBench | score: 59.4 -> 63.4 | Helped |
| Increase perceptual-weight rank from 64 to 128 (ablation) | VLoRA · MMBench | score: 63.4 -> 61.0 | Hurt |
| Increase perceptual-weight rank from 16 to 64 (ablation) | VLoRA · MME | score: 1212.7 -> 1311.3 | Helped |
Surprising findings
- Replacing LLaVA-v1.5's projector with a randomly initialized Q-Former drops MMBench to 60.7 and MME to 1241.5, and more pre-training data (CapsFus-30m) does not clearly help LLaVA-v1.5.
Limitations
- CLIP features may not suit conversion into model weights; a vision encoder designed for this paradigm may be needed.
- One perceptual weights generator is used per weight type, so correlations between generated weight types may be insufficient.
- MME performance trails LLaVA-v1.5 because the generator is randomly initialized and needs more pre-training data.
Experimental setup
Datasets: CapsFusion-120M LAION-COCO VQAv2 GQA OKVQA OCRVQA A-OKVQA TextCaps RefCOCO Visual Genome ShareGPT LLaVA-Instruct
Benchmarks: MMBench CCBench MME ScienceQA HallusionBench MMMU
Metrics: accuracy score GFLOPs
Baselines: LLaVA-v1.5 Qwen-VL-Chat InstructBLIP MiniGPT-4-v1 MiniGPT-4-v2 Idefics-instruct OpenFlamingo v2 Qwen-VL
Builds on: LoRA LLaVA-v1.5 DETR BLIP-2 Vicuna
Closest papers by meaning
- LION: Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge (CVPR 2024 · similarity 0.90)
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.90)
- ROSE: Rotate Your Large Language Model to See (CVPR 2026 · similarity 0.90)
- MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs (ICLR 2025 · similarity 0.90)
- DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perception (NeurIPS 2024 · similarity 0.90)
- Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.