Research map / Vision-language and multimodal / Multimodal LLMs
Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Mixture-of-Resolution Adaptation (MRA), a dual-pathway (low-res ViT + high-res ConvNeXt) design with MR-Adapters that injects high-resolution features into the low-resolution pathway, yielding LLaVA-HR at up to 1536px with efficient training and inference.
Problem
Existing MLLMs use low input resolutions, which hurts granular visual recognition, but increasing resolution greatly increases visual token count, computation, and training instability.
Approach
Use two visual pathways, a low-resolution ViT for global semantics and a high-resolution CNN with large downsampling for fine detail, and fuse high-resolution features into the ViT blocks via dynamically gated mixture-of-resolution adapters, keeping the LLM input short.
Key results
- LLaVA-HR outperforms existing MLLMs on 8 of 11 VL tasks, e.g., +9.4% (text says +9.6%) on TextVQA over LLaVA-1.5 at 1024px.
- At 1024px, LLaVA-HR trains in 20.7 GPU-hours on 8 A800s versus 69.4h for LLaVA-1.5 at 1022px, with 3x faster inference than LLaVA-1.5 at the same resolution.
- LLaVA-HR reaches 1554.9 on MME perception and 67.1 on TextVQA at 1024px, while LLaVA-1.5 at 1022px drops to 37.8 on TextVQA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply MRA instead of LLaVA-1.5 at 1024 resolution | LLaVA-1.5 · TextVQA | accuracy: +8.33% vs. +4.82% | Helped |
| Use high-to-low rather than low-to-high fusion (ablation) | LLaVA-HR · MME | score: 1463.5 -> 1524.8 | Helped |
| Keep the low-resolution pathway rather than remove it (ablation) | LLaVA-HR · VQAv2 | accuracy: 80.3 -> 81.8 | Helped |
| Scale up the high-resolution encoder from ConvNeXt-L to ConvNeXt-XXL (ablation) | LLaVA-HR · TextVQA | accuracy: +2.1% | Helped |
| Scale up the low-resolution encoder from ViT-L to ViT-G (ablation) | LLaVA-HR · TextVQA | accuracy: +0.9% | Helped |
| Use sum rather than concatenation for fusion (ablation) | LLaVA-HR · MME | score: 1508.8 -> 1524.8 | Helped |
| Use convolution blocks for the low-resolution mapping and MLP for the high-resolution mapping (ablation) | MR-Adapter · MME | score: 1499.0 -> 1524.8; 1517.9 -> 1524.8 | Helped |
| Use Tanh rather than alternative gates (ablation) | MR-Adapter · POPE | score: 86.9 -> 88.0; 87.8 -> 88.0 | Mixed |
| Increase image resolution for LLaVA-1.5 | LLaVA-1.5 · TextVQA | accuracy: +4.8% | Mixed |
Surprising findings
- LLaVA-1.5 performance drops sharply at 1022px resolution, attributed to visual sequence length exceeding the LLM's pre-trained context length.
- Using a ConvNeXt-only high-resolution backbone (Conv+MLP) is faster but loses 108.9 MME points versus the full MRA design.
Limitations
- Efficiency gains at very high resolution depend on dual-pathway design; inference speed drops at larger LLaVA-HR-X scale (12.9 tokens/s).
- Evaluated only on LLaVA-1.5 backbone; generalization to other MLLM architectures not shown in the text.
Experimental setup
Datasets: LCS-558K LLaVA-1.5 665K instruction data VQAv2 GQA OKVQA OCRVQA ScienceQA VizWiz TextVQA MME POPE SEED
Benchmarks: VQAv2 GQA OKVQA OCRVQA ScienceQA VizWiz TextVQA MME POPE SEED MM-Vet
Metrics: accuracy MME perception score training time GPU memory inference speed (tokens/s)
Baselines: LLaVA-1.5 InstructBLIP Qwen-VL-Chat BLIP-2 mPLUG-Owl2 Fuyu-8B IDEFICS Shikra
Builds on: LLaVA-1.5 CLIP ConvNeXt QFormer
Closest papers by meaning
- Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models (AAAI 2025 · similarity 0.90)
- Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification (ICLR 2025 · similarity 0.90)
- iLLaVA: An Image is Worth Fewer Than 1/3 Input Tokens in Large Multimodal Models (ICLR 2026 · similarity 0.89)
- HiRes-LLaVA: Restoring Fragmentation Input in High-Resolution Large Vision-Language Models (CVPR 2025 · similarity 0.89)
- CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models (CVPR 2026 · similarity 0.89)
- Accelerating Multimodal Large Language Models by Searching Optimal Vision Token Reduction (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.