Research map / Vision-language and multimodal / Multimodal LLMs
LLaVA-UHD v2: Exploiting Hierarchical Vision Granularity in MLLMs via Inverse Semantic Pyramid
Yipeng Zhang, Yifan Liu, Zonghao Guo, Yidan Zhang, Xuesong Yang, Xiaoying Zhang, Chi Chen, Jun Song, Yuan Yao, Tat-Seng Chua, Maosong Sun
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the Hiwin (hierarchical window) transformer, a plug-and-play visual encoder enhancement for MLLMs that exploits hierarchical visual semantic granularity to improve fine-grained perception.
Problem
Vision transformers used as visual encoders in multimodal LLMs perform poorly on fine-grained visual perception tasks, which is attributed to their limited ability to capture diverse visual semantic levels.
Approach
Introduces a Hierarchical window (Hiwin) transformer with an inverse semantic pyramid that builds multi-granularity visual features from a ViT, offered as a plug-and-play module for MLLMs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add VDIM to build inverse semantic pyramid (ablation) | LLaVA-UHD · Average of 15 benchmarks | score: 58.0 -> 60.0 | Improved |
| add VDIM (ablation) | LLaVA-UHD · OCR-Bench | score: 40.9 -> 50.4 | Improved |
| add VDIM (ablation) | LLaVA-UHD · HR-Bench | score: 45.6 -> 43.9 | Hurt |
| add hierarchical window attention on top of VDIM (ablation) | LLaVA-UHD+VDIM · Average of 15 benchmarks | score: 60.0 -> 61.7 | Improved |
| add hierarchical window attention on top of VDIM (ablation) | LLaVA-UHD+VDIM · HR-Bench | score: 43.9 -> 49.0 | Improved |
| add hierarchical window attention on top of VDIM (ablation) | LLaVA-UHD+VDIM · ScienceQA | score: 70.5 -> 69.8 | Hurt |
| add hierarchical window attention on top of VDIM (ablation) | LLaVA-UHD+VDIM · OCR-Bench | score: 50.4 -> 50.1 | No clear effect |
| add full Hiwin (VDIM + attention) (ablation) | LLaVA-UHD · MMMU-val | score: 37.0 -> 37.6 | No clear effect |
| add full Hiwin (VDIM + attention) (ablation) | LLaVA-UHD · RefCOCO/g/+ (REC avg) | accuracy: 68.3 -> 74.0 | Improved |
| construct pyramid with VDIM instead of bilinear interpolation (ablation) | LLaVA-UHD · Pyramid construction, average of 8 benchmarks | score: 62.0 -> 63.0 | Improved |
| use CLIP-ConvNeXt multi-stage feature pyramid (ablation) | LLaVA-UHD · Pyramid construction, average of 8 benchmarks | score: 59.9 -> 59.7 | Hurt |
| use Hiwin attention instead of perceiver resampler to compress pyramid (ablation) | Hiwin-LLaVA-UHD · pre-training loss | loss: 0.6110 -> 0.4368 | Improved |
Experimental setup
Builds on: LLaVA-UHD
Closest papers by meaning
- Lifting the Veil on Visual Information Flow in MLLMs: Unlocking Pathways to Faster Inference (CVPR 2025 · similarity 0.89)
- LION: Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge (CVPR 2024 · similarity 0.89)
- Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Models (CVPR 2026 · similarity 0.88)
- Granulon: Awakening Pixel-Level Visual Encoders with Adaptive Multi-Granularity Semantics for MLLM (CVPR 2026 · similarity 0.88)
- ROSE: Rotate Your Large Language Model to See (CVPR 2026 · similarity 0.88)
- Large Vision-Language Models Get Lost in Attention (ICML 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
