AI Paper Atlas

Research map / Vision-language and multimodal / Multimodal LLMs

LLaVA-UHD v2: Exploiting Hierarchical Vision Granularity in MLLMs via Inverse Semantic Pyramid

Yipeng Zhang, Yifan Liu, Zonghao Guo, Yidan Zhang, Xuesong Yang, Xiaoying Zhang, Chi Chen, Jun Song, Yuan Yao, Tat-Seng Chua, Maosong Sun

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 1 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes the Hiwin (hierarchical window) transformer, a plug-and-play visual encoder enhancement for MLLMs that exploits hierarchical visual semantic granularity to improve fine-grained perception.

Problem

Vision transformers used as visual encoders in multimodal LLMs perform poorly on fine-grained visual perception tasks, which is attributed to their limited ability to capture diverse visual semantic levels.

Approach

Introduces a Hierarchical window (Hiwin) transformer with an inverse semantic pyramid that builds multi-granularity visual features from a ViT, offered as a plug-and-play module for MLLMs.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add VDIM to build inverse semantic pyramid (ablation)LLaVA-UHD · Average of 15 benchmarksscore: 58.0 -> 60.0Improved
add VDIM (ablation)LLaVA-UHD · OCR-Benchscore: 40.9 -> 50.4Improved
add VDIM (ablation)LLaVA-UHD · HR-Benchscore: 45.6 -> 43.9Hurt
add hierarchical window attention on top of VDIM (ablation)LLaVA-UHD+VDIM · Average of 15 benchmarksscore: 60.0 -> 61.7Improved
add hierarchical window attention on top of VDIM (ablation)LLaVA-UHD+VDIM · HR-Benchscore: 43.9 -> 49.0Improved
add hierarchical window attention on top of VDIM (ablation)LLaVA-UHD+VDIM · ScienceQAscore: 70.5 -> 69.8Hurt
add hierarchical window attention on top of VDIM (ablation)LLaVA-UHD+VDIM · OCR-Benchscore: 50.4 -> 50.1No clear effect
add full Hiwin (VDIM + attention) (ablation)LLaVA-UHD · MMMU-valscore: 37.0 -> 37.6No clear effect
add full Hiwin (VDIM + attention) (ablation)LLaVA-UHD · RefCOCO/g/+ (REC avg)accuracy: 68.3 -> 74.0Improved
construct pyramid with VDIM instead of bilinear interpolation (ablation)LLaVA-UHD · Pyramid construction, average of 8 benchmarksscore: 62.0 -> 63.0Improved
use CLIP-ConvNeXt multi-stage feature pyramid (ablation)LLaVA-UHD · Pyramid construction, average of 8 benchmarksscore: 59.9 -> 59.7Hurt
use Hiwin attention instead of perceiver resampler to compress pyramid (ablation)Hiwin-LLaVA-UHD · pre-training lossloss: 0.6110 -> 0.4368Improved

Experimental setup

Builds on: LLaVA-UHD

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy