Research map / Vision-language and multimodal / Multimodal LLMs

GranViT: A Fine-Grained Vision Model With Autoregressive Perception For MLLMs

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GranViT, a ViT trained with region-level autoregressive objectives (Bbox2Caption and Caption2Bbox) plus localized self-distillation on a new 29M-image Gran-29M dataset, yielding a fine-grained vision encoder for MLLMs.

Problem

Existing vision encoders for MLLMs emphasize global image features and lack fine-grained regional perception, due to scarce region-level annotated data and no fine-grained pretraining paradigm aligned with LLMs.

Approach

Two-stage pretraining-adaptation: stage 1 trains the vision encoder and projector with a frozen lightweight LLM on bbox-to-caption regression plus a self-distillation loss aligning ROI-pooled student features with an EMA teacher's crop features; stage 2 freezes the encoder and tunes the LLM on caption-to-bbox regression so the encoder transfers across LLM sizes.

Key results

• GranViT averages 80.78 on fine-grained benchmarks vs 77.95 for the second-best (SAILViT) in the low-resolution setting, and 55.97 on OCR vs 53.33.
• With Qwen2.5-7B, GranViT reaches 67.47 average vs 67.11 for SAILViT, with DocVQA 73.14 vs 64.13.
• Under a frozen MLLM during SFT, GranViT scores 75.16 fine-grained vs 71.90 for SAILViT.

Surprising findings

• GranViT trails SigLIP2 on multimodal reasoning (average 50.60 vs 51.09) and the paper attributes this to limited reasoning-focused training.
• Tuning the vision encoder during stage 2 gives negligible gains over freezing it (fine-grained 77.17 tunable vs 77.24 frozen).

Limitations

• Reasoning benchmarks slightly lag the best baseline (e.g., MMMU 38.00 vs 42.00 for SigLIP2 in the low-resolution table).
• Annotations for natural images are generated by ViTDet and Qwen2.5-VL-7B, so label quality depends on those automatic tools.
• Compute uses 128 Ascend 910B NPUs, which is substantial and limits reproduction.

Experimental setup

Datasets: Gran-29M UMG-41M LAION FLICKR30k PaddleOCR-annotated OCR datasets BLIP-LAION-CC-SBU-558K Open-LLaVA-NeXT 1M LLaVA-One-Vision

Benchmarks: RefCOCO RefCOCO+ RefCOCOg BLINK MMVP MMBench MMStar HallusionBench GQA SEEDBench MMMU MathVista

Metrics: accuracy ACC@IoU0.5 ROUGE-L average score

Baselines: CLIP SigLIP SigLIP2 AIMv2 InternViT SAILViT

Builds on: SigLIP2 SAILViT LLaVA-NeXT Qwen2.5-VL ViTDet

Closest papers by meaning

• Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.91)
• Analyzing and Boosting the Power of Fine-Grained Visual Recognition for Multi-modal Large Language Models (ICLR 2025 · similarity 0.89)
• Improving fine-grained understanding in image-text pre-training (ICML 2024 · similarity 0.89)
• Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignment (ICLR 2025 · similarity 0.89)
• Fine-R1: Make Multi-modal LLMs Excel in Fine-Grained Visual Recognition by Chain-of-Thought Reasoning (ICLR 2026 · similarity 0.89)
• Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perception (ICML 2026 · similarity 0.89)