Research map / Vision-language and multimodal / Multimodal LLMs

Granulon: Awakening Pixel-Level Visual Encoders with Adaptive Multi-Granularity Semantics for MLLM

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Granulon is a DINOv3-based MLLM that uses a text-conditioned granularity Controller and an Adaptive Token Aggregation (AdaTA) module to produce multi-granularity visual tokens, evaluated under identical LLaVA-style settings against CLIP, SigLIP, and DINO encoders.

Problem

Pixel-level self-supervised encoders like DINOv3 capture fine detail but lack coarse semantic abstraction, while CLIP encoders emphasize global semantics and miss fine-grained detail; MLLMs need task-adaptive visual granularity.

Approach

Treat visual granularity as a text-conditioned control dimension: a Controller predicts pooling and cluster-count parameters from the question, and AdaTA performs granularity-guided pooling, attention-aware clustering, and quality-based selection to add semantic tokens alongside DINOv3 pixel tokens.

Key results

• On SEED-Bench with Qwen2.5, Granulon reaches 58.80% Recall, +7.89 over the CLIP baseline.
• On FLUX-Reason with Llama3.2, Granulon reaches 56.67% GPTscore vs 28.97% for CLIP and 19.49% for DINOv2.
• On the reasoning task with Llama3.2, hallucination drops from 61.3% (DINOv3) to 46.3% (Granulon).

Surprising findings

• Coarser semantic-token configurations (fewer clusters) performed better on A-OKVQA, while reasoning benefited from finer granularity.

Limitations

• The vision encoder and LLM are frozen and only the projector is trained, so results may not reflect full fine-tuning.
• Evaluation relies on GPT-4o as judge, with only a manual verification check reported.
• Controller supervision comes from GPT-4o-annotated granularity labels, whose quality and cost are not analyzed in detail.

Experimental setup

Datasets: FLUX-Reason SurgVLM SEED-Bench A-OKVQA CC12M_and_Imagenet21K_Recap

Benchmarks: SEED-Bench A-OKVQA CC12M_and_Imagenet21K_Recap FLUX-Reason SurgVLM

Metrics: Recall GPTscore BERTscore hallucination rate granularity score

Baselines: CLIP SigLIP DINOv2 DINOv3

Builds on: LLaVA DINOv3 DynamicViT EViT

Closest papers by meaning

• MaVEn: An Effective Multi-granularity Hybrid Visual Encoding Framework for Multimodal Large Language Model (NeurIPS 2024 · similarity 0.90)
• LION: Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge (CVPR 2024 · similarity 0.89)
• GranViT: A Fine-Grained Vision Model With Autoregressive Perception For MLLMs (ICLR 2026 · similarity 0.88)
• Lifting the Veil on Visual Information Flow in MLLMs: Unlocking Pathways to Faster Inference (CVPR 2025 · similarity 0.88)
• Multi-step Visual Reasoning with Visual Tokens Scaling and Verification (NeurIPS 2025 · similarity 0.88)
• HyperET: Efficient Training in Hyperbolic Space for Multi-modal Large Language Models (NeurIPS 2025 · similarity 0.88)