Research map / Vision-language and multimodal / Multimodal LLMs

Catching the Details: Self-Distilled RoI Predictors for Fine-Grained MLLM Perception

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SD-RPN, an annotation-free self-distillation method that turns denoised middle-layer MLLM attention maps into pseudo-RoI labels and trains a lightweight region proposal network on frozen MLLM layers to crop high-resolution regions for fine-grained perception.

Problem

Fine-grained perception in MLLMs needs high-resolution detail, but existing RoI methods either need large annotated data or rely on training-free attention that is noisy and requires multi-pass prefill or slow autoregressive decoding.

Approach

Denoise the MLLM's response-to-image attention (removing sink tokens by feature norm, selective fg/bg label assignment with a minimal enclosing box) to create sparse pseudo-RoI labels, then train an RPN built from the frozen MLLM's first B layers plus R trainable transformer blocks to predict the RoI map in a single partial forward pass.

Key results

• On LLaVA-1.5-7B, SD-RPN raises the five-benchmark Document/OCR average from 27.5 to 34.6 and TextVQA from 46.1 to 58.8.
• On V* Bench, SD-RPN improves LLaVA-1.5-7B overall from 50.3 to 70.7 (+20.4) using the 152K training set.
• On Qwen2.5-VL-7B, SD-RPN lifts the Document/OCR average from 81.5 to 84.5 and V* overall from 78.0 to 89.5.

Surprising findings

• Pseudo-labels generated from the model's own self-predicted responses outperform those from ground-truth SFT responses (62.6 vs 60.7 average), which the authors attribute to representational consistency.
• Raw attention regression with MSE gives only modest gains, and simple binary thresholding at equal fg/bg thresholds performs worse than label assignment with ignored ambiguous tokens.

Limitations

• Throughput drops relative to the baseline (0.40x to 0.62x) due to the two-stage inference and extra visual tokens.
• Failure cases include incomplete activation of the RoI, localization errors, and spatial relationship errors when crops are processed separately.
• Gains on Qwen2.5-VL are smaller on some benchmarks and the method relies on a fixed threshold and hyperparameters tuned on the validation setting.

Experimental setup

Datasets: GQA OCR-VQA TextVQA (training-free attention analysis)

Benchmarks: DocVQA ChartQA OCRBench InfoVQA TextVQA V* Bench POPE HR-Bench 4K HR-Bench 8K

Metrics: accuracy F1 throughput

Baselines: S^2 ViCrop DeepEyes Thyme Grounding-DINO

Builds on: ViCrop TwigVLM S^2 (scaling on scales)

Closest papers by meaning

• Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perception (ICML 2026 · similarity 0.91)
• Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models (AAAI 2025 · similarity 0.90)
• Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Models (CVPR 2026 · similarity 0.89)
• MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs (ICLR 2025 · similarity 0.89)
• Where MLLMs Attend and What They Rely On: Explaining Autoregressive Token Generation (CVPR 2026 · similarity 0.89)
• GranViT: A Fine-Grained Vision Model With Autoregressive Perception For MLLMs (ICLR 2026 · similarity 0.88)