Research map / Vision-language and multimodal / Human and embodied modeling

DynRefer: Delving into Region-level Multimodal Tasks via Dynamic Resolution

CVPR 2025 · Vision-language and multimodal › Human and embodied modeling · approach: Region-level vision-language alignment · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DynRefer, which trains region-level multimodal models on stochastically sampled nested views of varying resolution and selects views at inference by task or image priors, achieving SOTA on several region-level tasks with one model.

Problem

Region-level multimodal tasks (captioning, open-vocabulary attribute detection, region recognition) use fixed-resolution region inputs, which cannot adapt resolution and context to different task needs.

Approach

Nest multiple crops around the referred region at different interpolation coefficients, stochastically sample n views during training to align with multi-task language outputs, and at inference select views via task priors (known task) or a greedy perceptual-hash information search (unknown task).

Key results

• Achieves 115.7 CIDEr on RefCOCOg and 190.9 CIDEr on VG region captioning with a 4.2B model, outperforming 7.4B methods like GLaMM and Osprey.
• Reaches 29.2% mAP on OVAD attribute detection, improving over prior SOTA.
• Gets 89.2% accuracy on COCO open-vocabulary region recognition with EVA backbone, up to 8.8% over prior methods.

Measured effects reported in the paper

Surprising findings

• Increasing the number of nested views to 4 degrades performance, attributed to a harder-to-optimize region representation manifold.
• Simply increasing input image resolution yields limited gains relative to its large FLOPs cost.

Limitations

• Performance peaks at 3 views; 4 views degrade results.
• Image-prior view selection relies on a heuristic perceptual-hash information measure and greedy search.

Experimental setup

Datasets: Visual Genome VG-COCO RefCOCOg COCO OVAD

Benchmarks: RefCOCOg Visual Genome V1.2 VG V1.0 VG-COCO OVAD COCO-2017 val

Metrics: METEOR CIDEr mAP accuracy

Baselines: GRiT GLaMM Osprey RegionGPT Alpha-CLIP GPT4RoI ControlCap Kosmos-2 OVAD-Baseline BLIP2 DetCLIPv3 CapDet

Builds on: BLIP-2 GRiT Control captioning (ControlCap) SigLIP sigmoid loss RoI-Align

Closest papers by meaning

• Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perception (ICML 2026 · similarity 0.89)
• Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models (AAAI 2025 · similarity 0.88)
• VLM-R³: Region Recognition, Reasoning, and Refinement for Enhanced Multimodal Chain-of-Thought (NeurIPS 2025 · similarity 0.87)
• Seeing What Matters: A Training-Free Self-Guided Framework for Multimodal Detail Perception and Reasoning (CVPR 2026 · similarity 0.87)
• CVSearch: Empowering Multimodal LLMs with Cognitive Visual Search for High-Resolution Image Perception (ICML 2026 · similarity 0.87)
• Res-Bench: Benchmarking the Robustness of Multimodal Large Language Models to Dynamic Resolution Input (AAAI 2026 · similarity 0.87)