Research map / Vision-language and multimodal / Multimodal LLMs

ERGO: Efficient High-Resolution Visual Understanding for Vision-Language Models

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ERGO, an RL-trained coarse-to-fine high-resolution VQA model that uses a region-verification reward and box-area reward to learn reasoning-driven perception on downsampled images, cutting vision tokens while improving accuracy.

Problem

High-resolution LVLM inference is costly due to many vision tokens; coarse-to-fine pipelines fail when downsampled images hide the target object, since existing perception-driven "thinking with images" models need clear visual cues to localize regions.

Approach

A TCE reward combines a region-verification reward (a frozen reward model answers the query from the cropped region alone, without the original image) with a box-area step reward (crop must cover at most 60% of the image), plus accuracy and format rewards under GRPO, so the policy learns to crop contextually relevant regions even when the target is indiscernible.

Key results

• At 640x28x28 pixel constraint, ERGO reaches 81.7 on V* vs 56.5 for Qwen2.5-VL-7B and an average of 55.2 vs 40.3 across six high-res benchmarks.
• At 1280x28x28, ERGO averages 58.4 vs 52.6 for MGPO and 52.4 for the base Qwen2.5-VL-7B (original 16384 setting), with 42.7 on VisualProbe vs 15.2 base.
• Under vLLM on a single H100, ERGO runs at 2.6 s latency at 1280x28x28 versus 9.0-9.4 s for Mini-o3 and DeepEyes in original settings (abstract claims 3x speedup, 23% of vision tokens on V*).

Surprising findings

• Appending the ground-truth full-resolution sub-image to a low-resolution input recovers V* accuracy to 77.0, matching full-resolution performance.
• Linearly annealing the box-area constant degrades performance (51.0 vs 58.4), as the policy shrinks regions instead of improving region verification.

Limitations

• Relies on a large frozen reward model (Qwen2.5-VL-72B) for the region-verification reward, adding training cost.
• Fixed area threshold gamma=0.6 chosen from dataset statistics; the box constraint is a hand-set heuristic.
• Trained for a single tool call; multi-turn or multi-region settings are not the focus.

Experimental setup

Datasets: ArxivQA V* training set DeepEyes curated data TreeVGR VisCoT VGR

Benchmarks: V* Bench HR-Bench 4K HR-Bench 8K MME-RealWorld Lite TreeBench VisualProbe CV-Bench MMVP HallusionBench POPE MMBench AI2D

Metrics: accuracy average accuracy vision token count latency (s) target coverage score

Baselines: Qwen2.5-VL-7B-Instruct PixelReasoner DeepEyes TreeVGR MiniO3 MGPO VisionThink

Builds on: GRPO DeepEyes Qwen2.5-VL MGPO VisionThink

Closest papers by meaning

• Dr. Seg: Revisiting GRPO Training for Visual Large Language Models through Perception-Oriented Design (CVPR 2026 · similarity 0.90)
• Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.89)
• From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models (ICML 2026 · similarity 0.89)
• More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.89)
• DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learning (ICLR 2026 · similarity 0.89)
• Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.89)