Research map / Vision-language and multimodal / Multimodal LLMs

ViPER: Empowering the Self-Evolution of Visual Perception Abilities in Vision-Language Models

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Visual instruction tuning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ViPER, a self-bootstrapping two-stage RL framework where a Qwen2.5-VL model synthesizes its own perception data (caption self-refining and visual-operation predicting via diffusion-based image reconstruction) and trains on it, yielding Qwen-Viper-3B/7B with small gains on perception benchmarks.

Problem

Fine-grained visual perception is a bottleneck for VLMs; SFT hurts general capability, existing RFT emphasizes textual reasoning over perception, and high-quality perception data is scarce.

Approach

Structure perception learning as a coarse-to-fine two-stage task (caption self-refinement, then visual-operation prediction) where training data is self-synthesized by the VLM plus diffusion models performing image-level and instance-level reconstruction, and used in a GRPO-style RL loop without cold-start.

Key results

• Qwen-Viper-7B averages 66.0 vs 64.4 for Qwen2.5-VL-7B across seven benchmarks (+1.6); Qwen-Viper-3B averages 60.4 vs 58.7 (+1.7).
• On MMStar fine-grained perception subset, Qwen-Viper-7B gains +6.0 points (55.6 to 61.6) and Qwen-Viper-3B gains +4.4 points.
• Fine-tuning Qwen2.5-VL-7B on the ViRL39K dataset via GRPO yields an overall 64.3 vs 64.4, i.e., no gain.

Surprising findings

• Cold-start SFT with 1K Gemini-2.5-Pro CoT samples did not improve final reward, and the no-cold-start run eventually converged slightly higher.
• Training on ViRL39K via GRPO gave no overall improvement over the base Qwen2.5-VL-7B (64.3 vs 64.4).

Limitations

• Gains are modest (average +1.6 to +1.7 points) and no variance or significance reported.
• Data synthesis relies on external diffusion models (Qwen-Image, OmniGen2) and Gemini-2.5-Pro for cold-start ablation, so it is not fully self-contained.
• Fine-grained subdomain analysis uses only MMStar subsets and the paper does not compare against other perception-focused RL methods like DeepEyes.

Experimental setup

Datasets: Viper10K ViRL39K

Benchmarks: MMStar RealWorldQA MME-RW (en) BLINK (val) Mantis Eval HallusionBench CRPE (relation)

Metrics: accuracy absolute gain over base model

Baselines: Qwen2.5-VL-3B Qwen2.5-VL-7B InternVL2.5 InternVL3 GPT-4o Qwen2.5-VL-7B+ViRL39K

Builds on: GRPO DAPO Qwen2.5-VL BGE-M3

Closest papers by meaning

• VICO: Visual Environments Co-Evolving for Vision-Language Model Reasoning (NeurIPS 2026 · similarity 0.88)
• SenseSearch: Empowering Vision-Language Models with High-Resolution Agentic Search-Reasoning via Reinforcement Learning (CVPR 2026 · similarity 0.88)
• Unleashing Perception-Time Scaling to Multimodal Reasoning Models (ICLR 2026 · similarity 0.88)
• Testing the Limits of Fine-Tuning for Improving Visual Cognition in Vision Language Models (ICML 2025 · similarity 0.88)
• CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models (CVPR 2026 · similarity 0.88)
• Agentic Jigsaw Interaction Learning for Enhancing Visual Perception and Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.88)