Research map / Vision-language and multimodal / Multimodal LLMs

CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learning

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Applies RLVR (GRPO) to dense image captioning, rewarding captions by how well a vision-free LLM answers MCQs from the caption, and uses the resulting CapRL-3B to annotate a 5M-image caption dataset for LVLM pretraining.

Problem

Image captioning is trained with SFT on expensive, non-scalable annotations that encourage memorizing single ground-truth captions; applying RLVR to subjective captioning lacks an objective reward, and LVLM-as-judge or reward-model rewards are prone to hacking.

Approach

Define caption quality by utility: the reward is the accuracy of a separate, vision-free LLM (Qwen2.5-3B-Instruct) answering curated multiple-choice questions using only the generated caption, averaged over N shuffled samples, used with GRPO on a Qwen2.5-VL-3B policy.

Key results

• In the pretraining setting, CapRL-5M reaches 62.0 average vs 57.1 for DenseFusion-1M with Qwen2.5-3B + Qwen2.5-ViT, and CapRL-1M exceeds DenseFusion-1M on InfoVQA by 6.8 points.
• In the Prism setting, CapRL-3B averages 48.3, matching Qwen2.5-VL-72B (48.3) and well above Qwen2.5-VL-3B (39.9).
• CapRL-3B surpasses Qwen2.5-VL-3B by 14.9, 12.8, and 24.6 points on ChartQA, ChartQAPro, and InfoVQA respectively.

Surprising findings

• Using Qwen2.5-VL-3B as a LVLM judge drove the policy toward long irrelevant text, while UnifiedReward-as-judge caused captions to collapse to short outputs.
• A single QA pair per image already gives about 7.4% average gain over baseline, only 0.5% below using two QA pairs.

Limitations

• Reward depends on quality of MCQs generated by Qwen2.5-VL-72B and filtered by Qwen2.5-VL-3B, which may leak or be biased.
• Evaluation of captions relies on Prism framework with a fixed fine-tuned Qwen2.5-3B answering LLM, so it measures informativeness rather than all caption qualities.
• Experiments focus on a 3B policy and limited pretraining setups (three architectures).

Experimental setup

Datasets: ShareGPT4V-1M DenseFusion-1M BLIP-558K Open-LLaVA-NeXT-1M web-collected images

Benchmarks: InfoVQA DocVQA ChartQA RealWorldQA MathVista SEED2Plus MME-RealWorld MMBench MMStar MMVet AI2D GQA

Metrics: accuracy average accuracy

Baselines: Qwen2.5-VL-3B Qwen2.5-VL-7B Qwen2.5-VL-72B UnifiedRW-as-Judge-3B Qwen2.5VL-as-Judge-3B ShareGPT4V-1M DenseFusion-1M Vanilla

Builds on: GRPO DeepSeek-R1 ShareGPT4V Prism Framework DenseFusion

Closest papers by meaning

• BalCapRL : A Balanced Framework for RL-Based MLLM Image Captioning (NeurIPS 2026 · similarity 0.95)
• RePIC: Reinforced Post-Training for Personalizing Multi-Modal Language Models (NeurIPS 2025 · similarity 0.91)
• ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMs (NeurIPS 2025 · similarity 0.89)
• CapsFusion: Rethinking Image-Text Data at Scale (CVPR 2024 · similarity 0.89)
• Activating Visual Context and Commonsense Reasoning Through Masked Prediction in VLMs (AAAI 2026 · similarity 0.89)
• Learning by Correction: Efficient Tuning Task for Zero-Shot Generative Vision-Language Reasoning (CVPR 2024 · similarity 0.88)