Research map / Vision-language and multimodal / Multimodal LLMs

Unified Reinforcement and Imitation Learning for Vision-Language Models

NeurIPS 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Reinforcement learning for visual reasoning · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RIL, a unified RL and adversarial imitation learning method where small VLMs (Qwen2.5-VL-3B/7B, InternVL3-1B/2B/8B) are trained with a discriminator-based binary similarity reward from large teacher VLMs plus an LLM-as-a-Judge answer reward.

Problem

Large VLMs are impractical for resource-constrained devices, and existing RL methods for VLMs rely on answer parsing that does not handle open-ended questions or large architectural changes and long think traces add latency.

Approach

Combine Dr.GRPO with a GAIL-style LLM-based discriminator (initialized from the student) that outputs a binary similarity reward for teacher-likeness, plus an LLM-as-a-Judge binary answer reward, and mix responses from multiple large teachers into training.

Key results

• Qwen2.5-VL-7B with RIL (Both) reaches 79.7 on MathVista and 95.6 on ChartQA, versus 67.8 and 87.3 for the base model, with the Dr.GRPO-only baseline at 69.5 MathVista (per ablation).
• Qwen2.5-VL-3B with RIL gains roughly 10 points on MathVista (61.2 to 71.7) and reaches 95.7 on ChartQA.
• Using responses from both teachers (Both) outperforms a single teacher; e.g., InternVL3-8B with Both reaches 77.8 MathVista vs 74.1 with InternVL3-78B alone.

Measured effects reported in the paper

Surprising findings

• A fixed, pre-trained discriminator (not updated during RIL) already outperforms RL-only baselines, though continuous updating is still needed for the best results.
• Binary similarity reward outperforms continuous and multi-level discretizations of the discriminator score.

Limitations

• Current implementation focuses only on post-instruction-tuning alignment; extension to visual instruction tuning is future work.
• Training cost is higher than GRPO because of teacher, student, discriminator, and judge models.
• Answer reward depends on an LLM judge (Qwen2.5-32B), which may introduce evaluation noise.

Experimental setup

Datasets: COCO-ReM iNaturalist2018 VQA-v2 Super-CLEVR MAVIS Geometry3K AI2D ChartQA DocVQA InfoVQA TextVQA GQA

Benchmarks: AI2D ChartQA MathVista MMBench MMBench-CN MM-Vet MM-Vet-v2 MMMU MMMU-Pro MMStar BLINK SEED

Metrics: accuracy average score

Baselines: GRPO Dr.GRPO LLaVA-OneVision-7B InternVL2-8B MiniCPM-o2.6-8B Ovis2-8B Qwen2.5-VL-72B InternVL3-78B GPT-4o GPT-4.1 Claude-3.5-Sonnet Gemini-2.0-Flash

Builds on: GAIL GRPO Dr.GRPO DeepSeek-R1 MiniLLM DistilLLM LLaVA-KD VLsI

Closest papers by meaning

• VLMimic: Vision Language Models are Visual Imitation Learner for Fine-grained Actions (NeurIPS 2024 · similarity 0.92)
• Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning (ICLR 2024 · similarity 0.90)
• Code as Reward: Empowering Reinforcement Learning with VLMs (ICML 2024 · similarity 0.90)
• VLA-REPLICA: A Low-Cost, Reproducible Benchmark for Real-World Evaluation of Vision-Language-Action Models (NeurIPS 2026 · similarity 0.89)
• ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMs (NeurIPS 2025 · similarity 0.88)
• Bridging Environments and Language with Rendering Functions and Vision-Language Models (ICML 2024 · similarity 0.88)