Research map / Reinforcement learning / RL from feedback
Co-Reinforcement Learning for Unified Multimodal Understanding and Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoRL, a two-stage GRPO-based reinforcement learning framework (unified RL then task-specific refined RL) that jointly improves understanding and generation in unified multimodal LLMs, yielding ULM-R1 built on Janus-Pro-1B.
Problem
Reinforcement learning with GRPO has been applied mainly to multimodal reasoning, and its use for jointly improving visual generation and understanding in unified multimodal LLMs (ULMs) remains under-explored.
Approach
A Foundation-then-Specialization two-stage RL: first a unified GRPO stage with shared group ranking over generation rewards (bidirectional cycle consistency with LPIPS and SPICE, plus a token-level text-image matching reward computed by the ULM itself) and understanding rewards (accuracy and format); then a refined stage with task-specific rewards and KL regularization.
Key results
- ULM-R1 (1.5B) reaches 0.77 GenEval and 83.92 DPG, improving over Janus-Pro-1B baseline by about 4 GenEval points and surpassing SD3-Medium's 0.74 GenEval.
- ULM-R1 gains +15.2 on WeMath and +10.6 on LogicVista over its Janus-Pro-1B base, reaching 21.1 and 34.5.
- CoRL applied to Janus-Pro-7B with LoRA improves GenEval from 0.80 to 0.82 and MMMU from 41.0 to 44.6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply CoRL instead of the Janus-Pro baseline | Janus-Pro-1B · GenEval | overall: 0.73 -> 0.77 | Helped |
| Apply CoRL instead of the Janus-Pro baseline | Janus-Pro-1B · WeMath | accuracy: +15.2 | Helped |
| Apply CoRL instead of the Janus-Pro baseline | Janus-Pro-1B · LogicVista | accuracy: +10.6 | Helped |
| Use unified RL rather than cold SFT (ablation) | Janus-Pro-1B · GenEval | overall: 72.8 -> 75.9 | Helped |
| Use unified RL rather than cold SFT (ablation) | Janus-Pro-1B · WeMath | accuracy: 18.0 -> 14.0 | Helped |
| Add refined RL after unified RL (ablation) | Janus-Pro-1B · GenEval | overall: 75.9 -> 77.3 | Helped |
| Use unified RL before task-specific refinement instead of cold SFT (ablation) | Janus-Pro-1B · WeMath | accuracy: 22.5 -> 21.1 | Hurt |
| Add cycle consistency reward (ablation) | Janus-Pro-1B text-to-image generation · GenEval and DPG-Bench | average score: 77.8 -> 79.9 | Helped |
| Add text-image matching reward (ablation) | Janus-Pro-1B text-to-image generation · GenEval and DPG-Bench | average score: 77.8 -> 78.6 | Helped |
| Use text-image matching rather than CLIP reward with cycle consistency (ablation) | Janus-Pro-1B text-to-image generation · GenEval and DPG-Bench | average score: 80.2 -> 80.6 | Helped |
| Use LPIPS rather than other visual consistency measures (ablation) | Janus-Pro-1B text-to-image generation · DPG-Bench | overall: 83.9 | Helped |
| Increase reward balance coefficient from 0.8 to 0.9 (ablation) | Janus-Pro-1B CoRL · GenEval | overall: 77.3 -> 77.1 | Hurt |
Surprising findings
- Direct task-specific RL fails to give expected gains and can even impair the other task, particularly for visual generation.
- Cold SFT on the same data has minimal impact on visual generation, while unified RL improves both generation and understanding.
Limitations
- Notable performance gap remains between generation and understanding in ULMs.
- Understanding rewards are relatively simple and primary.
- Gains on complex reasoning do not scale linearly with model size (Janus-Pro-7B with LoRA gains less on WeMath).
Experimental setup
Datasets: COCO 2017 A-OKVQA GPT-VQA MCoT-Instruct M3CoT SQA-IMG ArxivQA TabMWP MAVIS-Instruct GeomVerse R-CoT
Benchmarks: GenEval WISE DPG-Bench MMStar MMMU WeMath MMVet POPE LogicVista MathVista MathVerse MathVision
Metrics: accuracy GenEval overall score WISE overall score DPG overall score
Baselines: Janus-Pro Janus SD3-Medium Emu3 Emu3-DPO HermesFlow SimpleAR Harmon ILLUME+ Show-o
Builds on: GRPO DeepSeek-R1 Janus-Pro SimpleAR DPO
Closest papers by meaning
- Towards Unified Multimodal Interleaved Generation via Group Relative Policy Optimization (NeurIPS 2025 · similarity 0.89)
- ParaUni: Enhance Generation in Unified Multimodal Model with Reinforcement-driven Hierarchical Parallel Information Interaction (CVPR 2026 · similarity 0.88)
- MSRL: Scaling Generative Multimodal Reward Modeling via Multi-Stage Reinforcement Learning (CVPR 2026 · similarity 0.88)
- GenRL: Multimodal-foundation world models for generalization in embodied agents (NeurIPS 2024 · similarity 0.87)
- Learning to Generate via Understanding: Understanding-Driven Intrinsic Rewarding for Unified Multimodal Models (CVPR 2026 · similarity 0.87)
- SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.