Research map / Vision-language and multimodal / Multimodal LLMs
Janus-Pro-R1: Advancing Collaborative Visual Comprehension and Generation via Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Janus-Pro-R1, a two-stage SFT-then-GRPO training that teaches Janus-Pro to iteratively self-check and regenerate images (an "aha" introspective CoT for text-to-image), extended to image editing.
Problem
Visual comprehension and generation in unified MLLMs remain independent, so comprehension does not improve generation and LLM reasoning is not used for image generation. The paper targets introspective, self-correcting text-to-image generation and unified generation such as editing.
Approach
Mixed SFT on three sub-tasks (text-to-image, self-evaluation of consistency with rationale, regeneration after self-reflection) followed by GRPO with bi-level QA-based rewards that score both final image quality and the model's self-assessment accuracy, letting the model interleave image generation and text judgment in a self-driven loop.
Key results
- On GenEval, Janus-Pro-R1 (7B, with aha) reaches 0.86 overall vs 0.80 for Janus-Pro-7B and 0.85 for GPT-4o.
- On T2I-CompBench color, Janus-Pro-R1 reaches 83.4 vs 63.6 for Janus-Pro-7B and 81.3 for T2I-R1.
- On PIE-Bench, Janus-Pro-R1-Edit obtains structure distance 35.87 and CLIP whole 24.78, better than EditAR (39.43, 24.87) and Janus-Pro-Edit (49.44, 24.16).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the Aha introspection process (ablation) | Janus-Pro-R1 on GenEval · GenEval | Overall: 0.83 -> 0.86 | Helped |
| add the Aha introspection process (ablation) | Janus-Pro-R1 on T2I-CompBench · T2I-CompBench | Color: 81.2 -> 83.4 | Helped |
| add the Aha introspection process (ablation) | Janus-Pro-R1 on T2I-CompBench · T2I-CompBench | Shape: 56.3 -> 59.4 | Helped |
| add the Aha introspection process (ablation) | Janus-Pro-R1 on T2I-CompBench · T2I-CompBench | Texture: 73.3 -> 75.2 | Helped |
| add the Aha introspection process (ablation) | Janus-Pro-R1 on DPG-Bench · DPG-Bench | Average: 85.02 -> 85.57 | Helped |
| use RL after SFT (ablation) | Janus-Pro on GenEval · GenEval | Overall: 0.81 -> 0.86 | Helped |
| combine Task-I text-to-image SFT with Task-II and Task-III regeneration SFT (ablation) | Janus-Pro-SFT on GenEval · GenEval | Overall: 0.79 -> 0.81 | Helped |
| use the 1B backbone instead of the 7B backbone for Janus-Pro-R1 (ablation) | Janus-Pro-R1 on GenEval · GenEval | Overall: 0.73 -> 0.71 | Hurt |
| lower positive-image selection threshold to >0.5 for all SFT subtasks (ablation) | Janus-Pro on text-to-image generation · GenEval, T2I-CompBench, and DPG-Bench | text-to-image performance | Hurt |
| use Janus-Pro-R1 instead of InternVL2.5-8B as the reward model | Janus-Pro-7B and Janus-1.3B text-to-image RL · GenEval | overall performance | Helped |
Surprising findings
- SFT alone yields only marginal gains over Janus-Pro-7B (0.81 vs 0.80 on GenEval), and regeneration after SFT does not improve first-round images; RL is needed for gains.
- At 1B scale, the aha-moment training does not improve over the backbone and even lowers the first-round image quality, suggesting a model-scale requirement.
Limitations
- Training data contains few counting prompts, so counting performance on GenEval lags SOTA models.
- Image editing data has poor aesthetic quality, so some edited outputs look less natural.
- Experiments are limited to simple generation and semantic evaluation due to compute; claims about unified comprehension and generation are preliminary.
Experimental setup
Datasets: GenEval T2I-CompBench DPG-Bench PIE-Bench FLUX-generated and Janus-Pro-generated images with Qwen-expanded prompts public image editing datasets (unnamed in text)
Benchmarks: GenEval T2I-CompBench DPG-Bench PIE-Bench
Metrics: GenEval overall score T2I-CompBench BLIP-VQA color/shape/texture DPG-Bench score PIE-Bench structure distance, PSNR, LPIPS, MSE, SSIM, CLIP s consistency ratio with GenEval standard evaluation GPT-4o reason reliability score
Baselines: Janus-Pro FLUX.1-dev SD3 DALL-E 3 GPT-4o Show-o Emu3 Infinity T2I-R1 MINT GOT EditAR
Builds on: Janus-Pro GRPO DeepSeekMath-style group relative policy optimization T2I-R1 InternVL2.5
Closest papers by meaning
- T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoT (NeurIPS 2025 · similarity 0.91)
- WEAVE: Unleashing and Benchmarking the In-context Interleaved Comprehension and Generation (CVPR 2026 · similarity 0.90)
- GoT-R1: Unleashing Reasoning Capability of Autoregressive Visual Generation with Reinforcement Learning (ICLR 2026 · similarity 0.90)
- ThinkGen: Generalized Thinking for Visual Generation (CVPR 2026 · similarity 0.90)
- Learning to Generate via Understanding: Understanding-Driven Intrinsic Rewarding for Unified Multimodal Models (CVPR 2026 · similarity 0.90)
- Thinking-while-Generating: Interleaving Textual Reasoning throughout Visual Generation (CVPR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.