Research map / Reinforcement learning / RL from feedback

Sketch Then Paint: Hierarchical Reinforcement Learning for Diffusion Multi-Modal Large Language Models

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: RL for LLM reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes HT-GRPO, a hierarchical token RL method for diffusion multimodal LLMs that uses a Global→Structure→Refinement inner-loop schedule, a fully-masked prompt-conditioned importance-ratio estimator, and generation-order-aware token credit weights, improving MMaDA and Lumina-DiMOO on GenEval and DPG-Bench.

Problem

RL optimization of diffusion multimodal LLMs for text-to-image generation is hard because many unmasking orders yield the same image, making importance ratios intractable, and existing methods ignore the hierarchical (layout-then-detail) generation process and assign uniform image-level rewards to a

Approach

Organize GRPO inner-loop updates into coarse-to-fine stages keyed by generation-order rank, compute importance ratios under a fully masked prompt-only context so no trajectory caching is needed, and reweight the group advantage per token so structural (early) tokens get more credit than refinement (late) tokens.

Key results

• On MMaDA, HT-GRPO raises GenEval overall from 0.56 to 0.83 (vs MaskGRPO 0.80, TraceGRPO 0.79) and DPG-Bench from 70.51 to 81.09.
• On Lumina-DiMOO, HT-GRPO reaches GenEval 0.92 and DPG-Bench 84.47, above MaskGRPO (0.88 / 82.24) and TraceGRPO (0.87 / 82.62).
• Ablation on MMaDA: removing hierarchical credit weighting drops GenEval-style score to 80.8 from 83.3 and replacing the fully masked context with revealed structural context drops it to 80.6.

Surprising findings

• TraceGRPO underperforms MaskGRPO on MMaDA for DPG-Bench (74.37 vs 75.81), despite trajectory-consistent contexts.

Limitations

• Uses a fixed structure ratio and fixed credit weights that may not adapt to prompt complexity or rollout uncertainty.
• Evaluated only on two dMLLM backbones (MMaDA and Lumina-DiMOO).

Experimental setup

Datasets: GenEval prompts DPG-Bench prompts

Benchmarks: GenEval DPG-Bench

Metrics: overall score DPG overall DeQA ImageReward HPSv3 UniPercept aesthetics UniPercept quality UniPercept structure

Baselines: MaskGRPO TraceGRPO UniGRPO SDXL DALL-E 3 Janus-Pro FLUX.1-dev

Builds on: GRPO MaskGRPO TraceRL UniGRPO

Closest papers by meaning

• Delving into RL for Image Generation with CoT: A Study on DPO vs. GRPO (NeurIPS 2025 · similarity 0.90)
• Reinforcement Learning Meets Masked Generative Models: Mask-GRPO for Text-to-Image Generation (NeurIPS 2025 · similarity 0.90)
• MMaDA: Multimodal Large Diffusion Language Models (NeurIPS 2025 · similarity 0.89)
• Towards Better Alignment: Training Diffusion Models with Reinforcement Learning Against Sparse Rewards (CVPR 2025 · similarity 0.89)
• Breaking the SFT Plateau: Multimodal Structured Reinforcement Learning for Chart-to-Code Generation (ICLR 2026 · similarity 0.89)
• Training Diffusion Models with Reinforcement Learning (ICLR 2024 · similarity 0.88)