Research map / Image generation / Text-to-image diffusion

RankE: End-to-End Post-Training for Discrete Text-to-Image Generation with Decoder Co-Evolution

NeurIPS 2026 · Image generation › Text-to-image diffusion · approach: T2I evaluation and controllable adapters · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RankE, an alternating end-to-end post-training method that co-evolves the AR policy and VQ decoder of discrete text-to-image models to counter a "latent covariate shift" that frozen-decoder RL induces.

Problem

Post-training of discrete autoregressive text-to-image models updates only the AR policy while the VQ decoder stays frozen, creating a distribution mismatch (latent covariate shift) that improves reward/CLIP but degrades FID.

Approach

Alternate between a GRPO token-level policy update and a decoder update that combines reward-weighted adversarial loss (Rank-GAN), differentiable reward backprop, reconstruction on ground-truth codes, and EMA-teacher LPIPS consistency, so the decoder tracks the policy's token distribution.

Key results

• On LlamaGen-XL (775M) with CLIP reward, RankE reaches FID 15.21 and CLIP 33.76 on MS-COCO 30K, versus 17.76 FID and 32.45 CLIP for standard frozen-decoder GRPO.
• On Janus-Pro-1B, RankE gives CLIP 33.86 and FID 25.19 versus Std. RL 33.60 and 25.59, with best average GenEval 0.750.
• With HPSv2 reward on LlamaGen-XL, RankE raises HPSv2 average from 0.2451 (Std. RL) to 0.2531.

Surprising findings

• Decoder-only adaptation raises CLIP to 33.41 without any policy signal, though FID degrades to 18.68.
• Removing the consistency loss raises CLIP to 34.17 but worsens FID to 19.03, showing a CLIP-FID asymmetry.

Limitations

• Higher peak VRAM (33 GB to 56 GB) from holding discriminator and EMA decoder.
• Gains depend on alignment between SFT corpus and backbone pretraining; on Janus-Pro RankE does not beat Base FID.
• VQ encoder kept frozen, so joint encoder training is not explored.

Experimental setup

Datasets: BLIP3o-60k MS-COCO 2014 validation

Benchmarks: MS-COCO 30K HPSv2 GenEval

Metrics: FID CLIP score HPSv2 score GenEval accuracy KL divergence of token distribution codebook entropy

Baselines: LlamaGen-SFT Std. RL (GRPO, frozen decoder) Base Decoder-only Janus-Pro

Builds on: GRPO REPA-E LlamaGen Janus-Pro reward-weighted regression

Closest papers by meaning

• David and Goliath: Small One-step Model Beats Large Diffusion with Score Post-training (ICML 2025 · similarity 0.88)
• Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decoding (ICLR 2025 · similarity 0.87)
• JetFormer: An autoregressive generative model of raw images and text (ICLR 2025 · similarity 0.87)
• When Worse is Better: Navigating the Compression Generation Trade-off In Visual Tokenization (NeurIPS 2025 · similarity 0.87)
• LANTERN: Accelerating Visual Autoregressive Models with Relaxed Speculative Decoding (ICLR 2025 · similarity 0.86)
• Speculative Jacobi-Denoising Decoding for Accelerating Autoregressive Text-to-image Generation (NeurIPS 2025 · similarity 0.86)