Research map / Image generation / Autoregressive and latent generation
PixelDiT2: Representation-Grounded Pixel Diffusion Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PixelDiT2, an end-to-end pixel diffusion transformer that conditions on spatial tokens from a frozen DINOv3 encoder applied to the noisy input at every step (via a timestep-conditioned projection and spatial AdaLN), reaching FID 1.46 on ImageNet-256 and 1.48 on ImageNet-512.
Problem
Pixel-space diffusion converges slowly and lags latent diffusion in final image quality, possibly because the denoiser must learn both denoising-friendly representations and pixel generation jointly.
Approach
Keep a pretrained vision encoder (DINOv3) frozen and run it on the current noisy image at every denoising step; a timestep-conditioned projection maps its patch tokens to noise-level-aware grounding tokens that modulate the pixel diffusion transformer via spatial AdaLN, retained at inference, alongside REPA loss.
Key results
- PixelDiT2-H/16 reaches FID 1.46 (IS 301.6) on ImageNet-256 at 600 epochs, and FID 1.48 on ImageNet-512 at 680 epochs, versus PixelDiT-XL's 1.81 at 850 epochs.
- In a matched DINOv3-S encoder study at 320 epochs, combining REPA and grounding gives FID 1.70 vs 1.75 for REPA only and 2.08 for grounding only.
- Replacing the DiT-style projection P_g with a parameter-matched linear projection worsens B/16 FID from 5.29 to 6.79 at epoch 200.
Surprising findings
- Scaling the grounding encoder from DINOv3-S to B or L worsens ImageNet-256 FID (1.547 and 1.552 vs 1.462), contrary to expectation.
- Frozen encoder adaptation variants (joint or two-stage) all underperform the off-the-shelf frozen encoder.
Limitations
- Does not close the full gap to the best latent baselines (e.g., RAE-XL, REPA-E).
- Running a frozen encoder at every sampling step adds inference cost.
- Evaluated only on class-conditional ImageNet; text-to-image left to future work.
Experimental setup
Datasets: ImageNet
Benchmarks: ImageNet-256x256 ImageNet-512x512
Metrics: FID-50K Inception Score GFLOPs
Baselines: PixelDiT-XL JiT-G/16 Latent Forcing REPA RAE-XL SiT-XL DiT-XL VAR-d30 MAR-H xAR-H
Builds on: PixelDiT JiT REPA DiT Latent Forcing
Closest papers by meaning
- Simpler Diffusion: 1.5 FID on ImageNet512 with Pixel-space Diffusion (CVPR 2025 · similarity 0.93)
- PixelART: Image-to-Layer Decomposition without Latents or Text-to-Image Pretraining (NeurIPS 2026 · similarity 0.91)
- Representation Entanglement for Generation: Training Diffusion Transformers Is Much Easier Than You Think (NeurIPS 2025 · similarity 0.90)
- L2P: Unlocking Latent Potential for Pixel Generation (NeurIPS 2026 · similarity 0.90)
- Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think (ICLR 2025 · similarity 0.89)
- OSCAR: One-Step Diffusion Codec Across Multiple Bit-rates (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.