Research map / Image generation / Autoregressive and latent generation
ADDP: Learning General Representations for Image Recognition and Generation with Alternating Denoising Diffusion Process
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ADDP, a single representation-learning framework that alternates pixel decoding (off-the-shelf VQ decoder) and VQ-token prediction (ViT encoder-decoder) under a masked diffusion ELBO, yielding a backbone usable for both image generation and dense recognition.
Problem
Generative representation learning methods that unify image generation and recognition (iGPT, ViT-VQGAN, MAGE) either work purely in pixel or purely in VQ-token space, and their recognition performance, especially on dense tasks, lags behind pixel-based MIM methods.
Approach
Alternate denoising between raw pixels and VQ tokens: at each step decode pixels from partially masked VQ tokens (paired with sampled unreliable tokens and a WeightedSum mapping), then predict new VQ tokens from those pixels with a learnable ViT encoder-decoder, trained via an ELBO of a masked-token diffusion process.
Key results
- ADDP (ViT-L) reaches FID 7.6 unconditional ImageNet 256 generation, improving on MAGE's 9.1 by 1.5 FID.
- ADDP ViT-L fine-tuning achieves 85.9% ImageNet top-1, about 2 points above MAGE (83.9%) and comparable to recognition-only methods such as MAE (85.9%).
- ADDP ViT-L obtains 54.6 AP box and 54.3 mIoU on COCO and ADE20k, well above MAGE-based results (31.4 AP box, 43.1 mIoU).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ADDP instead of MAGE for unconditional generation | ImageNet-1k 256×256 generation · ImageNet-1k 256×256 validation | FID: 9.1 -> 7.6 | Helped |
| Use ADDP instead of MAGE for classification fine-tuning | ImageNet-1k classification · ImageNet-1k | fine-tuning accuracy: 83.9 -> 85.9 | Helped |
| Use ADDP instead of MAGE for linear probing | ImageNet-1k classification · ImageNet-1k | linear probing accuracy: 78.9 -> 23.8 | Hurt |
| Use ADDP instead of MAGE for COCO object detection | COCO detection · COCO test-dev | box AP: 31.4$^*$ -> 54.6 | Helped |
| Condition unreliable-token prediction on zₜ instead of zₜ₋₁ (ablation) | ADDP · ImageNet-1k 256×256 generation | FID: 80.75 -> 23.63 | Helped |
| Use WeightedSum instead of Sampling for unreliable-token mapping (ablation) | ADDP · ImageNet-1k 256×256 generation | FID: 38.79 -> 23.63 | Helped |
| Use WeightedSum instead of ArgMax for unreliable-token mapping (ablation) | ADDP · ImageNet-1k 256×256 generation | FID: 26.62 -> 23.63 | Helped |
| Feed token inputs through both encoder and decoder (ablation) | ADDP · ImageNet-1k classification | fine-tuning accuracy: 83.5 -> 81.5 | Hurt |
| Remove token inputs to the decoder (ablation) | ADDP · ImageNet-1k 256×256 generation | FID: 23.63 -> 36.24 | Hurt |
| Predict δ(ẑ₀=z₀) instead of q(z̄ₜ₋₁|zₜ) (ablation) | ADDP · ImageNet-1k 256×256 generation | FID: 23.63 -> 39.41 | Hurt |
| Use fixed (50%) masking instead of the default masking strategy (ablation) | ADDP · ImageNet-1k 256×256 generation | FID: 39.41 -> 166.54 | Hurt |
| Reduce top-p sampling from 1.0 to 0.9 (ablation) | ADDP · ImageNet-1k 256×256 generation | FID: 7.6 -> 10.5 | Hurt |
Surprising findings
- Linear probing accuracy is very low (23.8% for ViT-L, 11.5% for ViT-B), which the authors attribute to the noisy synthetic training inputs.
- Conditioning the unreliable-token prediction on the current reliable tokens z_t, rather than z_{t-1}, substantially improves generation (FID 23.63 vs 80.75).
Limitations
- Relies on a pre-trained VQ encoder-decoder, which may constrain generation diversity.
- Linear probing performance is poor, attributed to the gap between noisy synthetic inputs and natural images.
- Only tested at 256x256 resolution; scaling to higher resolution left to future work.
Experimental setup
Datasets: ImageNet-1k COCO ADE20k
Benchmarks: ImageNet-1k 256x256 unconditional generation (FID, IS) ImageNet-1k classification (fine-tuning, linear probing) COCO detection ADE20k segmentation ImageNet-A/R/Sketch/C robustness
Metrics: FID IS top-1 accuracy AP box AP mask mIoU precision recall
Baselines: MAGE iGPT-L ViT-VQGAN MAE BEiT MoCo v3 CAE iBOT MaskGIT ADM BigGAN IC-GAN
Builds on: MAGE MaskGIT VQ-Diffusion MAE VQGAN
Closest papers by meaning
- Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think (ICLR 2025 · similarity 0.89)
- Representation Entanglement for Generation: Training Diffusion Transformers Is Much Easier Than You Think (NeurIPS 2025 · similarity 0.88)
- Learning Diffusion Models with Flexible Representation Guidance (NeurIPS 2025 · similarity 0.88)
- DenoiseRep: Denoising Model for Representation Learning (NeurIPS 2024 · similarity 0.88)
- Deconstructing Denoising Diffusion Models for Self-Supervised Learning (ICLR 2025 · similarity 0.88)
- Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.