Research map / Image generation / Image restoration
Deconstructing Denoising Diffusion Models for Self-Supervised Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Deconstructs a modern diffusion model (DiT) step by step into a classical denoising autoencoder, finding that a low-dimensional patch-wise PCA latent space with noise is the key ingredient, yielding l-DAE that approaches MAE in linear probing.
Problem
Understanding which components of modern denoising diffusion models matter for self-supervised representation learning, and whether a simple classical denoising autoencoder with additive noise can learn good representations.
Approach
Progressively simplify a DiT diffusion model into a classical DAE; the key finding is that adding Gaussian noise in a low-dimensional latent space (patch-wise PCA, no training needed) enables good representations, while class conditioning, perceptual/adversarial losses, input scaling and most DDM machinery are nonessential. The resulting l-DAE predicts the clean image from noisy inverse-PCA-projec
Key results
- Removing class conditioning, tokenizer changes and noise-schedule replacement raise DiT-L linear probe accuracy from 57.5% to 63.4%; patch-wise PCA tokenizer reaches 65.1%.
- The final l-DAE with predicted original image reaches 64.5% at 400 epochs and 69.6% at 1600 epochs; ViT-L encoder reaches 70.9% at 400 epochs.
- At 1600 epochs, l-DAE reaches 66.6% (ViT-B) and 75.0% (ViT-L) linear probe accuracy versus MAE's 68.0% and 75.8%, with MoCo v3 at 76.7% and 77.6%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove class-conditioning (ablation) | DiT baseline · ImageNet | linear probe accuracy: 57.5% to 62.5% | Helped |
| remove VQGAN perceptual loss (ablation) | DiT + remove class-conditioning · ImageNet | linear probe accuracy: 62.5% to 58.4% | Hurt |
| remove VQGAN adversarial loss (ablation) | VQGAN tokenizer without perceptual loss · ImageNet | linear probe accuracy: 58.4% to 59.0% | Helped |
| replace the generation-oriented noise schedule with a linearly decaying signal scale (ablation) | DiT with VQGAN tokenizer without perceptual or adversarial loss · ImageNet | linear probe accuracy: 59.0% to 63.4% | Helped |
| use a patch-wise low-dimensional tokenizer instead of a convolutional VAE tokenizer (ablation) | DiT · ImageNet linear probe, latent dim. 16 | accuracy: 63.4 -> 64.9 | Helped |
| use a single noise level instead of multi-level noise (ablation) | l-DAE · ImageNet | linear probe accuracy: 64.5% to 61.5% | Hurt |
| predict clean data rather than noise (ablation) | PCA DDM baseline · ImageNet | linear probe accuracy: 65.1% to 62.4% | Hurt |
| remove input scaling by fixing gamma_t = 1 (ablation) | clean-data-predicting PCA DDM · ImageNet | linear probe accuracy: 62.4% to 63.6% | Helped |
| operate on the image input with inverse PCA (ablation) | clean-data-predicting model with input scaling removed · ImageNet | linear probe accuracy: 63.6 -> 63.6 | No clear effect |
| predict the original image rather than its PCA reconstruction (ablation) | image-space model using inverse PCA · ImageNet | linear probe accuracy: 63.9 -> 64.5 | Helped |
| use random resized crop augmentation (ablation) | l-DAE · ImageNet | linear probe accuracy: 64.5 -> 65.0 | Helped |
| extend training from 400 to 1600 epochs (ablation) | l-DAE · ImageNet | linear probe accuracy: 65.0 -> 69.6 | Helped |
Surprising findings
- A patch-wise PCA tokenizer, requiring no gradient training, performs as well as learned VAE/AE tokenizers.
- Pixel-space denoising with a high-dimensional patch (16x16x3) drops linear probe accuracy to 23.6%, and self-supervised performance is not correlated with generation quality (FID worsens while accuracy improves).
Limitations
- Self-supervised accuracy still falls short of contrastive methods (MoCo v3), especially for small models.
- Compared with MAE, l-DAE is less training-efficient since it processes all patches rather than only unmasked ones.
- Fine-tuning gains over MAE are limited to ViT-B; with ViT-L l-DAE trails MAE (84.7 vs 85.9).
Experimental setup
Datasets: ImageNet
Benchmarks: ImageNet
Metrics: linear probe top-1 accuracy FID-50K fine-tuning accuracy
Baselines: DiT-L MoCo v3 MAE DDAE
Builds on: Denoising Diffusion Models DiT Latent Diffusion Models Denoising Autoencoder MAE
Closest papers by meaning
- Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think (ICLR 2025 · similarity 0.90)
- Denoising Autoregressive Representation Learning (ICML 2024 · similarity 0.89)
- Learning Diffusion Models with Flexible Representation Guidance (NeurIPS 2025 · similarity 0.89)
- Denoising Task Difficulty-based Curriculum for Training Diffusion Models (ICLR 2025 · similarity 0.89)
- Representation Entanglement for Generation: Training Diffusion Transformers Is Much Easier Than You Think (NeurIPS 2025 · similarity 0.89)
- SD-DiT: Unleashing the Power of Self-supervised Discrimination in Diffusion Transformer (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.