Research map / Image generation / Autoregressive and latent generation
SODA: Bottleneck Diffusion Models for Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SODA, a self-supervised diffusion model that pairs a bottlenecked image encoder with a denoising UNet trained for novel view synthesis, yielding strong linear-probe representations plus reconstruction and synthesis.
Problem
Diffusion models are mostly used for generation, and their representational capacity for self-supervised representation learning is underexplored; prior diffusion-based representation work is limited in scale and downstream performance.
Approach
Condition a DDPM-style UNet decoder on a compact latent produced by an image encoder from a different view of the same object (novel view synthesis as the self-supervised objective), with a tight information bottleneck, layer-wise modulation and masking, and an inverted noise schedule favoring medium noise levels.
Key results
- SODA reaches 72.2% top-1 ImageNet linear-probe accuracy, outperforming generative baselines such as MAE (68.0%) and iGPT-L (65.2%).
- SODA obtains 72.7% F1 on CelebA linear probing, above CLIP embeddings (71.1% F1).
- On single-view novel view synthesis on GSO, SODA improves FID to 1.51 versus 8.32 for DALL-E2 and 38.64 for SRT (geometry-free baselines).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the compact bottleneck (ablation) | SODA · ImageNet-1K linear probe | Top1: 34.2 -> 72.2 | Helped |
| add feature modulation (ablation) | SODA · ImageNet-1K linear probe | Top1: 56.7 -> 72.2 | Helped |
| use novel view synthesis instead of auto-encoding (ablation) | SODA · ImageNet-1K linear probe | Top1: 55.1 -> 72.2 | Helped |
| use the inverted noise schedule (ablation) | SODA · ImageNet-1K linear probe | Top1: 62.0 -> 72.2 | Helped |
| add the compact bottleneck (ablation) | SODA · GSO novel view synthesis | FID: 4.39 -> 1.51 | Helped |
| add feature modulation (ablation) | SODA · GSO novel view synthesis | FID: 4.10 -> 1.51 | Helped |
| use 20 sampling steps (ablation) | SODA · GSO novel view synthesis | FID: 1.51 -> 2.35 | Hurt |
| add layer modulation (ablation) | SODA · CelebA disentanglement | DCI Disentanglement: 65.61 -> 74.67 | Helped |
Surprising findings
- Removing the bottleneck drops ImageNet linear-probe accuracy from 72.2% to 34.2%, showing the compact latent is critical for representation quality.
- Using novel view synthesis (distinct source and target) instead of auto-encoding the same image adds about 17% ImageNet accuracy.
Limitations
- Experiments focus on single-object images; extension to dynamic compositional scenes is left open.
- Generative NVS uses probabilistic sampling, leading to slightly lower PSNR than PixelNeRF on GSO with more source views.
Experimental setup
Datasets: ImageNet CelebA LSUN ShapeNet Google Scanned Objects NMR CUB MPI3D 3DShapes SmallNORB
Benchmarks: ImageNet linear probe CelebA linear probe ImageNet reconstruction GSO ShapeNet NMR DCI disentanglement on CelebA, CUB, MPI3D, 3DShapes, SmallNORB
Metrics: Top-1 accuracy Top-5 accuracy F1 PSNR SSIM FID LPIPS DCI disentanglement DCI completeness DCI informativeness
Baselines: SimCLR BYOL SwAV DINO MAE BEiT iGPT BigBiGAN Palette unCLIP (DALL-E2) DALL-E VQGAN
Builds on: DDPM classifier-free guidance DRL (diffusion representation learning) DiffAE Palette
Closest papers by meaning
- Learning Diffusion Models with Flexible Representation Guidance (NeurIPS 2025 · similarity 0.89)
- DisCo-Diff: Enhancing Continuous Diffusion Models with Discrete Latents (ICML 2024 · similarity 0.88)
- PixelDiT2: Representation-Grounded Pixel Diffusion Transformers (NeurIPS 2026 · similarity 0.87)
- Compositional Discrete Latent Code for High Fidelity, Productive Diffusion Models (NeurIPS 2025 · similarity 0.87)
- Plug-and-Play Diffusion Distillation (CVPR 2024 · similarity 0.87)
- Deconstructing Denoising Diffusion Models for Self-Supervised Learning (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.