Research map / Image generation / Autoregressive and latent generation
Unified Generation, Reconstruction, and Representation: Generalized Diffusion with Adaptive Latent Encoding-Decoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EDDPMs, a generalized diffusion model that adds a learnable encoder-decoder at the first diffusion step, trained jointly with the diffusion objective, to unify generation, reconstruction, and representation across text, protein, and image data.
Problem
Deep generative models excel at only some of generation, reconstruction, and representation, and across data types; diffusion models lack compact semantic representations and struggle with discrete data such as text and proteins, while VAEs/latent diffusion train components separately or suffer post
Approach
Replace the first Gaussian noising-denoising step of a diffusion model with a parameterized neural encoder and decoder (e.g., BERT encoder and GPT-2 decoder for text, UNet encoder with DiffAE-style decoder for images), and train encoder, decoder and diffusion jointly under the standard DDPM-style objective, which adds an alignment loss and a reconstruction loss weighted by a hyperparameter w.
Key results
- On Yelp text, EDDPM achieves MAUVE 0.977 for generation distribution match versus 0.240 for LatentOps, with reconstruction BLEU 92.1 versus 87.6 for LatentOps.
- On image manipulation with CelebA-HQ, EDDPM obtains FID 13.48 at epsilon 0.1 versus 19.62 for DiffAE, and classification accuracy 0.893 versus 0.795.
- Training cost per epoch is 19.35 minutes versus 13.98 for LatentOps, roughly 1.3x, with overall training time of 7.41 hours versus 6.98 hours.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EDDPM instead of GPT4 | Yelp review reconstruction · Yelp review | reconstruction content: 100 -> 92.1 | Hurt |
| Use EDDPM instead of GPT2 | Yelp review generation · Yelp review | fluency (perplexity): 15.0 -> 16.4 | Hurt |
| Use EDDPM instead of LatentOps | Yelp review generation · Yelp review | distribution match (MAUVE): 0.240 -> 0.977 | Helped |
| Use EDDPM instead of GPT4 | Yelp review generation · Yelp review | diversity: 0.87 -> 0.79 | Hurt |
| Use EDDPM instead of LatentOps | Yelp review latent arithmetic · Yelp review | accuracy: 57.3 -> 57.1 | Hurt |
| Use EDDPM instead of LatentOps | Yelp review interpolation · Yelp review interpolation | fluency (perplexity): 32.5 -> 30.8 | Helped |
| Use EDDPM instead of Optimus-DAAE | Yelp review interpolation · Yelp review interpolation | distribution match (MAUVE): 0.770 -> 0.763 | Hurt |
| Use jointly trained EDDPM instead of DiffAE | CelebA-HQ image manipulation · CelebA-HQ manipulation | FID (ε=0.1): 19.62 -> 13.48 | Helped |
| Use jointly trained EDDPM instead of DiffAE | CelebA-HQ image manipulation · CelebA-HQ manipulation | FID (ε=0.3): 23.69 -> 18.43 | Helped |
Surprising findings
- GPT-4 with 20-shot prompting fails to capture domain-specific text characteristics, scoring low on generation distribution match (0.007 MAUVE).
- Standard text diffusion models GENIE and AR-Diffusion perform poorly on both reconstruction and generation, showing the difficulty of token-space diffusion for discrete text.
Limitations
- Only the first diffusion step is parameterized with a learnable encoder-decoder, so hierarchical latents are not explored.
- Not verified on other modalities such as video, audio, time series, or multimodal data.
- Text experiments use a single sentiment dataset (Yelp) and a relatively small BERT-small encoder.
Experimental setup
Datasets: BookCorpus Yelp review FFHQ CelebA CelebA-HQ LSUN-Bedroom LSUN-Horse Gifford GFP
Benchmarks: Yelp review FFHQ CelebA CelebA-HQ LSUN-Bedroom LSUN-Horse Gifford GFP
Metrics: BLEU MAUVE perplexity FID rFID transfer accuracy MSE Pearson Spearman
Baselines: LatentOps Optimus-DAAE GENIE AR-Diffusion GPT2 GPT4 DiffAE LDM DDIM StyleGAN-XL NVAE Consistency Models
Builds on: DDPM variational diffusion models DiffAE LatentOps Optimus
Closest papers by meaning
- Neural Diffusion Models (ICML 2024 · similarity 0.90)
- Generalization in VAE and Diffusion Models: A Unified Information-Theoretic Analysis (ICLR 2025 · similarity 0.89)
- Revisiting Diffusion Models: From Generative Pre-training to One-Step Generation (ICML 2025 · similarity 0.89)
- Improving Training Efficiency of Diffusion Models via Multi-Stage Framework and Tailored Multi-Decoder Architecture (CVPR 2024 · similarity 0.89)
- DisCo-Diff: Enhancing Continuous Diffusion Models with Discrete Latents (ICML 2024 · similarity 0.89)
- Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.