Research map / Image generation / GANs
Exploring Sparse MoE in GANs for Text-conditioned Image Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Aurora scales a GAN text-to-image generator with a sparse mixture-of-experts attention block (router conditioned on text-integrated latent) and a U-Net upsampler, reaching competitive FID with diffusion models at fast inference.
Problem
GANs struggle to scale up for large-scale text-conditioned image synthesis, lagging behind diffusion and autoregressive models in quality despite fast feedforward inference.
Approach
Introduce sparse MoE of single-point FFN experts into a GAN generator's attention blocks, with a router that conditions on both the feature map and the text-integrated global latent code, plus a two-stage base-plus-upsampler pipeline trained on public data.
Key results
- Base generator (0.986B params) achieves zero-shot FID30K of 6.20 on MS COCO at 64x64, outperforming GigaGAN (FID10K 9.18) and StyleGAN-T (7.30) in Table 2.
- Upsampled 512x512 output reaches zero-shot FID30K of 8.74 at 256x256 comparison in Table 3, best among GANs, vs GigaGAN 9.03, with 0.16s inference.
- Scaling base generator training from 64 to 320 A100 GPUs reduces zero-shot FID30K from 22.44 to 6.20 in the ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use re-captioned training data (ablation) | Aurora base generator · MS COCO, 64×64 | zero-shot FID30K: 22.44 -> 16.72 | Helped |
| Use re-captioned training data (ablation) | Aurora base generator · Training set, 64×64 | FID10K: 20.77 -> 14.31 | Helped |
| Add sparse MoE experts (ablation) | Aurora base generator · MS COCO, 64×64 | zero-shot FID30K: 16.72 -> 13.81 | Helped |
| Add sparse MoE experts (ablation) | Aurora base generator · Training set, 64×64 | FID10K: 14.31 -> 12.80 | Helped |
| Replace CLIP ViT-L/14 with OpenCLIP ConvNeXt-XXLarge (ablation) | Aurora base generator · MS COCO, 64×64 | zero-shot FID30K: 13.81 -> 12.03 | Helped |
| Replace CLIP ViT-L/14 with OpenCLIP ConvNeXt-XXLarge (ablation) | Aurora base generator · Training set, 64×64 | FID10K: 12.80 -> 11.10 | Helped |
| Increase training scale from 64 to 128 GPUs (ablation) | Aurora base generator · MS COCO, 64×64 | zero-shot FID30K: 12.03 -> 7.63 | Helped |
| Increase training scale from 128 to 320 GPUs (ablation) | Aurora base generator · MS COCO, 64×64 | zero-shot FID30K: 7.63 -> 6.20 | Helped |
| Increase training scale from 128 to 320 GPUs (ablation) | Aurora base generator · Training set, 64×64 | FID10K: 8.54 -> 7.96 | Helped |
| Use Aurora instead of StyleGAN-T | 64×64 text-to-image generation · MS COCO | zero-shot FID30K: 7.30 -> 6.20 | Helped |
| Use Aurora instead of GigaGAN | 64×64 text-to-image generation · Training set | FID10K: 9.18 -> 7.96 | Helped |
| Use Aurora instead of eDiff-I | 256×256 text-to-image generation · MS COCO | zero-shot FID30K: 6.95 -> 8.74 | Hurt |
Surprising findings
- GANs trained with text conditioning were found to be stable to optimize, contrary to common belief about GAN training instability.
- Aurora lags behind diffusion and autoregressive models at high resolution despite best GAN results, per the authors.
Limitations
- Struggles with complex images involving multiple objects, intricate shapes, or fine-grained details.
- Lags behind diffusion and autoregressive models at high resolution.
- Training requires very large compute (320 A100 GPUs for base generator).
Experimental setup
Datasets: LAION2B-en COYO-700M CC3M YFCC15M MS COCO
Benchmarks: MS COCO FID10K training set
Metrics: FID inference speed
Baselines: eDiff-I Stable Diffusion LAFITE StyleGAN-T GigaGAN DALL-E Parti-20B GLIDE Imagen GALIP
Builds on: GigaGAN StyleGAN2 Switch Transformer Imagen
Closest papers by meaning
- Large Scale GAN Training for High Fidelity Natural Image Synthesis (ICLR 2019 · similarity 0.89)
- AdaGAN: Boosting Generative Models (NeurIPS 2017 · similarity 0.89)
- E$^2$GAN: Efficient Training of Efficient GANs for Image-to-Image Translation (ICML 2024 · similarity 0.87)
- You Only Sample Once: Taming One-Step Text-to-Image Synthesis by Self-Cooperative Diffusion GANs (ICLR 2025 · similarity 0.87)
- RG-GAN: Dynamic Regenerative Pruning for Data-Efficient Generative Adversarial Networks (AAAI 2024 · similarity 0.87)
- MCGAN: Enhancing GAN Training with Regression-Based Generator Loss (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.