Research map / Image generation / Text-to-image diffusion
Adversarial Text to Continuous Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HyperCGAN, a hypernetwork-conditioned implicit-neural-representation GAN that generates text-conditioned continuous images using word-level attention-based weight modulation (WHAtt).
Problem
Text-to-image synthesis is dominated by pixel-array representations, which do not support continuous properties such as arbitrary-resolution generation and extrapolation; naive text conditioning of INR-based GANs performs poorly and trains unstably.
Approach
Condition an unconditional INR-GAN (INR-GAN) on text by using hypernetworks to produce modulating weight tensors, factorized with CP decomposition, and apply word-level scaled dot-product attention (WHAtt) over per-word weight tensors built from CLIP token features, so the decoder's weights are modulated at the word level.
Key results
- On CUB 256^2, HyperCGAN with word-level conditioning reaches FID 14.12 and CLIP-R 51.87% (tglobal rank 5), versus 57.25 FID for the sentence-level conditioned baseline.
- On COCO 256^2, HyperCGAN achieves FID 13.54 and CLIP-R 85.12%, outperforming most 2018-2022 discrete GANs such as DF-GAN, XMC-GAN and AttnGAN, with 57M parameters.
- In 2x and 4x extrapolation on COCO, HyperCGAN obtains ScaleInv-FID 62.01 and 85.4, beating InfinityGAN's pseudo-conditional variant (132.24 and 120.32) despite not being trained for extrapolation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add word-level conditioning with WHAtt (ablation) | INR-GAN on CUB 256² · CUB 256² | FID: 19.13 -> 14.12 | Helped |
| Add word-level conditioning with WHAtt (ablation) | INR-GAN on CUB 256² · CUB 256² | CLIP-R: 51.12% -> 51.87% | Helped |
| Increase INR rank from 1 to 5 with tproj conditioning (ablation) | INR-GAN on CUB 256² · CUB 256² | FID: 78.23 -> 69.92 | Helped |
| Increase INR rank from 1 to 5 with tglobal conditioning (ablation) | INR-GAN on CUB 256² · CUB 256² | FID: 62.52 -> 57.25 | Helped |
| Increase INR rank from 1 to 5 with tglobal conditioning (ablation) | INR-GAN on CUB 256² · CUB 256² | CLIP-R: 45.67% -> 51.81% | Helped |
| Add X-factorization (ablation) | INR-GAN on CUB 256² · CUB 256² | FID: 57.25 -> 19.13 | Helped |
| Add tglobal conditioning with rank 1 after X-factorization (ablation) | INR-GAN on CUB 256² · CUB 256² | FID: 19.13 -> 18.15 | Helped |
| Increase rank from 1 to 5 with tglobal conditioning after X-factorization (ablation) | INR-GAN on CUB 256² · CUB 256² | FID: 18.15 -> 14.12 | Helped |
| Increase rank from 1 to 5 with tglobal conditioning after X-factorization (ablation) | INR-GAN on CUB 256² · CUB 256² | CLIP-R: 53.78% -> 51.87% | Hurt |
| Use HyperCGAN instead of GALIP | Text-to-image generation on COCO 256² · COCO 256² | FID: 5.85 -> 13.54 | Hurt |
| Use HyperCGAN instead of GALIP | Text-to-image generation on COCO 256² · COCO 256² | CLIP-R: 99.84% -> 85.12% | Hurt |
| Use HyperCGAN instead of LAFITE | Text-to-image generation on ArtEmis 256² · ArtEmis 256² | FID: 12.04 -> 15.89 | Hurt |
Surprising findings
- Simply increasing INR rank in sentence-level conditioning does not improve FID or CLIP-R.
- Word-level modulation without factorization runs out of memory, so tensor factorization is required for it to be feasible.
Limitations
- Visual quality still lags recent discrete state-of-the-art (e.g., GALIP, LAFITE) in FID and CLIP-R.
- Struggles with object counting and correct object composition; artifacts such as wavy textures in extrapolation and superresolution.
- Generates pixels independently without spatial local context.
Experimental setup
Datasets: MS-COCO CUB ArtEmis WikiArt
Benchmarks: MS-COCO 256^2 CUB 256^2 ArtEmis 256^2
Metrics: FID CLIP-R R-precision ScaleInv-FID patch-FID
Baselines: AttnGAN ControlGAN DM-GAN DAE-GAN TIME DF-GAN SSA-GAN XMC-GAN LAFITE GALIP VQ-Diffusion-F InfinityGAN
Builds on: INR-GAN LAFITE AttnGAN hypernetworks CLIP
Closest papers by meaning
- DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-To-Image Synthesis (CVPR 2019 · similarity 0.90)
- StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis (ICML 2023 · similarity 0.89)
- DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesis (CVPR 2022 · similarity 0.89)
- Text to Image Generation With Semantic-Spatial Aware GAN (CVPR 2022 · similarity 0.89)
- Adversarial Generation of Continuous Images (CVPR 2021 · similarity 0.89)
- E$^2$GAN: Efficient Training of Efficient GANs for Image-to-Image Translation (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.