Research map / Image generation / Text-to-image diffusion
VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance
Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Castricato, Edward Raff
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add random augmentations (affine, perspective, noise, flip, color jitter) with multi-crop averaging of the CLIP loss (ablation) | VQGAN-CLIP · Qualitative augmentation ablation | visual quality | Improved |
| add weighted L2 regularization of the z-vector (ablation) | VQGAN-CLIP · Qualitative regularization ablation | visual quality | Improved |
| replace Codebook Sampling with z-quantize optimization (ablation) | VQGAN-CLIP · Qualitative comparison | speed and image quality | Improved |
| use VQGAN-CLIP instead of minDALL-E for text-to-image | text-to-image generation · Human text-image alignment rating (12 prompts) mean | mean rating (1-5): 2.7 -> 4.6 | Improved |
| use VQGAN-CLIP instead of GLIDE (CLIP) | text-to-image generation · Human text-image alignment rating (12 prompts) mean | mean rating (1-5): 3.3 -> 4.6 | Improved |
| use VQGAN-CLIP instead of GLIDE (CF) | text-to-image generation · Human text-image alignment rating (12 prompts) mean | mean rating (1-5): 2.3 -> 4.6 | Improved |
| use VQGAN-CLIP instead of GLIDE (CLIP) on prompt L | text-to-image generation · Human rating, prompt L | mean rating (1-5): 5.0 -> 4.7 | Hurt |
| use VQGAN-CLIP instead of minDALL-E on speed | text-to-image generation · Run-time on V100 | seconds per image: 16.3s -> 188.3s | Hurt |
| use VQGAN-CLIP instead of GLIDE (filtered) on speed | text-to-image generation · Run-time on V100 | seconds per image: 9.7s -> 188.3s | Hurt |
| use training-free VQGAN-CLIP instead of pretrained minDALL-E in total compute | text-to-image generation · Total training plus inference cost | V100-hours: minDALL-E cheaper after 858 V100-hours; GLIDE after 20200 | Mixed |
| use VQGAN-CLIP instead of Open-Edit for color and weather editing | image editing · Qualitative editing comparison | structure preservation | Improved |
| swap prompt wording (ablation) | VQGAN-CLIP · Prompt wording | quality of generation | Mixed |
Later papers that compared with this method
6 later accepted papers report VQGAN-CLIP as a baseline in their results tables; 5 of them report a better number on most of the results they share with it.
- Image Translation as Diffusion Visual Programmers (ICLR 2024 · better on 1 of 1 shared result)
- NULL-Text Inversion for Editing Real Images Using Guided Diffusion Models (CVPR 2023 · better on 1 of 1 shared result)
- Diffusion-based Image Translation using disentangled style and content representation (ICLR 2023 · better on 1 of 1 shared result)
- MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing (NeurIPS 2023 · better on 0 of 1 shared result)
- Zero-Shot Contrastive Loss for Text-Guided Diffusion Image Style Transfer (ICCV 2023 · better on 1 of 1 shared result)
- CLIPstyler: Image Style Transfer With a Single Text Condition (CVPR 2022 · better on 3 of 3 shared results)
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Imagine Before You Draw: Visual Prompt Engineering for Image Generation (NeurIPS 2026 · similarity 0.87)
- Instruct-CLIP: Improving Instruction-Guided Image Editing with Automated Data Refinement Using Contrastive Learning (CVPR 2025 · similarity 0.86)
- OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation (CVPR 2025 · similarity 0.85)
- Kosmos-G: Generating Images in Context with Multimodal Large Language Models (ICLR 2024 · similarity 0.85)
- VIXEN: Visual Text Comparison Network for Image Difference Captioning (AAAI 2024 · similarity 0.85)
- Visually Guided Decoding: Gradient-Free Hard Prompt Inversion with Language Models (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
