AI Paper Atlas

Research map / Image generation / Text-to-image diffusion

VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance

Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Castricato, Edward Raff

ECCV 2022 · Image generation › Text-to-image diffusion · approach: Personalized and region-controlled generation · 373 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add random augmentations (affine, perspective, noise, flip, color jitter) with multi-crop averaging of the CLIP loss (ablation)VQGAN-CLIP · Qualitative augmentation ablationvisual qualityImproved
add weighted L2 regularization of the z-vector (ablation)VQGAN-CLIP · Qualitative regularization ablationvisual qualityImproved
replace Codebook Sampling with z-quantize optimization (ablation)VQGAN-CLIP · Qualitative comparisonspeed and image qualityImproved
use VQGAN-CLIP instead of minDALL-E for text-to-imagetext-to-image generation · Human text-image alignment rating (12 prompts) meanmean rating (1-5): 2.7 -> 4.6Improved
use VQGAN-CLIP instead of GLIDE (CLIP)text-to-image generation · Human text-image alignment rating (12 prompts) meanmean rating (1-5): 3.3 -> 4.6Improved
use VQGAN-CLIP instead of GLIDE (CF)text-to-image generation · Human text-image alignment rating (12 prompts) meanmean rating (1-5): 2.3 -> 4.6Improved
use VQGAN-CLIP instead of GLIDE (CLIP) on prompt Ltext-to-image generation · Human rating, prompt Lmean rating (1-5): 5.0 -> 4.7Hurt
use VQGAN-CLIP instead of minDALL-E on speedtext-to-image generation · Run-time on V100seconds per image: 16.3s -> 188.3sHurt
use VQGAN-CLIP instead of GLIDE (filtered) on speedtext-to-image generation · Run-time on V100seconds per image: 9.7s -> 188.3sHurt
use training-free VQGAN-CLIP instead of pretrained minDALL-E in total computetext-to-image generation · Total training plus inference costV100-hours: minDALL-E cheaper after 858 V100-hours; GLIDE after 20200Mixed
use VQGAN-CLIP instead of Open-Edit for color and weather editingimage editing · Qualitative editing comparisonstructure preservationImproved
swap prompt wording (ablation)VQGAN-CLIP · Prompt wordingquality of generationMixed

Later papers that compared with this method

6 later accepted papers report VQGAN-CLIP as a baseline in their results tables; 5 of them report a better number on most of the results they share with it.

Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy