Research map / Image generation / Autoregressive and latent generation
Named Entity Driven Zero-Shot Image Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces StyleEntity, a zero-shot text-guided StyleGAN image manipulation method trained only on named-entity text (no images or per-prompt data), plus Prompt Ensemble Latent Averaging (PELA) at inference to suppress noisy manipulation directions.
Problem
Text-guided image manipulation in StyleGAN latent space generalizes poorly to user prompts outside the training distribution, and collecting diverse descriptive prompts for training is impractical.
Approach
Use named entities (celebrity names, breed names) as text proxies for descriptive attributes in CLIP space to train a single MLP mapper with a contrastive loss over all entities, then at inference average manipulation directions over many named-entity prompts (PELA) to denoise edits for unseen user prompts.
Key results
- In FID-CLIP trade-off curves, StyleEntity lies bottom-right of StyleCLIP GD, FFCLIP, and DeltaEdit, indicating less image change at equal CLIP score.
- Human preference study (5 participants, 100 prompts): StyleEntity 39.6% vs DeltaEdit 14.2%, FFCLIP 11.0%, StyleCLIP GD 8.2%-15.0%.
- PELA with N=256 adds about 10 ms per image (44 ms vs 34 ms without PELA); default ensemble size N=64.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with contrastive loss instead of StyleCLIP loss (ablation) | StyleEntity · facial image manipulation | FID-CLIP trade-off | Helped |
| add PELA to average manipulation directions from named entities (ablation) | StyleEntity · facial image manipulation | FID-CLIP trade-off | Helped |
| increase the PELA ensemble size (ablation) | StyleEntity · facial image manipulation | manipulation stability | Helped |
| use PELA with N = 256 instead of removing PELA (ablation) | StyleEntity · StyleGAN image manipulation | inference time: 34 -> 44 | Hurt |
| use positive prompt “with beard” with negative manipulation strength instead of prompt “without beard” | StyleEntity · beard removal | attribute removal | Helped |
Surprising findings
- Existing methods and StyleEntity struggle with removal prompts such as 'without beard'; using 'with beard' with negative manipulation strength works better.
Limitations
- Negative prompts such as 'without beard' do not work directly and require a workaround with negative strength.
- Human evaluation is small (five participants) and the test set is 100 prompts.
Experimental setup
Datasets: Celebrity-Names-90k Cat-Breeds-101 Dog-Breeds-354 FFHQ AFHQ
Benchmarks: FFHQ-based facial manipulation test set of 100 prompts
Metrics: FID CLIP score human preference
Baselines: StyleCLIP GD FFCLIP DeltaEdit
Builds on: StyleCLIP FFCLIP e4e CLIP StyleGAN2
Closest papers by meaning
- Learning to Manipulate Artistic Images (AAAI 2024 · similarity 0.85)
- Unleashing In-context Learning of Autoregressive Models for Few-shot Image Manipulation (CVPR 2025 · similarity 0.83)
- Zero-shot Image Editing with Reference Imitation (NeurIPS 2024 · similarity 0.83)
- Prompt Augmentation for Self-supervised Text-guided Image Manipulation (CVPR 2024 · similarity 0.83)
- Learning Action and Reasoning-Centric Image Editing from Videos and Simulation (NeurIPS 2024 · similarity 0.82)
- On the Test-Time Zero-Shot Generalization of Vision-Language Models: Do We Really Need Prompt Learning? (CVPR 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.