Research map / Image generation / Editing and style transfer
EmoEdit: Evoking Emotions through Image Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EmoEdit, a content-aware affective image manipulation method that builds a 40,120-pair dataset (EmoEditSet) and trains a plug-and-play Emotion Adapter (Q-Former-style) on InstructPix2Pix with diffusion plus instruction loss.
Problem
Affective Image Manipulation: modify a user image so it evokes a target emotion while preserving composition; prior methods mostly adjust color and style and lack precise emotional shifts.
Approach
Edit image content (not just color/style) guided by emotion factor trees derived from clustering EmoSet with CLIP and summarized by GPT-4V; an adapter using Q-Former attention fuses target emotion and input image into an emotion embedding, trained with diffusion loss plus an instruction loss that aligns the embedding to a content instruction's text embedding.
Key results
- EmoEdit reaches 50.09% Emo-A and 0.335 Emo-S versus 38.21% and 0.221 for the best baseline SDEdit.
- EmoEdit achieves PSNR 16.62 and SSIM 0.571, the best pixel-level scores among compared methods.
- In a 41-participant user study, EmoEdit received 70.12% votes for structure integrity, 75.73% for emotion fidelity, and 89.12% for balance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EmoEdit instead of SDEdit | AIM image editing · AIM evaluation set | PSNR: 15.43 -> 16.62 | Helped |
| Use EmoEdit instead of AIF | AIM image editing · AIM evaluation set | SSIM: 0.537 -> 0.571 | Helped |
| Use EmoEdit instead of PnP | AIM image editing · AIM evaluation set | LPIPS: 0.381 -> 0.289 | Helped |
| Use EmoEdit instead of PnP | AIM image editing · AIM evaluation set | CLIP-I: 0.851 -> 0.828 | Hurt |
| Use EmoEdit instead of SDEdit | AIM image editing · AIM evaluation set | Emo-A: 38.21% -> 50.09% | Helped |
| Use EmoEdit instead of SDEdit | AIM image editing · AIM evaluation set | Emo-S: 0.221 -> 0.335 | Helped |
| Add the Emotion adapter (ablation) | EmoEdit · AIM image editing | image editing quality | Helped |
| Add instruction loss (ablation) | EmoEdit · AIM image editing | semantic clarity | Helped |
| Add diffusion loss (ablation) | EmoEdit · AIM image editing | structure preservation | Helped |
| Decrease image guidance scale (ablation) | EmoEdit · AIM image editing | emotion intensity | Helped |
| Decrease image guidance scale (ablation) | EmoEdit · AIM image editing | structure preservation | Hurt |
Surprising findings
- Global editing methods SDEdit and PnP performed relatively well, suggesting emotion effects are more global than local.
Limitations
- Depends heavily on EmoSet, which may introduce biases and limit emotion factor coverage.
- Only eight emotion categories and a limited set of visual emotion factors are covered.
- Emotion accuracy metric relies on the training classifier and is prone to bias, requiring more human feedback.
Experimental setup
Datasets: EmoSet MagicBrush MA5K Unsplash
Benchmarks: 405-image inference set (3,240 pairs, 8 emotions)
Metrics: PSNR SSIM LPIPS CLIP-I Emo-A Emo-S user preference
Baselines: SDEdit PnP InsDiff Pix2Pix-Zero ControlNet BLIP-Diffusion CLVA AIF
Builds on: InstructPix2Pix Q-Former EmoSet Composable Diffusion
Closest papers by meaning
- EmoGen: Emotional Image Content Generation with Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.90)
- EmoVIT: Revolutionizing Emotion Insights with Visual Instruction Tuning (CVPR 2024 · similarity 0.89)
- EmoNet-Face: An Expert-Annotated Benchmark for Synthetic Emotion Recognition (NeurIPS 2025 · similarity 0.87)
- EEmo-Logic: A Unified Dataset and Multi-Stage Framework for Comprehensive Image-Evoked Emotion Assessment (ICML 2026 · similarity 0.86)
- Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approach (ICLR 2026 · similarity 0.86)
- Nano-EmoX: Unifying Multimodal Emotional Intelligence from Perception to Empathy (CVPR 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.