Research map / Image generation / Text-to-image diffusion
EmoGen: Emotional Image Content Generation with Text-to-Image Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Emotional Image Content Generation (EICG), building an emotion space aligned to CLIP space via an MLP mapping network, with attribute loss and emotion confidence, to generate semantically clear, emotion-faithful images from emotion categories.
Problem
Text-to-image diffusion models generate concrete concepts well but struggle with abstract emotions; existing emotion image transfer via color/style achieves limited emotion accuracy with fixed content.
Approach
Learn an emotion representation space from EmoSet with a ResNet-50 encoder, map it non-linearly into the frozen CLIP text space to condition Stable Diffusion, and add an attribute loss (object/scene labels) plus a per-attribute emotion-confidence weight that balances LDM loss vs. attribute loss.
Key results
- EmoGen achieves FID 41.60 and 76.25% emotion accuracy (Emo-A), versus 44.05 FID and 70.77% for Stable Diffusion in Table 1.
- Semantic clarity Sem-C 0.633 and semantic diversity Sem-D 0.0335, best among compared methods (Textual Inversion Sem-D 0.0282).
- User study with 14 participants: 82.14% emotion accuracy by user voting; 88.39% of responses say emotions are triggered by content/semantics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use EmoGen instead of Stable Diffusion | Emotional image generation · Emotional image generation | FID: 44.05 -> 41.60 | Helped |
| use EmoGen instead of Stable Diffusion | Emotional image generation · Emotional image generation | Emo-A: 70.77% -> 76.25% | Helped |
| add nonlinear mapping network F (ablation) | EmoGen · Emotional image generation | FID: 57.54 -> 41.60 | Helped |
| add nonlinear mapping network F (ablation) | EmoGen · Emotional image generation | Sem-D: 0.0261 -> 0.0335 | Helped |
| add attribute loss Lattr (ablation) | EmoGen · Emotional image generation | Sem-C: 0.592 -> 0.633 | Helped |
| add attribute loss Lattr (ablation) | EmoGen · Emotional image generation | Sem-D: 0.0270 -> 0.0335 | Helped |
| add attribute loss Lattr (ablation) | EmoGen · Emotional image generation | Emo-A: 65.75% -> 76.25% | Helped |
| add emotion confidence αij (ablation) | EmoGen · Emotional image generation | Sem-C: 0.591 -> 0.633 | Helped |
| add emotion confidence αij (ablation) | EmoGen · Emotional image generation | Emo-A: 74.88% -> 76.25% | Helped |
| add emotion confidence αij (ablation) | EmoGen · Emotional image generation | FID: 43.30 -> 41.60 | Hurt |
Surprising findings
- Comparison methods all achieve comparable emotion accuracy (70-75%), so emotion accuracy alone is deemed insufficient for EICG, motivating the added semantic metrics.
- Existing prior emotion-transfer work reports only 29% emotion accuracy, indicating color/style-based methods are weak at evoking emotion.
Limitations
- Assumes content is either emotional or emotion-agnostic, a binary simplification of the content-emotion relationship.
- Focuses only on content as the emotional factor, ignoring color and style.
- Emotion accuracy measured by a pre-trained classifier may not match human cognition; comparable Emo-A across methods suggests limited discrimination.
Experimental setup
Datasets: EmoSet
Benchmarks: EmoSet
Metrics: FID LPIPS Emo-A Sem-C Sem-D CLIP-img CLIP-txt user preference
Baselines: Stable Diffusion Textual Inversion DreamBooth
Builds on: Stable Diffusion CLIP Textual Inversion DreamBooth EmoSet
Closest papers by meaning
- EmoEdit: Evoking Emotions through Image Manipulation (CVPR 2025 · similarity 0.90)
- EmoVIT: Revolutionizing Emotion Insights with Visual Instruction Tuning (CVPR 2024 · similarity 0.85)
- A Unified and Interpretable Emotion Representation and Expression Generation (CVPR 2024 · similarity 0.85)
- EmoNet-Face: An Expert-Annotated Benchmark for Synthetic Emotion Recognition (NeurIPS 2025 · similarity 0.85)
- Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approach (ICLR 2026 · similarity 0.84)
- MMIG-Bench: Towards Comprehensive and Explainable Evaluation of Multi-Modal Image Generation Models (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.