Research map / Image generation / Autoregressive and latent generation
A Unified and Interpretable Emotion Representation and Expression Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes C2A2, a unified 3D emotion space (canonical, compound, action units, arousal-valence) with a GANmut-based method to learn the third axis Z, and a DreamBooth-style number+text-to-image diffusion model that generates continuous compound expressions.
Problem
Emotion models (categorical, compound, AU, arousal-valence) are inconsistent and not mutually mappable, and text-to-image diffusion models cannot precisely control continuous facial expressions, especially compound ones.
Approach
Lift the 2D arousal-valence space to 3D by placing fear above and sadness below the AV plane, learn the third axis Z implicitly from AU pseudo-labels and AV labels via a GANmut-style conditional discriminator, and inject a number encoder embedding of the 3D vector into DreamBooth's text conditioning so diffusion models accept continuous emotion values.
Key results
- The 3D model reaches FED 16.06 versus 20.35 for the 2D-AV model and 61.15 for DreamBooth, with ERE 0.0536 vs 0.0774.
- In an expert psychologist study (8 raters), the 3D model averages 4.19 on the seven compound emotions that 2D-AV cannot represent, versus 3.01 for the 2D-AV model and 1.47 for DreamBooth.
- The 3D representation covers 15 of 17 target compound emotions in the table, compared with 6 of 17 for the 2D AV representation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use 3D C2A2 instead of 2D AV representation (ablation) | emotion-conditioned image generation · AffectNet | FED: 20.347 -> 16.060 | Helped |
| use 3D C2A2 instead of 2D AV representation (ablation) | emotion-conditioned image generation · AffectNet | ERE: 0.0774 -> 0.0536 | Helped |
| use 3D C2A2 instead of 2D AV representation (ablation) | emotion-conditioned image generation · AffectNet | SS: 0.905 -> 0.806 | Helped |
| use 3D C2A2 instead of 2D AV representation (ablation) | emotion-conditioned image generation · 7 compound emotions (average) | user rating: 3.01 -> 4.19 | Helped |
| use 3D C2A2 instead of 2D AV representation (ablation) | emotion-conditioned image generation · all emotions | user rating: 3.50 -> 4.03 | Helped |
| use 3D C2A2 instead of DreamBooth | emotion-conditioned image generation · AffectNet | FED: 61.147 -> 16.060 | Helped |
| use 3D C2A2 instead of DALL.E 3 | emotion-conditioned image generation · 7 compound emotions (average) | user rating: 3.31 -> 4.19 | Helped |
Surprising findings
- DALL-E 3 scored higher than the 3D model on overall average user rating (3.72 vs 4.03 is lower; the table shows DALL-E 3 at 3.72 vs 3D at 4.03 overall), while DreamBooth trained on AffectNet scored lowest among methods.
Limitations
- Emotion conditioning is not fully disentangled; attributes such as age leak into the generated expression.
- Identity is not preserved during expression control.
- Text descriptions beyond facial attributes need further exploration.
Experimental setup
Datasets: AffectNet
Benchmarks: AffectNet
Metrics: Fréchet Emotion Distance (FED) Emotion Reconstruction Error (ERE) Smoothness Score (SS) FID user rating (1-5)
Baselines: DreamBooth Stable Diffusion DALL-E 3 2D-AV model AUs model
Builds on: DreamBooth GANmut Stable Diffusion CLIP
Closest papers by meaning
- VidEmo: Affective-Tree Reasoning for Emotion-Centric Video Foundation Models (NeurIPS 2025 · similarity 0.86)
- AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models (ICML 2025 · similarity 0.85)
- OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition (ICML 2025 · similarity 0.85)
- Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approach (ICLR 2026 · similarity 0.85)
- Bridge Then Begin Anew: Generating Target-Relevant Intermediate Model for Source-Free Visual Emotion Adaptation (AAAI 2025 · similarity 0.85)
- EmoGen: Emotional Image Content Generation with Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.