Research map / Large language models / Language, safety and interpretability
Understanding Emotional Body Expressions via Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EAI-LLM, which tokenizes 3D skeleton sequences into LLaMA-7B inputs (via a CTR-GCN encoder, multi-granularity tokens, and a unified skeleton token module) to recognize body-movement emotions and generate textual explanations.
Problem
Skeleton-based emotion recognition methods focus on classification accuracy and cannot give textual explanations, and heterogeneous 3D skeleton datasets (different joint counts and frame lengths) hinder joint training.
Approach
Treats 3D skeleton sequences as a specialized language: a multi-granularity tokenizer produces spatial, temporal and semantic tokens, a masking-based unified token module merges heterogeneous datasets, and a skeleton-text contrastive alignment with LoRA-tuned LLaMA-7B yields both labels and descriptions.
Key results
- Joint training with semantic tokens reaches 85.44% on Emilya, 71.17% on KDAE and 66.97% on EGBM, the best on KDAE and EGBM among compared methods.
- On emotion description, EAI-LLM scores Rouge 0.2018 and METEOR 0.2148, above GPT-4 (0.1282, 0.2006) and Gemini 1.5 Pro (0.1007, 0.1823).
- Training recognition before description causes catastrophic forgetting, dropping average recognition accuracy by 37.06% (semantic) and 61.86% (spatio-temporal).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| combine spatial and temporal tokens rather than use either token type alone (ablation) | EAI-LLM · Emilya, KDAE, and EGBM | accuracy: Emilya: 85.08 and 83.62 -> 86.36; KDAE: 62.99 and 61.21 -> 63.35; EGBM: 53.21 and 49.54 -> 55.96 | Helped |
| jointly train datasets using semantic tokens instead of separate training (ablation) | EAI-LLM · Emilya, KDAE, and EGBM | accuracy: Emilya: 80.63 -> 85.44; KDAE: 67.97 -> 71.17; EGBM: 61.47 -> 66.97 | Helped |
| jointly train datasets using spatio-temporal tokens instead of separate training (ablation) | EAI-LLM · Emilya, KDAE, and EGBM | accuracy: Emilya: 86.36 -> 86.42; KDAE: 63.35 -> 62.28; EGBM: 55.96 -> 63.30 | Mixed |
| use spatio-temporal tokens instead of semantic tokens (ablation) | EAI-LLM · Emotion description task | Rouge: 0.1939 -> 0.2774 | Helped |
| train emotion description before emotion recognition instead of the reverse order (ablation) | EAI-LLM with semantic tokens · Three-dataset average | accuracy: 44.98 -> 81.30 | Helped |
| train emotion description before emotion recognition instead of the reverse order (ablation) | EAI-LLM with spatio-temporal tokens · Three-dataset average | accuracy: 19.59 -> 81.40 | Helped |
| freeze the skeleton encoder rather than fine-tune it (ablation) | EAI-LLM · Emilya | accuracy: 13.82 -> 80.63 | Helped |
| fine-tune the LLM with LoRA instead of keeping it frozen (ablation) | EAI-LLM with frozen skeleton encoder · Emilya | accuracy: 73.75 -> 80.63 | Helped |
| increase the number of projection layers from one to three (ablation) | EAI-LLM · Emilya | accuracy: 80.63 -> 78.26 | Hurt |
| add cross-entropy guidance to semantic alignment for spatio-temporal tokens (ablation) | EAI-LLM · Emilya | accuracy: 76.43 -> 86.36 | Helped |
| add cross-entropy guidance to semantic alignment for semantic tokens (ablation) | EAI-LLM · Emilya | accuracy: 78.81 -> 80.63 | Helped |
| use joint rather than separate training with the CTR-GCN encoder (ablation) | EAI-LLM · Emilya, KDAE, and EGBM | accuracy: 80.63 -> 85.44; 67.97 -> 71.17; 61.47 -> 66.97 | Helped |
Surprising findings
- Recognition-first training order causes severe catastrophic forgetting of recognition accuracy when description fine-tuning follows.
- Fine-tuning the skeleton encoder end-to-end drops recognition accuracy to around 13%, so freezing the encoder is critical.
Limitations
- Accuracy on Emilya (85.44%) lags behind CTR-GCN (89.77%) and GAP (89.16%).
- Hallucinated or multi-label outputs are counted as errors, and fine-grained descriptions can be unrelated to the input sequence.
Experimental setup
Datasets: Emilya KDAE EGBM
Benchmarks: Emilya KDAE EGBM
Metrics: accuracy Rouge BLEU METEOR
Baselines: AGCN CTR-GCN GAP GPT-4 Gemini 1.5 Pro
Builds on: CTR-GCN LLaMA CLIP LoRA ActionCLIP
Closest papers by meaning
- LLMs are Good Action Recognizers (CVPR 2024 · similarity 0.86)
- AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models (ICML 2025 · similarity 0.86)
- $E^3$: Exploring Embodied Emotion Through A Large-Scale Egocentric Video Dataset (NeurIPS 2024 · similarity 0.86)
- MA-Bench: Towards Fine-grained Micro-Action Understanding (CVPR 2026 · similarity 0.86)
- Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning (NeurIPS 2024 · similarity 0.86)
- Emotion-Coherent Reasoning for Multimodal LLMs via Emotional Rationale Verifier (AAAI 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.