Research map / Vision-language and multimodal / Multimodal LLMs

Nano-EmoX: Unifying Multimodal Emotional Intelligence from Perception to Empathy

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Nano-EmoX, a 2.2B multimodal language model unifying six affective tasks across a three-level perception-understanding-interaction hierarchy, trained with a three-phase P2E curriculum.

Problem

Affective multimodal language models are fragmented: they specialize in one cognitive level of emotion processing (perception, understanding, or interaction) and are resource-heavy, so a single compact model covering all levels is lacking.

Approach

A cognitively inspired three-level task hierarchy guides a curriculum (P2E) that moves from perception to fusion to empathy with chain-of-thought, paired with a small LM (Qwen2.5-1.5B) using a temporal facial encoder and a gated multi-expert visual-speech fusion encoder.

Key results

• Nano-EmoX (2.2B) reaches 91.13 Hit Rate on AvaMERG ERG, a 22.54% gain over prior SOTA and above AffectGPT (s) at 90.88.
• Nano-EmoX achieves a 74.01 average on MSA/MER/OV-MER, best on MER2023, MELD and OV-MERD among compared models.
• On MIntRec 2.0, Nano-EmoX surpasses GPT-4 (3-shot) by 12.4% accuracy and 7.6% WF1.

Surprising findings

• Reverse P2E (empathy-to-perception curriculum) causes large drops, e.g., ERG Hit Rate falls from 91.13 to 57.64.
• Nano-EmoX without MSA-specific fine-tuning keeps competitive MSA accuracy, attributed to implicit learning from related tasks.

Limitations

• Underperforms substantially larger SOTA models on MIR task.
• Performance on MIR improves with more visual tokens, suggesting limits of the current token budget.
• Automated ERI evaluation relies on GPT-3.5-Turbo as judge.

Experimental setup

Datasets: FERV39K CAER CREMA-D M3ED MIntRec MIntRec2.0 MER-Caption+ MERR-Fine AvaMERG

Benchmarks: MER-UniBench EMER MIntRec MIntRec 2.0 AvaMERG OV-MERD

Metrics: Emotion Wheel Hit Rate Weighted Average F1 Clue Overlap Label Overlap Accuracy Weighted F1 Weighted Precision Dist-1 Dist-2 Likert empathy/insight/safety ratings

Baselines: AffectGPT (s) SALMONN MiniCPM-V-2.6-8B Qwen2-VL-7B Emotion-LLaMA AffectGPT MobileVLM V2 R1-Omni Omni-Emotion Emotion-Qwen GPT-4 Empatheia

Builds on: AffectGPT Emotion-LLaMA FaceXFormer Q-Former E3RG

Closest papers by meaning

• MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Models (ICLR 2026 · similarity 0.90)
• EEmo-Logic: A Unified Dataset and Multi-Stage Framework for Comprehensive Image-Evoked Emotion Assessment (ICML 2026 · similarity 0.88)
• EmpathyChat: Structured Cognitive Reasoning in Empathetic Spoken Dialogue (NeurIPS 2026 · similarity 0.88)
• AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models (ICML 2025 · similarity 0.88)
• Unveiling the Cognitive Compass: Theory-of-Mind–Guided Multimodal Emotion Reasoning (ICLR 2026 · similarity 0.87)
• Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning (NeurIPS 2024 · similarity 0.87)