Research map / Vision-language and multimodal / Multimodal LLMs

Facial-R1: Aligning Reasoning and Recognition for Facial Emotion Analysis

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Interleaved image-text generation with MLLMs · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Facial-R1, a three-stage SFT, GRPO-based RL with AU-F1 and emotion-accuracy rewards, and iterative data synthesis on Qwen2.5-VL-7B to produce explainable facial emotion analysis, releasing the FEA-20K dataset.

Problem

Vision-language models for facial emotion analysis (AU recognition, AU-based emotion reasoning, and emotion recognition) hallucinate reasoning and produce reasoning that conflicts with the final emotion label, while annotated reasoning data is scarce and costly.

Approach

A three-stage alignment: minimal SFT on 300 GPT-4o-mini samples for basic emotion reasoning, then GRPO with verifiable AU-F1 and emotion-accuracy rewards to align reasoning with recognition, then iterative self-generated data synthesis (with ground-truth AU/emotion conditioning and filtering) to expand training data.

Key results

• Facial-R1 achieves 73.1% F1 on DISFA AU recognition, the best among compared methods, and 92.1% accuracy on RAF-DB (vs. FMAE 93.4%).
• On FEA-20K emotion reasoning, Facial-R1 reaches ROUGE-L 37.3 versus 32.3 for GPT-4o and 30.1 for EmoLA.
• Ablation: removing the RL stage drops DISFA F1 from 73.1 to 54.3; removing the AU reward drops DISFA F1 by 17.3 points in the reward ablation.

Surprising findings

• Qwen2.5-VL-7B zero-shot achieves only 20.7% AU F1 on BP4D while Facial-R1 reaches 67.4%, showing large gains from minimal supervision.
• Facial-R1 slightly trails specialized end-to-end FMAE on RAF-DB emotion accuracy (92.1 vs 93.4) and Norface on BP4D AU F1.

Limitations

• Reliance on GPT-4o-mini for SFT samples and as the automatic judge for reasoning quality.
• Training images are not redistributed due to copyright, so users must download source datasets.
• Does not beat specialized end-to-end models on some recognition metrics (e.g., RAF-DB, BP4D AU F1).

Experimental setup

Datasets: DISFA BP4D RAF-AU FER2013 AffectNet RAF-DB FABA-Instruct

Benchmarks: DISFA BP4D RAF-AU FABA-Instruct FER2013 AffectNet RAF-DB FEA-20K

Metrics: F1 accuracy ROUGE-L SEGE/REGE GPT-judged similarity score (0-10)

Baselines: GPT-4o GPT-4o-mini LLaVA-Next-7B InternVL-7B Qwen2.5-VL-7B FMAE Norface J-AA-Net Face-LLaVA EmoLA ExpLLM Exp-BLIP

Builds on: GRPO Qwen2.5-VL FABA-Instruct DeepSeek-R1 format reward

Closest papers by meaning

• EmoNet-Face: An Expert-Annotated Benchmark for Synthetic Emotion Recognition (NeurIPS 2025 · similarity 0.88)
• VidEmo: Affective-Tree Reasoning for Emotion-Centric Video Foundation Models (NeurIPS 2025 · similarity 0.87)
• AffectGPT-RL: Revealing Roles of Reinforcement Learning in Open-Vocabulary Emotion Recognition (NeurIPS 2026 · similarity 0.87)
• Navigating Label Ambiguity for Facial Expression Recognition in the Wild (AAAI 2025 · similarity 0.87)
• EmoThinker: Advancing Visual-Acoustic Emotion Analysis via Structural Token Selection and Chain-of-Thought Reasoning (CVPR 2026 · similarity 0.86)
• AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models (ICML 2025 · similarity 0.86)