Research map / Large language models / Agents, code and math
CAREBench: Evaluating LLMs' Emotion Understanding by Assessing Cognitive Appraisal Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CAREBench, an appraisal-theory benchmark with first- and third-person inferential-chain annotations (reasoning, appraisal ratings, multi-label emotions) on real narratives, and evaluates six LLMs on four process-level research questions.
Problem
Existing emotion understanding evaluations rely on discrete label prediction and single-step, single-perspective appraisal datasets, which miss the mental-state inference underlying appraisal and subjective heterogeneity across observers.
Approach
A dual-perspective, process-level benchmark where each real-world narrative carries an inferential chain (situation → appraisal reasoning → appraisal ratings → multi-label emotion with intensity), enabling step-wise, intervention-based, and distributional evaluation of LLMs.
Key results
- Claude and GPT rank top on most tasks; models match or beat the human third-person observer on appraisal rating, emotion intensity, and negative emotion label identification but fall short on appraisal reasoning and positive emotion labels.
- Injecting first-person appraisal reasoning significantly improves appraisal prediction for all models (e.g., Llama RMSE -0.052), whereas injecting model-predicted reasoning degrades appraisal and intensity prediction for most models.
- Injecting core appraisal reasoning gives the most consistent gains in rating distribution fitting (e.g., Llama Wasserstein 0.978 to 0.837), yet disagreement pattern reproduction stays near random (Spearman rho at most 0.282).
Surprising findings
- Predicted (self-generated) appraisal reasoning hurts downstream emotion prediction for most models, indicating models' own reasoning is noisy.
- Fine-tuned domain models PsyLLM and EmoLlama lead on some upstream metrics but do not show corresponding gains on downstream emotion label selection.
Limitations
- Data come only from English-speaking participants in the US and UK, limiting cultural generalizability.
- Strong negative-positive imbalance in emotion samples and only 500 narratives with five third-person annotators constrain distributional estimates.
- Micro-narratives framework with LLM-assisted composition may introduce extraneous details into narratives.
Experimental setup
Datasets: CAREBench
Benchmarks: CAREBench
Metrics: BLEU ROUGE-L BERTScore RMSE example-F1 micro-F1 macro-F1 Wasserstein distance Spearman correlation precision@k
Baselines: Claude-Sonnet-4.6 GPT-5.2 Llama 3.1-8B-Instruct Qwen 3.5-9B PsyLLM EmoLLaMa-chat-13B
Builds on: Crowd-Envent micro-narratives framework Big Five Inventory-10
Closest papers by meaning
- Apathetic or Empathetic? Evaluating LLMs' Emotional Alignments with Humans (NeurIPS 2024 · similarity 0.91)
- MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Models (ICLR 2026 · similarity 0.88)
- Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approach (ICLR 2026 · similarity 0.87)
- Unveiling the Cognitive Compass: Theory-of-Mind–Guided Multimodal Emotion Reasoning (ICLR 2026 · similarity 0.86)
- Emotion-Coherent Reasoning for Multimodal LLMs via Emotional Rationale Verifier (AAAI 2026 · similarity 0.86)
- EEmo-Logic: A Unified Dataset and Multi-Stage Framework for Comprehensive Image-Evoked Emotion Assessment (ICML 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.