Research map / Video / Video segmentation and tracking
egoEMOTION: Egocentric Vision and Physiological Signals for Emotion and Personality Recognition in Real-world Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces egoEMOTION, a 43-participant, 50+ hour dataset pairing Project Aria egocentric eye-tracking/IMU/PPG signals with physiological sensors and self-reported affect, emotion, and Big Five personality, plus classical and deep learning baselines for three benchmark tasks.
Problem
Egocentric vision benchmarks ignore the emotional states and personality traits of the wearer, which shape behavior; no public egocentric dataset couples emotion and personality labels with multimodal sensing.
Approach
A dataset combining Project Aria egocentric eye-tracking, head IMU, nosepad PPG with ECG, EDA, and respiration, using induced video clips and naturalistic tasks, with emoti-SAM and weighted Mikels' Wheel self-reports and BFI-2 personality labels; baselines show egocentric signals (eye-tracking, IMU) outperform wearable physiological signals.
Key results
- Classical SVM/Random Forest on all modalities reaches mean F1 0.75 for continuous affect (arousal, valence, dominance), versus 0.68 for the best deep baseline (CNN).
- For discrete 9-class emotion, egocentric glasses alone reach mean F1 0.46 versus 0.24 for wearable devices and 0.11 random.
- For personality, all modalities give mean F1 0.59 versus 0.53 for the random baseline; egocentric glasses alone give 0.57 versus 0.50 for wearables.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use egocentric-glasses signals instead of wearable-device signals | Classical baseline · Continuous affect | F1: 0.70 -> 0.74 | Helped |
| Use egocentric-glasses signals instead of wearable-device signals | Classical baseline · Discrete emotions | F1: 0.24 -> 0.46 | Helped |
| Use egocentric-glasses signals instead of wearable-device signals | Classical baseline · Personality traits | F1: 0.50 -> 0.57 | Helped |
| Use all modalities instead of egocentric-glasses signals alone | Classical baseline · Continuous affect | F1: 0.74 -> 0.75 | Helped |
| Use all modalities instead of egocentric-glasses signals alone | Classical baseline · Discrete emotions | F1: 0.46 -> 0.46 | No clear effect |
| Use all modalities instead of egocentric-glasses signals alone | Classical baseline · Personality traits | F1: 0.57 -> 0.59 | Helped |
| Use the classical method instead of CNN | All modalities · Continuous affect | F1: 0.68±0.07 -> 0.75±0.13 | Helped |
| Use the classical method instead of WER | All modalities · Continuous affect | F1: 0.60±0.16 -> 0.75±0.13 | Helped |
| Use the classical method instead of CNN | All modalities · Discrete emotions | F1: 0.22±0.02 -> 0.46±0.17 | Helped |
| Use the classical method instead of WER | All modalities · Discrete emotions | F1: 0.21±0.04 -> 0.46±0.17 | Helped |
| Use the classical method instead of CNN | All modalities · Personality traits | F1: 0.41±0.25 -> 0.59±0.49 | Helped |
| Use the classical method instead of WER | All modalities · Personality traits | F1: 0.44±0.28 -> 0.59±0.49 | Helped |
Surprising findings
- Classical machine learning outperformed the deep learning baselines (CNN and WER) on all three benchmark tasks.
- Egocentric glasses signals (eye-tracking, IMU) outperformed traditional physiological wearable signals such as ECG and EDA.
Limitations
- Ground-truth labels rely on retrospective self-reports that may suffer recall bias and miss temporal dynamics.
- Participant pool is mostly young adults (19-29), limiting demographic generalization.
- Limited representation of the low-arousal, low-valence quadrant; no longitudinal recordings.
Experimental setup
Datasets: egoEMOTION
Benchmarks: egoEMOTION
Metrics: F1 score
Baselines: CNN WER Random
Builds on: Circumplex Model of Affect Mikels' Wheel Big Five Inventory-2
Closest papers by meaning
- FEEL: Quantifying Heterogeneity in Physiological Signals for Generalizable Emotion Recognition (NeurIPS 2025 · similarity 0.87)
- $E^3$: Exploring Embodied Emotion Through A Large-Scale Egocentric Video Dataset (NeurIPS 2024 · similarity 0.86)
- Pose as a Modality: A Psychology-Inspired Network for Personality Recognition with a New Multimodal Dataset (AAAI 2025 · similarity 0.86)
- ANNEXE: Unified Analyzing, Answering, and Pixel Grounding for Egocentric Interaction (CVPR 2025 · similarity 0.86)
- The Audio-Visual Conversational Graph: From an Egocentric-Exocentric Perspective (CVPR 2024 · similarity 0.85)
- EmoNet-Face: An Expert-Annotated Benchmark for Synthetic Emotion Recognition (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.