Research map / Video / Video-language understanding
OSMO: Open-vocabulary Self-eMOtion Tracking
Mohamed Abdelfattah, Bugra Tekin, Fadime Sener, Necati Cihan Camgoz, Eric Sauser, Shugao Ma, Alexandre Alahi, Edoardo Remelli
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add SENSE chain-of-thought reasoning supervision (ablation) | AffectGPT baseline · OSMO XSub | mean gain: +8.2 mean | Improved |
| add dialogue history context (ablation) | AffectGPT baseline · OSMO XSub | mean gain: +7.7 mean | Improved |
| add memory queries of past multimodal expressions (ablation) | AffectGPT baseline · OSMO XSub | mean gain: +6.8 mean | Improved |
| add prior emotions as text input (ablation) | AffectGPT baseline · OSMO XSub | mean gain: +3.6 mean | Improved |
| expand dialogue history from 1 to 16 utterances (ablation) | OSIRIS · OSMO XSub | OVER Hit Rate gain over AffectGPT: +6.4 -> +11.4 | Improved |
| increase number of prior emotions from 0 to 4 (ablation) | OSIRIS · OSMO XSub | OVER Hit Rate gain: 7.6 -> 10.9 | Improved |
| enlarge memory size beyond one slot (ablation) | OSIRIS · OSMO XSub | OVER Hit Rate gain: 10.6 -> 11.0 | No clear effect |
| add eye infrared input (ablation) | OSIRIS · OSMO XSub | mean change: -1.6 mean without eyes; -3.9 mIoU | Improved |
| add dialogue text modality (ablation) | OSIRIS · OSMO XSub | mean change: -8.5 mean when removed | Improved |
| add audio modality (ablation) | OSIRIS · OSMO XSub | mean change: -6.8 mean when removed | Improved |
| add video modality (ablation) | OSIRIS · OSMO XSub | mean change: -4.6 mean when removed | Improved |
| train on OSMO instead of E3 dataset | OSIRIS and E3-LLaMA · OSMO XSet | SOS: 44.4 -> 60.7 | Improved |
Closest papers by meaning
- OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition (ICML 2025 · similarity 0.88)
- $E^3$: Exploring Embodied Emotion Through A Large-Scale Egocentric Video Dataset (NeurIPS 2024 · similarity 0.88)
- AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models (ICML 2025 · similarity 0.88)
- egoEMOTION: Egocentric Vision and Physiological Signals for Emotion and Personality Recognition in Real-world Tasks (NeurIPS 2025 · similarity 0.87)
- AffectGPT-RL: Revealing Roles of Reinforcement Learning in Open-Vocabulary Emotion Recognition (NeurIPS 2026 · similarity 0.86)
- FEEL: Quantifying Heterogeneity in Physiological Signals for Generalizable Emotion Recognition (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
