Research map / Video / Video segmentation and tracking
MART: Masked Affective RepresenTation Learning via Masked Temporal Distribution Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MART, an MAE-style self-supervised method that masks emotionally salient video segments (guided by lexicon-verified affective cues and cross-modal attention) and recovers the temporal emotion distribution, plugged into existing video emotion analysis backbones.
Problem
Video emotion analysis suffers from limited labeled data, and generic MAE-style pretraining on low-level pixels does not capture high-level, temporally varying affective cues.
Approach
Uses SenticNet-based emotional lexicon extraction verified hierarchically (sentiment, emotion, degree) against temporal CAM emotion scores, then masks high cross-modal-attention affective tokens with a progressive ratio and recovers feature- and patch-level temporal emotion distributions, plus a pull/push temporal affective complementary contrastive loss.
Key results
- MART reaches 79.10% sentiment accuracy on Ekman6 versus 68.23% for the MBT baseline (Table 1).
- On Ekman6, masking with MART gives a 10.58% gain over the MBT baseline in emotion-recognition ablation (Table 4 discussion).
- Hierarchical lexicon verification gives about a 2.62% consistent improvement across four metrics on three video emotion datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add MART masked affective representation learning | Baseline [48] · Ekman6 | AccE: 48.08 -> 52.66 | Helped |
| add MART masked affective representation learning | Baseline [48] · VideoEmotion8 | AccE: 43.61 -> 50.83 | Helped |
| add MART masked affective representation learning | Baseline [48] · PERR | AccE: 56.33 -> 57.89 | Helped |
| add MART masked affective representation learning | Baseline [48] · IEMOCAP | AccE: 65.79 -> 66.29 | Helped |
| add MART masked affective representation learning | Baseline [48] · AffWild2 | AccE: 63.50 -> 65.23 | Helped |
| add emotion-oriented masking (ablation) | MART on Ekman6 · Ekman6 | AccE: 52.16 -> 53.17 | Helped |
| add hierarchical lexicon verification (ablation) | MART on Ekman6 · Ekman6 | AccE: 51.42 -> 53.17 | Helped |
| replace affective masking with frame masking (ablation) | MART on Ekman6 · Ekman6 | AccE: 53.17 -> 49.17 | Hurt |
| replace affective masking with random masking (ablation) | MART on Ekman6 · Ekman6 | AccE: 53.17 -> 51.42 | Hurt |
| replace affective masking with tube masking (ablation) | MART on Ekman6 · Ekman6 | AccE: 53.17 -> 49.07 | Hurt |
| replace affective masking with attention masking (ablation) | MART on Ekman6 · Ekman6 | AccE: 53.17 -> 49.67 | Hurt |
| replace temporal emotion distribution recovery with pixel targets (ablation) | MART on Ekman6 · Ekman6 | AccE: 53.17 -> 48.58 | Hurt |
Limitations
- Relies on transcripts extracted from video and on the quality of the affective lexicon.
- Gains over the strongest SSL baselines are modest in several metrics (e.g., a 3.19% margin in the paper's own summary).
Experimental setup
Datasets: VideoEmotion8 Ekman6 PERR IEMOCAP AffWild2 SenticNet-5
Benchmarks: VideoEmotion8 Ekman6 PERR IEMOCAP AffWild2
Metrics: accuracy F1 score
Baselines: MBT MoCov3 AudioMAE SSAST MaskFeat MAEst TCL MILAN VideoMAE MGMAE
Builds on: MAE VideoMAE MBT MaskFeat SenticNet
Closest papers by meaning
- VidEmo: Affective-Tree Reasoning for Emotion-Centric Video Foundation Models (NeurIPS 2025 · similarity 0.87)
- AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models (ICML 2025 · similarity 0.86)
- Bridge Then Begin Anew: Generating Target-Relevant Intermediate Model for Source-Free Visual Emotion Adaptation (AAAI 2025 · similarity 0.84)
- Facial-R1: Aligning Reasoning and Recognition for Facial Emotion Analysis (AAAI 2026 · similarity 0.84)
- $E^3$: Exploring Embodied Emotion Through A Large-Scale Egocentric Video Dataset (NeurIPS 2024 · similarity 0.84)
- Enhancing Multimodal Affective Analysis with Learned Live Comment Features (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.