Research map / Video / Video segmentation and tracking
AVF-MAE++: Scaling Affective Video Facial Masked Autoencoders via Efficient Audio-Visual Self-Supervised Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Scales audio-visual masked autoencoders for affective video facial analysis with dual masking, an improved LGI-Former encoder, an iterative audio-visual correlation module (IAV-CL), and three-stage progressive semantics injection, pre-trained on a 1.36M-clip hybrid dataset.
Problem
Affective video facial analysis (AVFA) suffers from scarce labeled data, and the scaling behavior of audio-visual masked autoencoder pre-training and its intra-/inter-modal correlation modeling are under-explored.
Approach
Extends HiCMAE with an audio-visual dual masking strategy for efficient decoding, a region-based local-global interaction encoder (improved LGI-Former), an IAV-CL module with parallel dense interactions and iterative evolutionary refinement, and a progressive semantics injection training pipeline (self-supervised pre-training, supervised post-pretraining on labeled hybrid data, then fine-tuning).
Key results
- AVF-MAE++ (H) reaches 60.24% WAR on MAFW (11-class), the first method above 60% WAR on that dataset, versus 58.45% for MMA-DFER.
- On MER-MULTI, AVF-MAE++ (H) achieves 72.26 WA-F1, above the 71.33 of HiCMAE and 70.32 of the best combined baseline.
- On the MER task, AVF-MAE++ shows a 10.21% UF1 gain over HTNet on CAS(ME)3 (67.88 UF1 for Large).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Scale the model from Base to Huge (ablation) | AVF-MAE++ · MAFW (11-class) | WAR: 57.50 -> 60.24 | Helped |
| Use dual masking with the improved modality encoder instead of HiCMAE's masking and vanilla ViT (ablation) | AVF-MAE++ (B) · MAFW | WAR: 56.17 -> 56.12 | No clear effect |
| Use dual masking and vanilla LGI-Former (ablation) | AVF-MAE++ (B) · MAFW | speedup: 1.49× | Helped |
| Add DiER Units and HAFE Layer (ablation) | AVF-MAE++ (B) · MAFW | WAR: 56.31 -> 57.02 | Helped |
| Add DiER Units (ablation) | AVF-MAE++ (B) · MAFW | WAR: 56.31 -> 56.81 | Helped |
| Add the HAFE Layer (ablation) | AVF-MAE++ (B) · IEMOCAP | WAR: 69.33 -> 69.86 | Helped |
| Set the number of DiER Units to 2 instead of 1 (ablation) | AVF-MAE++ (L) · MAFW | WAR: 56.83 -> 57.69 | Helped |
| Set the number of DiER Units to 4 instead of 2 (ablation) | AVF-MAE++ (L) · MAFW | WAR: 57.69 -> 57.05 | Hurt |
| Scale pre-training data from VoxCeleb2-dev to Unlabeled Hybrid (ablation) | AVF-MAE++ (B) · MAFW | WAR: 56.64 -> 57.02 | Helped |
| Scale pre-training data from VoxCeleb2-dev to Unlabeled Hybrid (ablation) | AVF-MAE++ (L) · MER24-T&V | WA-F1: 69.02 -> 70.32 | Helped |
| Use progressive semantics injection (ablation) | AVF-MAE++ · MAFW and MER24-T&V | performance | Helped |
| Scale from Large to Huge (ablation) | AVF-MAE++ · RAVDESS | WAR: 87.57 -> 87.57 | Hurt |
Surprising findings
- Performance gains from Large to Huge are much smaller on some datasets and sharper declines appear on small MER datasets, indicating overfitting for large models.
- The dual masking plus improved encoder yields a 1.46x speedup with competitive performance, while the vanilla LGI-Former variant showed lower WAR on MAFW than HiCMAE.
Limitations
- Overfitting on small downstream datasets, especially for large models.
- Performance saturates on some datasets as model capacity grows.
- Data scaling is limited compared to general-domain work; amplified AVFA data is unexplored.
Experimental setup
Datasets: CN-Celeb MER2024 VoxCeleb2-dev AV-Speech CelebV-HQ MAFW DFEW MER-MULTI MER24-T&V IEMOCAP CREMA-D RAVDESS
Benchmarks: MAFW DFEW MER-MULTI MER24-T&V CREMA-D RAVDESS MSP-IMPROV IEMOCAP AVCAffe Werewolf-XL SAMM CASME II
Metrics: UAR WAR WA-F1 UF1 PCC
Baselines: HiCMAE UniLearn MAE-DFER SVFAP DFER-CLIP FineCLIPER HTNet VQ-MAE-AV HuBERT WavLM-Plus
Builds on: HiCMAE VideoMAE V2 LGI-Former MAE HCMCL loss
Closest papers by meaning
- MART: Masked Affective RepresenTation Learning via Masked Temporal Distribution Distillation (CVPR 2024 · similarity 0.83)
- Extending Video Masked Autoencoders to 128 frames (NeurIPS 2024 · similarity 0.83)
- PrefAce: Face-Centric Pretraining with Self-Structure Aware Distillation (AAAI 2024 · similarity 0.83)
- VideoMAC: Video Masked Autoencoders Meet ConvNets (CVPR 2024 · similarity 0.82)
- EVEREST: Efficient Masked Video Autoencoder by Removing Redundant Spatiotemporal Tokens (ICML 2024 · similarity 0.82)
- The Dynamic Duo of Collaborative Masking and Target for Advanced Masked Autoencoder Learning (AAAI 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.