Research map / Label-efficient and robust learning / Recognition and long-tail
CARAT: Contrastive Feature Reconstruction and Aggregation for Multi-Modal Multi-Label Emotion Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CARAT, a multi-modal multi-label emotion recognition framework that combines contrastive, reconstruction-based fusion of label-specific modality features with shuffle-based aggregation, reporting state-of-the-art results on CMU-MOSEI and M3ED.
Problem
Multi-modal multi-label emotion recognition (MMER) requires capturing modality-specific and label-specific discriminative features, plus modeling label-to-label and modality-to-label dependencies, which unified fusion representations fail to handle.
Approach
Extracts label-wise attention features per modality, fuses them via a two-level cross-modal reconstruction network whose intrinsic prototypes are trained with supervised contrastive learning, uses a max-pooling-like network to select the most relevant modality per label, and applies sample-wise and modality-wise shuffling to create new label co-occurrence combinations for aggregation.
Key results
- On CMU-MOSEI aligned, CARAT reaches Micro-F1 0.581 and Acc 0.494, versus 0.569 for TAILOR and AMP.
- On M3ED, CARAT reaches Micro-F1 0.788 and Acc 0.664, outperforming AMP (0.782 Micro-F1, 0.654 Acc).
- Ablation on aligned CMU-MOSEI shows removing both shuffle processes drops Micro-F1 from 0.581 to 0.574 and removing both reconstruction levels drops it to 0.555.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CARAT instead of AMP | MMER · CMU-MOSEI aligned | Micro-F1: 0.569 -> 0.581 | Helped |
| Use CARAT instead of AMP | MMER · CMU-MOSEI unaligned | Micro-F1: 0.535 -> 0.544 | Helped |
| Use CARAT instead of AMP | MMER · M3ED | Micro-F1: 0.782 -> 0.788 | Helped |
| Add aggregated features to most-relevant-modality features (ablation) | CARAT · CMU-MOSEI aligned | Micro-F1: 0.563 -> 0.569 | Helped |
| Add most-relevant-modality features to aggregated features (ablation) | CARAT · CMU-MOSEI aligned | Micro-F1: 0.565 -> 0.569 | Helped |
| Add contrastive learning loss (ablation) | CARAT · CMU-MOSEI aligned | Micro-F1: 0.571 -> 0.581 | Helped |
| Add encoding and decoding (ablation) | CARAT · CMU-MOSEI aligned | Micro-F1: 0.569 -> 0.581 | Helped |
| Add reconstruction loss (ablation) | CARAT · CMU-MOSEI aligned | Micro-F1: 0.573 -> 0.581 | Helped |
| Add first-level reconstruction (ablation) | CARAT · CMU-MOSEI aligned | Micro-F1: 0.572 -> 0.581 | Helped |
| Add second-level reconstruction (ablation) | CARAT · CMU-MOSEI aligned | Micro-F1: 0.566 -> 0.581 | Helped |
| Add both reconstruction levels (ablation) | CARAT · CMU-MOSEI aligned | Micro-F1: 0.555 -> 0.581 | Helped |
| Add sample-wise shuffle (ablation) | CARAT · CMU-MOSEI aligned | Micro-F1: 0.575 -> 0.581 | Helped |
Limitations
- Gains over the strongest baselines are small (about 0.01 Micro-F1 on CMU-MOSEI aligned), and results are from single runs with no variance reported.
- Many hyperparameters (loss weights, queue size, layer counts) are tuned on the CMU-MOSEI test-setting validation, with limited analysis on M3ED.
Experimental setup
Datasets: CMU-MOSEI M3ED
Benchmarks: CMU-MOSEI M3ED
Metrics: Accuracy Micro-F1 Precision Recall
Baselines: BR LP CC SGM LSAN ML-GCN MulT MISA MMS2S HHMPN TAILOR AMP
Builds on: supervised contrastive learning label-specific representations MulT MISA
Closest papers by meaning
- Multi-to-Single: Reducing Multimodal Dependency in Emotion Recognition Through Contrastive Learning (AAAI 2025 · similarity 0.89)
- Beyond Single Emotion: Multi-label Approach to Conversational Emotion Recognition (AAAI 2025 · similarity 0.86)
- EMOE: Modality-Specific Enhanced Dynamic Emotion Experts (CVPR 2025 · similarity 0.85)
- AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models (ICML 2025 · similarity 0.85)
- OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition (ICML 2025 · similarity 0.85)
- Navigating Label Ambiguity for Facial Expression Recognition in the Wild (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.