Research map / Vision-language and multimodal / Cross-modal learning
Multi-to-Single: Reducing Multimodal Dependency in Emotion Recognition Through Contrastive Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Multi-to-Single (M2S), a contrastive pre-training framework on paired unlabeled multimodal physiological data that lets emotion recognition run from a single modality (EEG, EYE, ECG, PPS) by freezing the encoder and tuning a classifier.
Problem
Emotion recognition from physiological signals usually needs multiple modalities at test time, but in practice some modalities (especially EEG) are hard to collect reliably, and existing cross-modal methods generalize poorly across modality pairs.
Approach
Pre-train two encoders per modality (emotion-related transformer with STS attention and emotion-independent MLP) with CLUB mutual-information minimization, reconstruction, InfoNCE contrastive alignment, and a Multi-to-Multi CPC module that predicts future embeddings of each modality from the current embeddings of all modalities; at fine-tuning, freeze the ER encoder and train only a classifier on
Key results
- On SEED, EEG→EYE cross-modal balanced accuracy of M2S is 91.72% vs 85.23% for the best prior baseline (CLIP), and EYE→EEG reaches 91.92%.
- On SEED-IV, M2S EEG-only fine-tuning reaches 85.47% balanced accuracy vs 83.81% for MAET (supervised multimodal), and on SEED-V 81.22% vs 76.53% for MAET.
- On DREAMER, M2S EEG→ECG balanced accuracy for arousal is 93.52% vs 89.28% for CLIP baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add reconstruction loss (ablation) | Multi-to-Single · SEED, EEG→EYE | balanced accuracy (%): 86.90/10.34 -> 91.72/9.53 | Helped |
| add CPC loss (ablation) | Multi-to-Single · SEED, EEG→EYE | balanced accuracy (%): 83.90/10.36 -> 91.72/9.53 | Helped |
| add CLUB loss (ablation) | Multi-to-Single · SEED, EEG→EYE | balanced accuracy (%): 87.63/9.52 -> 91.72/9.53 | Helped |
| add contrastive loss (ablation) | Multi-to-Single · SEED, EEG→EYE | balanced accuracy (%): 76.94/8.30 -> 91.72/9.53 | Helped |
| add reconstruction loss (ablation) | Multi-to-Single · SEED-IV, EEG→EYE | balanced accuracy (%): 78.41/12.99 -> 84.70/11.35 | Helped |
| add CPC loss (ablation) | Multi-to-Single · SEED-IV, EEG→EYE | balanced accuracy (%): 78.61/13.33 -> 84.70/11.35 | Helped |
| add CLUB loss (ablation) | Multi-to-Single · SEED-IV, EEG→EYE | balanced accuracy (%): 82.34/11.72 -> 84.70/11.35 | Helped |
| add contrastive loss (ablation) | Multi-to-Single · SEED-IV, EEG→EYE | balanced accuracy (%): 73.18/10.22 -> 84.70/11.35 | Helped |
| add reconstruction loss (ablation) | Multi-to-Single · SEED-V, EEG→EYE | balanced accuracy (%): 73.53/12.61 -> 82.77/11.24 | Helped |
| add CPC loss (ablation) | Multi-to-Single · SEED-V, EEG→EYE | balanced accuracy (%): 77.12/13.11 -> 82.77/11.24 | Helped |
| add CLUB loss (ablation) | Multi-to-Single · SEED-V, EEG→EYE | balanced accuracy (%): 73.23/11.35 -> 82.77/11.24 | Helped |
| add contrastive loss (ablation) | Multi-to-Single · SEED-V, EEG→EYE | balanced accuracy (%): 71.27/12.50 -> 82.77/11.24 | Helped |
Surprising findings
- Single-modality fine-tuning on the target modality performs slightly better than cross-modal testing after pre-training.
Limitations
- Evaluated only on subject-dependent settings with a fixed set of public datasets
- Many hyperparameters (loss coefficients, time window, prediction steps) are tuned on the test-visible heatmaps
Experimental setup
Datasets: SEED SEED-IV SEED-V DEAP DREAMER
Benchmarks: SEED SEED-IV SEED-V DEAP DREAMER
Metrics: balanced accuracy Cohen's kappa F1 score
Baselines: BDAE-regressor BDAE-cGAN CLIP ECO-FET Uni-Code BDAE ETF VigilanceNet MAET
Builds on: Contrastive Predictive Coding InfoNCE CLUB Contrastive Multiview Coding CLIP
Closest papers by meaning
- EMOE: Modality-Specific Enhanced Dynamic Emotion Experts (CVPR 2025 · similarity 0.90)
- CARAT: Contrastive Feature Reconstruction and Aggregation for Multi-Modal Multi-Label Emotion Recognition (AAAI 2024 · similarity 0.89)
- AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models (ICML 2025 · similarity 0.89)
- Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning (NeurIPS 2024 · similarity 0.88)
- OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition (ICML 2025 · similarity 0.88)
- MODA: MOdular Duplex Attention for Multimodal Perception, Cognition, and Emotion Understanding (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.