Research map / Vision-language and multimodal / Cross-modal learning

Cross-modal Representation Flattening for Multi-modal Domain Generalization

NeurIPS 2024 · Vision-language and multimodal › Cross-modal learning · approach: Unimodal-to-multimodal representation transfer · 6 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Cross-Modal Representation Flattening (CMRF), which flattens high-loss regions between modalities in a shared representation space by distilling from moving-averaged teacher interpolations of paired multi-modal representations, improving multi-modal domain generalization on EPIC-Kitchens and HAC.

Problem

Multi-modal domain generalization (MMDG): models trained on multi-modal source domains must generalize to unseen target domains with the same modalities; SAM-style flat-minima methods give limited gains in this setting due to modality competition and discrepant uni-modal flatness.

Approach

Optimize flatness in a shared representation-space loss landscape rather than parameter space; build cross-modal representation mixup from an SMA teacher, distill the interpolations into each modality's student representation with adaptive per-modality weights based on teacher validation accuracy, plus a multi-modal supervised contrastive loss.

Key results

• On HAC multi-source video-audio-flow DG, CMRF reaches 72.44% average vs 70.13% for SimMMDG and 65.07% for Base.
• On EPIC-Kitchens video-audio multi-source DG, CMRF averages 63.91% vs 59.63% for Base and 61.19% for SAM.
• Uni-modal audio accuracy from CMRF-trained network on EPIC-Kitchens reaches 43.13% vs 40.04% for uni-audio training.

Measured effects reported in the paper

Surprising findings

• Uni-modal generalization of a naively jointly trained multi-modal network is worse than uni-modal training, indicating modality competition.
• SAM improves the stronger modality but gives marginal or negative benefit to the weaker modality.

Limitations

• Adaptive weights need uni-modal validation accuracy from the teacher, which becomes costly as the validation set scales.
• Error bars not reported due to computational expense.
• Evaluated only on two video-based benchmarks with video, audio, and optical flow modalities.

Experimental setup

Datasets: EPIC-Kitchens Human-Animal-Cartoon (HAC)

Benchmarks: EPIC-Kitchens HAC

Metrics: Top-1 accuracy

Baselines: Base SAM SAGM SWAD EoA RNA-Net SimMMDG Gradient Blending OGM-GE PMR

Builds on: Sharpness-aware minimization Manifold Mixup SWAD EoA SimMMDG

Closest papers by meaning

• Modality-Aware SAM: Sharpness-Aware-Minimization Driven Gradient Modulation for Harmonized Multimodal Learning (NeurIPS 2025 · similarity 0.88)
• Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data (ICLR 2024 · similarity 0.88)
• Same Concept, Different Directions: Cross-Modal Feature Heterogeneity in Sparse Autoencoders (NeurIPS 2026 · similarity 0.88)
• Seeking Consistent Flat Minima for Better Domain Generalization via Refining Loss Landscapes (CVPR 2025 · similarity 0.87)
• Beyond Modality Collapse: Representation Blending for Multimodal Dataset Distillation (NeurIPS 2025 · similarity 0.87)
• Learning Modality Knowledge Alignment for Cross-Modality Transfer (ICML 2024 · similarity 0.87)