Research map / Vision-language and multimodal / Cross-modal learning
An Optimal Transport-based Latent Mixer for Robust Multi-modal Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an optimal transport-based mixer (OTM) that aligns and augments unaligned, distributed multi-modal latent codes via a stochastic fused Gromov-Wasserstein barycenter, using per-agent Wasserstein autoencoders without sharing raw data.
Problem
Multi-modal learning when data from different modalities are unaligned and scattered across agents that cannot share raw data, so single-modality agents may be under-informed and non-robust.
Approach
Each agent trains a Wasserstein autoencoder on its own modality; a central server computes a stochastic fused Gromov-Wasserstein barycenter over latent codes and returns randomly masked pushforward mixtures, which agents use to augment their local latent codes in the reconstruction and supervised losses.
Key results
- On unaligned Caltech7 clustering, MWAE+OTM reaches purity 0.5788 versus 0.3568 for GWMAC and 0.3112 for MVC-UM.
- On CMU-MOSI classification with late fusion, OTM raises accuracy from 0.4922 to 0.5062 under noisy test data; on MUJOCO regression with tensor fusion, MSE drops from 1.583e-3 to 1.369e-3 with OTM.
- Under noise, OTM improves MUJOCO tensor fusion MAE from 0.1971 to 0.1906, but single-modality results show OTM degrades some significant modalities.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add OTM to MWAE | MWAE clustering · Caltech7, unaligned | purity: 0.3568 -> 0.5788 | Helped |
| Add OTM to MWAE | MWAE clustering · ORL, unaligned | purity: 0.5431 -> 0.6550 | Helped |
| Add OTM to MWAE | MWAE clustering · Movies, unaligned | purity: 0.1928 -> 0.2925 | Helped |
| Add OTM to MWAE | MWAE clustering · Prokaryotic, unaligned | purity: 0.5479 -> 0.5438 | No clear effect |
| Add OTM to Late fusion | Late fusion classification · AV-MNIST | accuracy: 0.7295 -> 0.7316 | Helped |
| Add OTM to MI matrix | MI matrix classification · ENRICO | accuracy: 0.4815 -> 0.5034 | Helped |
| Add OTM to Late fusion | Late fusion classification · CMU-MOSI | accuracy: 0.5194 -> 0.5368 | Helped |
| Add OTM to LRTF | LRTF classification · CMU-MOSI | accuracy: 0.5245 -> 0.5327 | Helped |
| Add OTM to MFM | MFM classification · CMU-MOSI | accuracy: 0.5391 -> 0.5410 | Helped |
| Add OTM to Late fusion | Late fusion regression · CMU-MOSI | MAE: 1.3710 -> 1.3630 | Helped |
| Add OTM to Tensor fusion | Tensor fusion regression · MUJOCO | MAE: 1.583×10−3 -> 1.369×10−3 | Helped |
| Add OTM to LRTF | LRTF classification · CMU-MOSI, modality 1 | accuracy: 0.5190 -> 0.5131 | Hurt |
Surprising findings
- OTM can degrade performance on some informative modalities (e.g., on Prokaryotic and CMU-MOSI single-modality results), which the authors attribute to introduced noise.
Limitations
- Performance degrades for some informative modalities, as the mixing introduces noise.
- Computational cost of FGW barycenter computation is a bottleneck; authors suggest sliced FGW as future speedup.
- Evaluated only on small-scale benchmarks with uniform-distribution assumptions and synthetic unalignment by random permutation.
Experimental setup
Datasets: Caltech7 ORL Movies Prokaryotic ENRICO CMU-MOSI AV-MNIST MUJOCO
Benchmarks: Caltech7 ORL Movies Prokaryotic ENRICO CMU-MOSI AV-MNIST MUJOCO
Metrics: clustering purity classification accuracy MAE MSE
Baselines: MCCA DCCAE AttnAE MVKSC MultiNMF GWMAC MVC-UM Late fusion MFM Tensor Fusion LRTF MI matrix
Builds on: Wasserstein autoencoder fused Gromov-Wasserstein distance Mixup GWMAC MultiBench
Closest papers by meaning
- Quantifying and Enhancing Multi-modal Robustness with Modality Preference (ICLR 2024 · similarity 0.84)
- On the Comparison between Multi-modal and Single-modal Contrastive Learning (NeurIPS 2024 · similarity 0.83)
- Generative Modeling of Class Probability for Multi-Modal Representation Learning (CVPR 2025 · similarity 0.83)
- Multi-Modal Latent Variables for Cross-Individual Primary Visual Cortex Modeling and Analysis (AAAI 2025 · similarity 0.83)
- Multi-modal contrastive learning adapts to intrinsic dimensions of shared latent variables (NeurIPS 2025 · similarity 0.83)
- ReconBoost: Boosting Can Achieve Modality Reconcilement (ICML 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.