Research map / Vision-language and multimodal / Cross-modal learning
MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MMPareto, a gradient-integration method for multitask-like multimodal learning that combines Pareto (MGDA-style) min-norm weighting with magnitude enhancement to keep unimodal gradients from conflicting with the multimodal loss while boosting SGD noise for better generalization.
Problem
Multimodal learning with added unimodal losses (to fix imbalanced modality utilization) creates gradient conflicts between the multimodal and unimodal objectives on shared unimodal encoders; conventional Pareto integration underperforms the uniform baseline in this setting.
Approach
Observes that multimodal gradients have smaller magnitude and batch covariance than unimodal ones, so conventional Pareto weights the multimodal gradient more and yields sharper minima. MMPareto keeps the Pareto direction (or equal weights when non-conflicting) and rescales the integrated gradient to the uniform-baseline magnitude times a factor gamma, enhancing SGD noise for flatter minima.
Key results
- On CREMA-D with CNN late fusion, MMPareto reaches 75.13% accuracy vs 71.10% for the uniform baseline and 72.01% for the best prior method (G-Blending).
- On Colored-and-gray-MNIST, MMPareto achieves 81.88% accuracy vs 75.68% for the uniform baseline; conventional Pareto drops to 62.00%.
- Conventional Pareto underperforms the uniform baseline on multimodal tasks (e.g., 66.64% vs 68.31% on Kinetics Sounds), while MMPareto reaches 70.13%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MMPareto instead of G-Blending | ResNet-18 multimodal model · CREMA-D | Acc: 72.01 -> 75.13 | Helped |
| Use MMPareto instead of Uniform baseline | ResNet-18 multimodal model · CREMA-D | Acc audio: 63.44 -> 65.46 | Helped |
| Use MMPareto instead of Unimodal pre-trained & fine-tune | ResNet-18 multimodal model · CREMA-D | Acc video: 60.22 -> 55.24 | Hurt |
| Use MMPareto instead of G-Blending | ResNet-18 multimodal model · Kinetics Sounds | Acc: 68.90 -> 70.13 | Helped |
| Use MMPareto instead of Audio-only training | ResNet-18 audio encoder · Kinetics Sounds | Acc audio: 53.63 -> 56.40 | Helped |
| Use MMPareto instead of Unimodal pre-trained & fine-tune | ResNet-18 multimodal model · Kinetics Sounds | Acc video: 50.07 -> 53.05 | Helped |
| Use MMPareto instead of G-Blending | MBT · CREMA-D from scratch | Acc: 46.38 -> 48.66 | Helped |
| Use MMPareto instead of G-Blending | MBT · CREMA-D from scratch | macro F1: 45.16 -> 48.17 | Helped |
| Use MMPareto instead of G-Blending | MBT · CREMA-D with pretrain | Acc: 69.91 -> 70.43 | Helped |
| Use MMPareto instead of Uniform baseline | MBT · CREMA-D with pretrain | macro F1: 70.11 -> 71.17 | Helped |
| Use MMPareto instead of G-Blending | MBT · Kinetics Sounds from scratch | Acc: 44.69 -> 45.20 | Helped |
| Use MMPareto instead of G-Blending | MBT · Kinetics Sounds from scratch | macro F1: 44.19 -> 45.26 | Helped |
Surprising findings
- Conventional Pareto integration, which is successful in standard multi-task learning, falls below the uniform baseline in multimodal settings.
- Several prior imbalanced multimodal methods (e.g., OGM-GE on CREMA-D with MBT) perform worse than the one-joint-loss baseline under transformer cross-modal interaction.
Limitations
- Convergence analysis relies on relatively ideal assumptions.
- Gradient covariance and correlation between losses are not used in the weighting; only magnitude is adjusted.
- Gamma rescale factor is a fixed hyperparameter (1.5) chosen empirically.
Experimental setup
Datasets: CREMA-D Kinetics Sounds Colored-and-gray-MNIST ModelNet40 CMU-MOSI MultiMNIST NYUv2
Benchmarks: CREMA-D Kinetics Sounds Colored-and-gray-MNIST ModelNet40 CMU-MOSI MultiMNIST NYUv2
Metrics: accuracy macro F1 unimodal accuracy mIoU pixel accuracy abs error rel error
Baselines: Uniform baseline Conventional Pareto G-Blending OGM-GE Greedy PMR AGM GradNorm PCGrad MetaBalance
Builds on: Pareto MTL (MGDA) OGM-GE MBT PCGrad
Closest papers by meaning
- Multimodal Representation Learning by Alternating Unimodal Adaptation (CVPR 2024 · similarity 0.90)
- Improving Multimodal Learning Balance and Sufficiency through Data Remixing (ICML 2025 · similarity 0.88)
- Asymmetric Reinforcing Against Multi-Modal Representation Bias (AAAI 2025 · similarity 0.87)
- Classifier-guided Gradient Modulation for Enhanced Multimodal Learning (NeurIPS 2024 · similarity 0.87)
- MokA: Multimodal Low-Rank Adaptation for MLLMs (NeurIPS 2025 · similarity 0.86)
- Rethinking Multimodal Learning from the Perspective of Mitigating Classification Ability Disproportion (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.