Research map / Vision-language and multimodal / Cross-modal learning
Mitigating Pervasive Modality Absence Through Multimodal Generalization and Refinement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MGR, a ViLT-based multimodal framework that adds unimodal features, mixed-context prompts, and graph-pooling redistribution to iteratively refine biased tokens under pervasive modality absence.
Problem
Multimodal models degrade when modalities are absent, especially when absence is pervasive across both training and inference, and existing reconstruction methods need supervision that is rarely available.
Approach
Learn generalized multimodal representations by fusing a ViLT multimodal Transformer output with unimodal Transformer features, then iteratively identify biased tokens using mixed-context prompts (absence status plus feature/correlation context) and redistribute their information to unbiased tokens via graph-based pooling instead of discarding them.
Key results
- Under absence prevalence κ=0.7, MGR reaches 69.42 average F1 across four datasets versus 68.48 for the best prior method MSPs.
- Under κ=1 on three datasets, MGR averages 70.26 versus 69.34 for MAPs.
- Ablation shows removing the mixed-context prompt drops κ=0.7 average from 68.40 to 67.01.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MGR instead of the strongest reported baseline | ViLT on CMU-MOSEI · CMU-MOSEI (κ=0.7) | F1 score: 84.90 /0.77 -> 85.27 /0.45 | Helped |
| Use MGR instead of the strongest reported baseline | ViLT on CMU-MOSI · CMU-MOSI (κ=0.7) | Accuracy: 71.55 /1.67 -> 71.07 /1.84 | Hurt |
| Use MGR instead of the strongest reported baseline | ViLT on CMU-MOSI · CMU-MOSI (κ=1) | F1 score: 66.73 /1.90 -> 67.46 /1.86 | Helped |
| Use MGR instead of the strongest reported baseline | ViLT on CMU-MOSEI · CMU-MOSEI (κ=1) | Accuracy: 76.08 /0.82 -> 76.53 /0.31 | Helped |
| Add mixed-context prompting instead of MAPs prompts (ablation) | MGR on CMU-MOSI · CMU-MOSI (κ=0.7) | F1 score: 66.92 -> 69.23 | Helped |
| Add mixed-context prompting instead of MAPs prompts (ablation) | MGR on CMU-MOSEI · CMU-MOSEI (κ=0.7) | F1 score: 83.44 -> 85.27 | Helped |
| Add the diversity loss Ldiv (ablation) | MGR on CMU-MOSEI · CMU-MOSEI (κ=0.7) | F1 score: 83.53 -> 85.27 | Helped |
| Add unimodal enhancement features hu (ablation) | MGR on AV-MNIST · AV-MNIST (κ=0.7) | F1 score: 53.73 -> 58.15 | Helped |
| Increase absence prevalence from κ=0.7 to κ=1 | MGR on CMU-MOSI · CMU-MOSI | F1 score: 69.23 /2.29 -> 67.46 /1.86 | Hurt |
Limitations
- Gains over the strongest prior method (MSPs) are modest on some datasets, e.g. CMU-MOSI accuracy is lower than MSPs at κ=0.7 (71.07 vs 71.55).
- Evaluated only on video/multimodal benchmarks with three or two modalities; generality to other multimodal models is left to future work.
Experimental setup
Datasets: CMU-MOSI CMU-MOSEI UR-FUNNY AV-MNIST
Benchmarks: CMU-MOSI CMU-MOSEI UR-FUNNY AV-MNIST
Metrics: F1 score accuracy
Baselines: ViLT MTL MaskMentor MaPLe MAPs MSPs
Builds on: ViLT MAPs MSPs distance correlation graph-based message passing
Closest papers by meaning
- Do More Modalities Always Help? A Geometric Perspective on Missing-Modality Robustness (NeurIPS 2026 · similarity 0.90)
- Asymmetric Reinforcing Against Multi-Modal Representation Bias (AAAI 2025 · similarity 0.90)
- Retrieval-Augmented Dynamic Prompt Tuning for Incomplete Multimodal Learning (AAAI 2025 · similarity 0.88)
- MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data (NeurIPS 2026 · similarity 0.88)
- Deep Correlated Prompting for Visual Recognition with Missing Modalities (NeurIPS 2024 · similarity 0.88)
- Knowledge Bridger: Towards Training-Free Missing Modality Completion (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.