Research map / Vision-language and multimodal / Cross-modal learning

Do More Modalities Always Help? A Geometric Perspective on Missing-Modality Robustness

NeurIPS 2026 · Vision-language and multimodal › Cross-modal learning · approach: Multimodal fusion with missing modalities · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that multimodal-trained models can underperform unimodal models when one modality is missing at inference, links this to principal-subspace rotations in cross-modal layers, and proposes Geodesic Unlearning, a one-layer Grassmannian geodesic subspace edit that improves missing-image F1.

Problem

Models trained on full multimodal inputs can underperform unimodal models when one modality is entirely absent at deployment; existing missing-modality methods rely on recovery or adaptive fusion and ignore harmful internal cross-modal dependencies.

Approach

Characterizes the degradation via principal-angle rotations of input subspaces in cross-modal interaction layers and task-aware margin harm, then applies a post-training Grassmannian geodesic edit to one selected layer that rotates its principal input subspace toward a unimodal reference while keeping the output factor, singular values, and residual.

Key results

• On TI-T F1, GU reaches 60.88 on Hateful Memes (+2.38 over best baseline), 81.41 on IU-XRay (+1.27), and 74.41 on SNLI-VE (+2.21), editing one layer post-training.
• Across 27 architecture-dataset settings, mean principal angle correlates with missing-modality gap at r=0.91 (R^2=0.82) after within-family z-scoring.
• GU adds no inference parameters or FLOPs; its edit takes 230-858 s versus post-hoc LoRA's 1812-2000 s, and total GU cost including reference training is below Dual CE's.

Surprising findings

• Models trained on full modalities underperform unimodal text-only models under missing-image inference in all 27 architecture-dataset settings.
• A simple null-token pathway remains competitive with multimodal-specific missing-modality methods (SMIL, Flex-MoE, DyMo) in this bimodal complete-absence setting.

Limitations

• Evaluation limited to classification tasks, including multiple-choice VQA; no free-form generation.
• Limited to base-scale models; no large-scale experiments due to resource constraints.
• Gains are smaller for CLIP-style and BLIP-2 architectures where dependencies are more distributed.

Experimental setup

Datasets: Hateful Memes SNLI-VE IU-XRay A-OKVQA

Benchmarks: Hateful Memes SNLI-VE IU-XRay A-OKVQA

Metrics: F1 missing-modality gap unimodal robustness gap principal angle

Baselines: SMIL Flex-MoE DyMo MoRA OGM-GE LoRA Dual CE layer interpolation linear subspace interpolation null-token

Builds on: SVD-based low-rank structure Grassmann geodesic distance machine unlearning LoRA

Closest papers by meaning

• Mitigating Pervasive Modality Absence Through Multimodal Generalization and Refinement (AAAI 2025 · similarity 0.90)
• Knowledge Bridger: Towards Training-Free Missing Modality Completion (CVPR 2025 · similarity 0.90)
• RedCore: Relative Advantage Aware Cross-Modal Representation Learning for Missing Modalities with Imbalanced Missing Rates (AAAI 2024 · similarity 0.89)
• Gradient-Guided Modality Decoupling for Missing-Modality Robustness (AAAI 2024 · similarity 0.89)
• MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data (NeurIPS 2026 · similarity 0.89)
• Probabilistic Conformal Distillation for Enhancing Missing Modality Robustness (NeurIPS 2024 · similarity 0.89)