Research map / Vision-language and multimodal / Multimodal LLMs

Diagnosing and Mitigating Modality Interference in Multimodal Large Language Models

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Diagnoses "modality interference" in MLLMs via perturbation-based causal analysis and proposes a fine-tuning framework combining heuristic and masked adversarial embedding perturbations with KL/JS consistency regularization, improving robustness and VQA accuracy.

Problem

MLLMs are fragile when a non-essential modality carries spurious or misleading content (e.g., misleading text in image classification, irrelevant images in text QA), so predictions are distorted by irrelevant modalities.

Approach

Perturb the task-irrelevant modality (heuristic text/image perturbations plus masked, raw-gradient adversarial perturbations on embeddings) during fine-tuning and enforce output consistency between clean and perturbed inputs, framed as a causal intervention to block spurious pathways.

Key results

• On LLaVA-1.5-7B, the proposed method keeps Caltech-101 perturbed accuracy at 99.0% (vs 57.4% vanilla) and raises VQA overall accuracy from 63.8% to 66.8%.
• On InstructBLIP-7B, Caltech-101 perturbed accuracy rises from 17.5% to 98.3%, with VQA overall accuracy 64.8% vs 56.4% vanilla.
• On Qwen2.5-VL-3B, VQA overall accuracy improves from 72.5% to 76.4%, while unimodal perturbed accuracy stays near clean accuracy.

Surprising findings

• Heuristic augmentation alone fails to generalize to unseen OCR-style noise and can degrade clean Food-101 accuracy on InstructBLIP-7B (67.9% to 55.5%).
• Chain-of-thought prompting and the reasoning model R1-OneVision-7B do not mitigate modality interference and can degrade perturbed accuracy.

Limitations

• Analysis limited to multiple-choice formats for main results; free-form evaluation only in appendix.
• Some perturbed results remain weak for specific variants (e.g., ADV on InstructBLIP-7B falls to 72.2% on Mini-ImageNet perturbed), and the adversarial component adds training overhead.

Experimental setup

Datasets: Mini-ImageNet Caltech-101 OpenBookQA MMLU LLaVA-Instruct TextCaps

Benchmarks: Mini-ImageNet Caltech-101 OpenBookQA MMLU ScienceQA-IMG MM-Bench-EN Seed-Bench-IMG Food-101 ARC Challenge

Metrics: accuracy prediction change rate

Baselines: CoT prompting VLMClassifier-1 VLMClassifier-2 I-MoF R1-OneVision-7B FFT with D^VQA

Builds on: PGD LLaVA VLMClassifier Chain-of-Thought prompting Pearl causal framework

Closest papers by meaning

• XModBench: Benchmarking Cross-Modal Capabilities and Consistency in Omni-Language Models (ICLR 2026 · similarity 0.90)
• Compose and Fuse: Revisiting the Foundational Bottlenecks in Multimodal Reasoning (ICLR 2026 · similarity 0.90)
• Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs (NeurIPS 2026 · similarity 0.90)
• DeepAlign: Mitigating Modality Conflict through Modality-Specific Alignment (CVPR 2026 · similarity 0.89)
• Beyond task performance: evaluating and reducing the flaws of large multimodal models with in-context-learning (ICLR 2024 · similarity 0.89)
• TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs (NeurIPS 2026 · similarity 0.89)