Research map / Vision-language and multimodal / Cross-modal learning

MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data

NeurIPS 2026 · Vision-language and multimodal › Cross-modal learning · approach: Unimodal-to-multimodal representation transfer

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Modality Self-Play (MSP), which trains an audio-vision LLM (Qwen3-8B + LoRA) using single real modalities paired with generated text proxies for the missing modality, never seeing real paired audio-vision data, and reports zero-shot results on AVQA, MUSIC-AVQA and OmniBench.

Problem

Multimodal LLMs typically require paired cross-modal data for joint reasoning, which is expensive to curate and hard to scale across modality combinations.

Approach

Uses the model itself to generate text proxies describing the unseen modality from a real anchor modality; proxies are mapped into the modality embedding space via a ridge-regression text-to-modality bridge over frozen contrastive encoders (CLAP, MetaCLIP), so the LLM learns composition without any real paired non-text data.

Key results

• MSP (LoRA r=512) reaches 65.65 on MUSIC-AVQA, above GPT-4o (50.08) and OneLLM (47.60), while using no real paired non-text data.
• MSP (LoRA r=256) reaches 85.73 on AVQA zero-shot, the highest reported zero-shot AVQA number among methods not finetuned on AVQA per the paper.
• On OmniBench, MSP (r=256) scores 43.26, below Qwen2.5-Omni (56.13) but above Gemini-1.5-Pro (42.91) and Baichuan-Omni-1.5 (42.90).

Surprising findings

• Replacing a real modality with its generated proxy at inference lowers accuracy, but the model still trains effectively, with the bridge adding roughly 2.7 to 7 points over proxies without it.
• A text-only Qwen3-4B fine-tuned with MSP proxies reaches 86.00 on AVQA, close to the 87.45 fully paired upper bound.

Limitations

• Performance depends on proxy quality and semantic consistency; noisy proxies degrade results.
• Inherits coverage limitations and biases of the frozen contrastive encoders (CLIP/CLAP).
• Only audio and vision are studied; larger scales and more modalities are untested.

Experimental setup

Datasets: AudioCaps WavCaps LLaVA-665K Clotho-AQA MSRVTT-QA MSVD-QA VQAv2 AVQA MUSIC-AVQA OmniBench

Benchmarks: AVQA MUSIC-AVQA OmniBench MUSIC-AVQA-R

Metrics: accuracy

Baselines: ImageBind-LLM X-InstructBLIP ChatBridge OneLLM Qwen2.5-VL GPT-4o PSAC HGA HCRN AnyGPT Baichuan-Omni-1.5 Gemini-1.5-Pro

Builds on: LLaVA ImageBind-LLM DeCap CapDec ChatBridge

Closest papers by meaning

• TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs (NeurIPS 2026 · similarity 0.91)
• Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs (NeurIPS 2026 · similarity 0.91)
• UNIALIGN: Scaling Multimodal Alignment within One Unified Model (CVPR 2025 · similarity 0.89)
• Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs (NeurIPS 2024 · similarity 0.89)
• What to align in multimodal contrastive learning? (ICLR 2025 · similarity 0.89)
• Do More Modalities Always Help? A Geometric Perspective on Missing-Modality Robustness (NeurIPS 2026 · similarity 0.89)