Research map / Vision-language and multimodal / Multimodal LLMs

TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces TokenSwap, which replaces text concepts with generated images in MMLU to build TokenSwap-Bench, measures a modality gap across 42 MLLMs, and shows TokenSwap-style training reduces the gap.

Problem

Multimodal LLMs should give consistent answers to semantically equivalent text and image-interleaved inputs, but performance drops when textual concepts are replaced by images (the "modality gap").

Approach

Concept-level substitution of text spans with semantically aligned generated or retrieved images to create image-interleaved versions of existing text benchmarks, filtered by validity, importance, and caption-guided round-trip checks; then uses the same substitution on training data to reduce the gap.

Key results

• Across 42 MLLMs the modality gap ranges from 4.2% (Gemini-3-Flash) to 47.4% (InternVL2-8B), averaging 19.6% ± 3.3%.
• Reasoning models show smaller average gaps (10.1% ± 2.2%) than non-reasoning models (25.5% ± 3.7%), Welch t = -7.48, p < 0.001.
• TokenSwap post-training with generated images reduces the Qwen2-VL-7B gap from 0.268 (text training) to 0.167; on IIIT 5K-Word OCR accuracy rises from 90.03% to 92.70%.

Surprising findings

• Text-only training widens the modality gap (0.232 baseline to 0.268 after text-only post-training).
• Prompting strategies (CoT, few-shot) and scaling training compute do not reliably reduce the modality gap; a 10x increase in FLOPs reduces the gap by only about 2.8%.

Limitations

• The measured gap depends on image quality; retrieved images produce larger gaps than generated ones.
• TokenSwap training requires in-distribution data, and domain mismatch may limit transfer.
• Training experiments are at a limited scale (Qwen2-VL-7B and a smaller InternVL3-8B replication).

Experimental setup

Datasets: MMLU GSM8K Magpie-Pro LLaVA post-training data DataComp-Small IIIT 5K-Word SEAM

Benchmarks: TokenSwap-Bench GSM8K (TokenSwap variant) IIIT 5K-Word SEAM

Metrics: accuracy modality gap (absolute) relative modality gap

Baselines: SEAM Text training Baseline (LLaVA post-training)

Builds on: MMLU GSM8K Magpie-Pro CLIP

Closest papers by meaning

• Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs (NeurIPS 2024 · similarity 0.93)
• Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs (ICML 2026 · similarity 0.92)
• LoMo: Local Modality Substitution for Deeper Vision-Language Fusion (NeurIPS 2026 · similarity 0.91)
• MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data (NeurIPS 2026 · similarity 0.91)
• Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs (NeurIPS 2026 · similarity 0.91)
• Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings (NeurIPS 2025 · similarity 0.91)