Research map / Vision-language and multimodal / Multimodal LLMs

Wings: Learning Multimodal LLMs without Text-only Forgetting

NeurIPS 2024 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Wings, an MLLM with parallel visual and textual low-rank residual attention learners routed by attention weights, to prevent text-only forgetting while retaining multimodal performance.

Problem

Multimodal LLMs catastrophically forget text-only instruction-following abilities of their initial LLM when trained on multimodal instructions, and existing fixes require extra text/interleaved data or separate models.

Approach

Identifies a cross-layer attention shift (Pearson correlation of layer-level attention before vs. after the image) linked to text-only forgetting, and adds parallel visual and textual Low-Rank Residual Attention (LoRRA) learners in each attention block, combined by a router driven by the main attention weights.

Key results

• Wings (Qwen1.5-7B + SigLIP) reduces text-only forgetting relative to the same-backbone LLaVA-style baseline, improving MMLU by 60.53 vs 51.16 and CMMLU by 69.82 vs 60.46 under identical training data.
• Wings-pro achieves state-of-the-art on 13 of 18 evaluated datasets among equal-scale open-source 7B MLLMs, e.g. MMLU 61.3, RACE-High 69.6+, ScienceQA 83.1 and HallusionBench 47.3.
• Wings-1.8B reaches MMLU 44.9 and CMMLU 50.9, outperforming DeepSeek-VL 1.3B and MiniCPM-V 2.4B on several text-only tasks while staying competitive on multimodal tasks.

Measured effects reported in the paper

Surprising findings

• Fine-tuning with multimodal instructions causes text-only forgetting on 12 of 16 text datasets relative to the initial Qwen LLM, with drops up to 9.70 on MMLU.
• LoRA-style parameter-efficient methods forget less text-only knowledge but underperform on multimodal questions.

Limitations

• Integrating visual learners requires restarting feature alignment training, adding cost.
• Wings-1.8B trades performance for edge deployment.
• Still requires some text-only replay data to enhance overall performance.

Experimental setup

Datasets: LLaVA-v1.5 training data ALLaVA SynthDog ArXivQA Stanford Alpaca Alpaca GPT-4 LIMA UltraChat WebQA BELLE-0.5M MMLU CMMLU

Benchmarks: MMLU CMMLU ARC-Easy ARC-Challenge WinoGrande OpenbookQA RACE-Middle RACE-High WSC CHID HellaSwag SIQA

Metrics: accuracy forgetting delta vs. initial LLM MME score

Baselines: LLaVA-v1.5 LLaVA-Next DeepSeek-VL Qwen-VL Monkey MiniCPM-V ShareGPT4V InstructBLIP IDEFICS O-Flamingo GPT-4V Gemini Pro Vision

Builds on: LLaVA LoRA Mixture-of-Experts

Closest papers by meaning

• Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs (ICML 2026 · similarity 0.92)
• ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Models (NeurIPS 2024 · similarity 0.91)
• TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs (NeurIPS 2026 · similarity 0.90)
• Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception (CVPR 2024 · similarity 0.90)
• Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings (NeurIPS 2025 · similarity 0.89)
• Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs (NeurIPS 2024 · similarity 0.89)