Research map / Vision-language and multimodal / Multimodal LLMs
IAA: Inner-Adaptor Architecture Empowers Frozen Large Language Model with Multimodal Capabilities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the Inner-Adaptor Architecture (IAA), which inserts trainable multimodal layers inside a frozen Llama3-8B so the MLLM gains vision-language and grounding ability without degrading text-only NLP performance.
Problem
Unfreezing the LLM for vision-language instruction tuning degrades its NLP capabilities (catastrophic forgetting), while prior approaches that freeze the LLM have failed to reach strong multimodal performance.
Approach
Insert multiple trainable insertion layers (initialized from the corresponding frozen LM layers) at varying depths inside the frozen LLM, with a separate multimodal embedding layer and LM head, so image tokens interact directly with each text-oriented transformer layer; text-only inference uses the original model path.
Key results
- IAA-14 reaches MME-P 1591.5, MMB-EN 74.9, MMB-CN 70.5, and MMMU 39.9, exceeding listed frozen-LLM and many unfrozen-LLM baselines.
- With the same 1.2M data, IAA-8 scores MMLU 68.4 and C-Eval 51.3 versus 55.8 and 40.5 for LLaVA-Llama3, which degrades on text tasks.
- IAA-14 achieves RefCOCO val 90.2 and RefCOCO+ val 83.4 with 2M grounding data, competitive with models trained on far larger grounding sets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace the ControlNet-inspired adaptor design with the refined architecture that removes feature propagation between insertion layers (ablation) | IAA · General multimodal benchmarks | MME-P: 1425.4 -> 1556.0 | Helped |
| replace the refined architecture with the final Figure (c) structure (ablation) | IAA · General multimodal benchmarks | MME-P: 1556.0 -> 1581.8 | Helped |
| add a multimodal embedding layer and multimodal language model head (ablation) | IAA Figure (c) · General multimodal benchmarks | MME-P: 1563.4 -> 1581.8 | Helped |
| remove Stage 1 alignment pre-training (ablation) | IAA · General multimodal benchmarks | MME-P: 1581.8 -> 1512.1 | Hurt |
| remove Stage 2 pre-training (ablation) | IAA · General multimodal benchmarks | MME-P: 1581.8 -> 1565.4 | Hurt |
| increase insertion layers from 8 to 14 (ablation) | IAA · General multimodal benchmarks | MME-P: 1581.8 -> 1591.5 | Helped |
| increase insertion layers from 14 to 22 (ablation) | IAA · General multimodal benchmarks | MME-P: 1591.5 -> 1531.7 | Hurt |
| train using IAA instead of LLaVA architecture on the same 1.2M data (ablation) | Llama3-8B multimodal model · MME | MME-P: 1506.0 -> 1560.2 | Helped |
| add 0.3 million training records, mainly Chinese data (ablation) | IAA-8 · General multimodal benchmarks | MME-P: 1560.2 -> 1581.8 | Helped |
| train with IAA rather than LLaVA-Llama3 | Llama3-8B · MMLU | score: 55.8 -> 68.4 | Helped |
| train with IAA rather than LLaVA-Llama3 | Llama3-8B · C-Eval | score: 40.5 -> 51.3 | Helped |
| increase insertion layers from 8 to 14 (ablation) | IAA · MMBench-EN | MMB-EN-T: 72.7 -> 74.9 | Helped |
Surprising findings
- The LLaVA-Llama3 baseline's text-only MMLU and C-Eval fall well below the base Llama3, indicating that multimodal training with an unfrozen LLM causes forgetting.
- Applying a high learning rate to the inner-adaptor during joint pre-training caused training loss overflow, requiring a two-stage schedule.
Limitations
- Adds extra parameters and reduces inference speed relative to an unfrozen LLM.
- Joint pre-training with the inner-adaptor is unstable at high learning rates, forcing a compromise on projector alignment.
- Evaluation on a limited set of benchmarks; grounding comparisons exclude models trained on very large data.
Experimental setup
Datasets: LLaVA-Pretrain 558K ALLaVA LLaVA-Instruct-150K DocVQA VSR ScienceQA RefCOCO COCO Flickr30k Entities Objects365 in-house dataset
Benchmarks: MME MMBench-EN MMBench-CN MMMU MMLU C-Eval BBH HumanEval MATH RefCOCO RefCOCO+ RefCOCOg
Metrics: MME perception score MMBench accuracy MMMU val accuracy MMLU C-Eval BBH HumanEval MATH Acc@0.5 grounding
Baselines: LLaVA-1.5 LLaVA-Llama3 Qwen-VL-Chat CogVLM mPLUG-Owl2 Honeybee Yi-VL DeepSeek-VL OpenFlamingo Llama-AdapterV2 MiniGPT-4 BLIP-2
Builds on: LLaVA-1.5 ControlNet Flamingo
Closest papers by meaning
- NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints (NeurIPS 2025 · similarity 0.90)
- Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models (CVPR 2026 · similarity 0.90)
- Towards General Continuous Memory for Vision-Language Models (NeurIPS 2025 · similarity 0.89)
- Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs (NeurIPS 2024 · similarity 0.89)
- ROSE: Rotate Your Large Language Model to See (CVPR 2026 · similarity 0.89)
- Frozen Language Models Are Gradient Coherence Rectifiers in Vision Transformers (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.