Research map / Vision-language and multimodal / Multimodal LLMs
Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes frozen LLMs (mainly Vicuna-v1.5-7B) fed with image, video, audio and text prompts, finding perceptual and textual tokens occupy different cones yet are implicitly aligned inside the LLM and share activated weights, and uses these findings for proxy metrics, compute skipping, and a shared subnetwork.
Problem
Frozen LLMs can generalize to multimodal (perceptual) inputs without multimodal finetuning, but the reason for this generalization is poorly understood.
Approach
Shows that perceptual and textual tokens remain in distinct narrow cones yet become implicitly aligned inside LLM blocks (especially after self-attention), attributes this to the residual-stream-with-refinement (steering) block architecture, and exploits the findings for alignment-based proxy metrics, visual-token FFN skipping, and a task-agnostic pruned subnetwork (alpha-SubNet).
Key results
- Perceptual and textual tokens activate highly overlapping LLM weights, e.g., IoU of 0.69 between GQA and VQAv2 subnetworks versus 0.67 for GQA vs AudioCaps.
- Cross-modal (perceptual vs textual) cosine similarity inside blocks reaches about 0.45 for Vicuna-v1.5 versus about 0.1 in the residual stream.
- The alpha-SubNet, formed by intersecting pruning masks across modalities (~50% sparsity), gives performance comparable to task-specific Wanda pruning, especially on VQAv2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use higher implicit multimodal alignment as a model-selection signal | LLaVA-1.5 variants and checkpoints · Multimodal benchmarks | task performance | Helped |
| Increase implicit multimodal alignment | LLaVA-1.5 variants and checkpoints · POPE and COCO | hallucinations | Helped |
| Transfer a task's pruning mask to another task within the same modality | Vicuna-v1.5 pruning · Captioning with an OKVQA mask | CIDEr: ~1 point CIDEr | Hurt |
| Use the same task's subnetwork mask across different modalities | Multimodal LLM pruning · Captioning across image, video, audio, and text | task performance | Hurt |
| Skip visual-token computations in FFN layers | LLaVA-1.5 · Multitask setup | task performance | Hurt |
| Use the intersected, task- and modality-agnostic α-SubNet instead of magnitude pruning or a random mask | OPT with CLIP encoders · COCO image captioning and VQAv2 | task performance | Helped |
| Use α-SubNet instead of task-specific Wanda pruning | OPT with CLIP encoders · COCO image captioning and VQAv2 | task performance | No clear effect |
| Freeze the LLM rather than train it during multimodal pretraining (ablation) | LLaVA-1.5 variants · Multimodal input alignment | cross-modal similarity | Hurt |
| Use an MLP connector instead of transformer-based pooling (ablation) | LLaVA-1.5 variants without pretraining · Multimodal input alignment | cross-modal similarity | Helped |
| Use all visual tokens (ablation) | LLaVA-1.5 variants · Multimodal input alignment | cross-modal similarity | Helped |
| Add multimodal pretraining (ablation) | LLaVA-1.5 variants · Multimodal input alignment | cross-modal similarity | Helped |
| Use AvgSim or MinSim instead of the other reported cross-modal similarity measures (ablation) | Single-task setup · Perceptual-text token alignment | cross-modal similarity | Mixed |
Surprising findings
- Perceptual tokens have massive norms in the single-task setup, and their norms increase during mapping-module training.
- Narrow-cone effect is weaker when using median token similarity (MedSim), suggesting not all tokens are anisotropic.
Limitations
- Only frozen LLMs up to 7B parameters with concatenated perceptual tokens are studied.
- Generalization to larger, proprietary, or differently architected multimodal models remains untested.
- Modality-specific weights are not fully excluded, only shown to be a small fraction.
Experimental setup
Datasets: VQAv2 GQA OKVQA COCO caption MSVD-QA MSRVTT-QA MSRVTT AudioCaps Clotho Clotho-AQA TextVQA SEED
Benchmarks: VQAv2 GQA OKVQA COCO caption MSVD-QA MSRVTT-QA AudioCaps Clotho-AQA TextVQA SEED POPE
Metrics: CIDEr accuracy IoU cosine similarity
Baselines: LLaVA-1.5 magnitude pruning random mask
Builds on: Wanda LLaVA-1.5
Closest papers by meaning
- TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs (NeurIPS 2026 · similarity 0.93)
- Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs (ICML 2026 · similarity 0.92)
- The Alignment Illusion in Multimodal Large Language Models (NeurIPS 2026 · similarity 0.91)
- Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models (NeurIPS 2025 · similarity 0.91)
- SILMM: Self-Improving Large Multimodal Models for Compositional Text-to-Image Generation (CVPR 2025 · similarity 0.91)
- Towards Multimodal Large Language Models with Both Training and Inference Efficiency (ICML 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.