Research map / Vision-language and multimodal / Multimodal LLMs
Matryoshka Multimodal Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes M^3, which trains LLaVA models to represent images as nested coarse-to-fine sets of visual tokens (576 down to 1) so granularity can be chosen per instance at inference.
Problem
LMMs like LLaVA embed images into a fixed large number of visual tokens, which is inefficient for high-resolution images and video and offers no way to trade information density for efficiency at inference time.
Approach
Train the visual encoder and LLM so that visual tokens form nested sets (1, 9, 36, 144, 576) obtained by successive average pooling, with the loss averaged over all scales, giving one model that works at any token budget.
Key results
- LLaVA-1.5-M^3 with 9 visual tokens surpasses Qwen-VL-Chat (256 tokens) on MMBench and beats InstructBLIP-7B and -13B across listed benchmarks.
- The oracle using the trained LLaVA-NeXT-M^3 picks as few as 8.9 tokens on average and is about 8 points better than LLaVA-NeXT with 576 tokens per grid across six benchmarks.
- On MMBench with LLaVA-NeXT, M^3 at 9 tokens reaches 67.35 versus 45.45 for average pooling at 9 tokens.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use M³ with 9 visual tokens instead of LLaVA-1.5-7B with 576 tokens | LLaVA-1.5-7B · MMBench | score: 64.8 -> 63.1 | No clear effect |
| Use LLaVA-NeXT-M³ rather than the specific-scale-trained baseline | LLaVA-NeXT · MMBench | score: 67.78 -> 69.50 | Helped |
| Use LLaVA-NeXT-M³ rather than the specific-scale-trained baseline | LLaVA-NeXT · ChartQA | score: 45.40 -> 50.24 | Helped |
| Reduce the M³ video representation from 2880 to 720 tokens | LLaVA-NeXT-7B-M³ · ActivityNet | accuracy: 53.9 -> 55.0 | Helped |
| Replace average pooling with sequential sampling (ablation) | M³ · TextVQA | score: 51.97 -> 44.05 | Hurt |
| Replace average pooling with spatial sampling (ablation) | M³ · TextVQA | score: 51.97 -> 44.19 | Hurt |
| Train the LLM along with the visual encoder instead of freezing it (ablation) | LLaVA-NeXT-M³ · TextVQA | score: 36.15 -> 51.97 | Helped |
| Initialize LLM weights from LLaVA instead of text-only pretrained weights (ablation) | M³ · TextVQA | score: 46.84 -> 47.04 | Helped |
| Average losses over all visual token scales instead of randomly selecting a scale (ablation) | M³ · TextVQA | score: 47.04 -> 51.97 | Helped |
| Initialize LLM weights from LLaVA and average losses over all scales (ablation) | M³ · AI2D | score: 63.02 -> 66.77 | Helped |
Surprising findings
- Full visual tokens often do not give the best video performance; 720 or 180 tokens beat 2880 on 4 of 6 video benchmarks.
- COCO-style and most natural-image benchmarks retain near-full accuracy with only about 9 visual tokens per image.
Limitations
- No learned visual token predictor; a large gap remains between the oracle and fixed-scale performance.
- Performance drops sharply at very few tokens for dense tasks such as TextVQA and DocVQA.
- Video evaluation uses zero-shot collage inference with only 6 frames, and the model is not trained on video.
Experimental setup
Datasets: LLaVA-1.5 visual instruction data LLaVA-NeXT visual instruction data
Benchmarks: MMBench GQA POPE VizWiz SEEDBench ScienceQA MMMU DocVQA ChartQA AI2D TextVQA MSVD-QA
Metrics: accuracy
Baselines: LLaVA-1.5-7B LLaVA-NeXT-7B Qwen-VL Qwen-VL-Chat InstructBLIP Average pooling Spatial sampling Sequential sampling Video-LLaMA Video-ChatGPT Video-LLaVA
Builds on: LLaVA-1.5 LLaVA-NeXT Matryoshka Representation Learning CLIP
Closest papers by meaning
- Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings (NeurIPS 2025 · similarity 0.91)
- LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token (ICLR 2025 · similarity 0.90)
- Matryoshka Query Transformer for Large Vision-Language Models (NeurIPS 2024 · similarity 0.90)
- MaVEn: An Effective Multi-granularity Hybrid Visual Encoding Framework for Multimodal Large Language Model (NeurIPS 2024 · similarity 0.89)
- DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models (CVPR 2025 · similarity 0.89)
- Directed-Tokens: A Robust Multi-Modality Alignment Approach to Large Language-Vision Models (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.