Research map / Vision-language and multimodal / Multimodal LLMs
Matryoshka Query Transformer for Large Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Matryoshka Query Transformer (MQT), which randomly keeps only the first m of M latent query tokens during LLaVA-1.5 training so one model can encode images into any number of visual tokens (2 to 256) at inference.
Problem
LVLMs encode each image into a fixed number of visual tokens (e.g., 576 for LLaVA-1.5), so they cannot adapt the token budget to varying computational constraints or tasks.
Approach
Adapt Matryoshka Representation Learning to visual tokens: a query transformer with M latent queries is trained at each step using only a randomly chosen prefix of m queries (tail-token dropping), so earlier tokens carry the most information and any prefix length can be used at inference, with a single training run instead of MRL's multi-configuration passes.
Key results
- MQT-LLaVA with 256 visual tokens matches or exceeds LLaVA-1.5 (576 tokens) on 11 benchmarks, outperforming it on 6 of 11, with a normalized average of 59.4 vs 59.2.
- Reducing to 16 tokens gives 8x fewer TFLOPs on MMBench with only a 2.4-point drop.
- With only 2 visual tokens, MQT-LLaVA scores 65.0 on ScienceQA-IMG and 32.5 on MMMU, outperforming InstructBLIP (32 tokens) on all 8 benchmarks it shares with it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MQT-LLaVA with 256 visual tokens instead of LLaVA-1.5 with 576 tokens | LLaVA-1.5 · 11 vision-language benchmarks | Avg: 59.2 -> 59.4 | Helped |
| Replace fixed-token QT-LLaVA training with MQT elastic-token training | QT-LLaVA · 11 vision-language benchmarks | Avg: 58.8 -> 59.4 | Helped |
| Reduce inference-time visual tokens from 256 to 16 | MQT-LLaVA · MMBench | score: 64.3 -> 61.9; 8x speed-up | Hurt |
| Reduce inference-time visual tokens from 256 to 2 | MQT-LLaVA · ScienceQA-IMG | score: 67.6 -> 65.0 | Hurt |
| Reduce inference-time visual tokens from 256 to 2 | MQT-LLaVA · MMMU | score: 34.8 -> 32.5 | Hurt |
| Replace linear token-number selection with log-based selection (ablation) | MQT-LLaVA · 11 vision-language benchmarks | Avg: 59.4 -> 57.3; 3.5% lower | Hurt |
| Replace attention-then-projection with projection-then-attention (ablation) | MQT-LLaVA · 11 vision-language benchmarks | Avg: 59.4 -> 56.6 | Hurt |
| Apply elastic query-transformer training during first-stage pretraining (ablation) | MQT-LLaVA · 11 vision-language benchmarks | Avg: 59.4 -> 56.7; 4.5% | Hurt |
| Use 16 rather than 144 inference-time visual tokens | MQT-LLaVA · ScienceQA-IMG | score: 67.5 -> 67.5 | Helped |
| Use 256 rather than 144 inference-time visual tokens | MQT-LLaVA · 11 vision-language benchmarks | Avg: 58.9 -> 59.4 | Helped |
Surprising findings
- Training with fewer tokens on average improves some benchmarks over the fixed 256-token baseline (e.g., VizWiz 53.1 vs 51.1), which the authors attribute to a regularization effect.
- ScienceQA performance with 16 tokens exceeds that with 144 tokens, suggesting fewer tokens can help some tasks.
Limitations
- Maximum inference token count is capped at the training maximum of 256; larger numbers are not explored.
- Performance drops sharply on some tasks (e.g., MME, POPE, MMBench) once below a threshold of roughly 16 tokens.
Experimental setup
Datasets: LLaVA-558K LLaVA-665K
Benchmarks: VizWiz ScienceQA-IMG VQA-v2 GQA POPE MME Perception MME Cognition MMMU MMBench LLaVA-Bench (In-the-Wild) MM-Vet
Metrics: accuracy normalized average TFLOPs
Baselines: LLaVA-1.5 QT-LLaVA BLIP-2 InstructBLIP Shikra IDEFICS Qwen-VL
Builds on: Matryoshka Representation Learning LLaVA-1.5 Q-Former BLIP-2
Closest papers by meaning
- Inference Optimal VLMs Need Fewer Visual Tokens and More Parameters (ICLR 2025 · similarity 0.90)
- Matryoshka Multimodal Models (ICLR 2025 · similarity 0.90)
- Conical Visual Concentration for Efficient Large Vision-Language Models (CVPR 2025 · similarity 0.90)
- iLLaVA: An Image is Worth Fewer Than 1/3 Input Tokens in Large Multimodal Models (ICLR 2026 · similarity 0.90)
- Accelerating Multimodal Large Language Models by Searching Optimal Vision Token Reduction (CVPR 2025 · similarity 0.90)
- Rethinking Token Reduction for Large Vision-Language Models (CVPR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.