Research map / Vision-language and multimodal / Multimodal LLMs
Look Less, Think Faster: Joint Token-Compute Adaptation for Multimodal LLMs
Pengcheng Wang, Zhiquan Wang, Jayoung Lee, Zhuoyan Xu, Ran Xu, Saurabh Bagchi, Yin Li, Somali Chaterji
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add vision-side token controller jointly with LLM compute controller | AdaLLaVA (compute-only) · VQAv2 at 50% FLOPs | accuracy (%): 67.9 -> 74.4 | Improved |
| add token controller to compute-only adaptation | AdaLLaVA · TextVQA at 50% FLOPs | accuracy (%): 29.8 -> 54.4 | Improved |
| joint end-to-end training vs sequential AdaLLaVA-PruMerge | SmartVL · VQAv2 at 50% FLOPs | accuracy (%): 74.5 (sequential) vs 74.4 | No clear effect |
| joint control vs sequential composition on GQA | SmartVL · GQA at 50% FLOPs | accuracy (%): 60.1 (AdaLLaVA-PruMerge) vs 59.8 | Hurt |
| SmartVL vs AdaLLaVA on MMBench at 50% FLOPs | SmartVL · MMBench | accuracy (%): 63.3 (AdaLLaVA) vs 62.0 | Hurt |
| add head-group (width) gating on top of layer gating (ablation) | SmartVL-L · TextVQA full budget | accuracy (%): 58.1 -> 57.5 | Hurt |
| jointly train token and compute controllers rather than separately (ablation) | SmartVL · POPE/VQAv2/TextVQA | optimal allocation | Improved |
| scale joint adaptation to 13B backbone | SmartVL · VQAv2 val 13B | accuracy (%): 72.27 (8.27T) -> 75.45 (7.25T) | Improved |
| learned budget-conditioned token controller vs FastV / PruMerge+ | SmartVL token controller · VQAv2 | accuracy vs FLOPs | Improved |
Closest papers by meaning
- Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference (AAAI 2025 · similarity 0.91)
- Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification (ICLR 2025 · similarity 0.91)
- Inference Optimal VLMs Need Fewer Visual Tokens and More Parameters (ICLR 2025 · similarity 0.90)
- Accelerating Multimodal Large Language Models by Searching Optimal Vision Token Reduction (CVPR 2025 · similarity 0.90)
- Variation-aware Vision Token Dropping for Faster Large Vision-Language Models (CVPR 2026 · similarity 0.90)
- Improving Visual Token Reduction via Rectifying Distortions for Efficient Multimodal LLM Inference (ICML 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
