Research map / Vision-language and multimodal / Multimodal LLMs
LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents
Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, Lei Zhang, Jianfeng Gao and 1 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train LLaVA with tool-use data (LLaVA-Plus All Tools) instead of plain LLaVA | LLaVA · LLaVA-Bench (In-the-Wild) | relative score (All): 57.1 -> 69.5 | Improved |
| train LLaVA with tool-use data on SEED-Bench | LLaVA · SEED-Bench | average accuracy: 44.45 -> 55.01 | Improved |
| use tool outputs at test time without training to use them (ablation) | LLaVA · LLaVA-Bench (COCO) | relative score (All): 81.2 -> 59.1 | Hurt |
| use tool outputs at test time without training on SEED-Bench (ablation) | LLaVA · SEED-Bench | average accuracy: 44.45 -> 51.54 | Improved |
| add thoughts (chain-of-thought) to the unified data format (ablation) | LLaVA-Plus (Fly) · LLaVA-Bench (In-the-Wild) | relative score (All): 48.7 -> 59.1 | Improved |
| add thoughts to data format on COCO set (ablation) | LLaVA-Plus (Fly) · LLaVA-Bench (COCO) | relative score (All): 79.4 -> 80.4 | Improved |
| use all tools as additional context rather than on the fly (ablation) | LLaVA-Plus · LLaVA-Bench (Tools) | All: 76.5 -> 82.3 | Improved |
| visual instruction tuning of tool use (LLaVA-Plus) vs GPT4Tools (ablation) | LLaVA-Plus · LLaVA-Bench (In-the-Wild) | relative score (All): 40.7 -> 59.1 | Improved |
| train open-source LMM as tool planner instead of chaining tools with GPT-4 (ablation) | LLaVA-Plus · LLaVA-Bench (Tools) | All: 76.9 -> 82.3 | Improved |
| LLaVA-Plus 7B with tools on MM-Vet | LLaVA-7B · MM-Vet | Total: 24.1 -> 27.5 | Improved |
| LLaVA-Plus 7B with tools on MM-Vet math | LLaVA-7B · MM-Vet | Math: 8.1 -> 7.7 | No clear effect |
| LLaVA-Plus 13B vs LLaVA 13B on VisIT-Bench | LLaVA-13B · VisIT-Bench | ELO: 1095 -> 1203 | Improved |
Closest papers by meaning
- Improved Baselines with Visual Instruction Tuning (CVPR 2024 · similarity 0.90)
- DeepEyesV2: Toward Agentic Multimodal Model (ICLR 2026 · similarity 0.89)
- LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token (ICLR 2025 · similarity 0.89)
- Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models (CVPR 2025 · similarity 0.88)
- LLaVA-Critic: Learning to Evaluate Multimodal Models (CVPR 2025 · similarity 0.88)
- 3D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformer (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
