AI Paper Atlas

Research map / Vision-language and multimodal / Multimodal LLMs

LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents

Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, Lei Zhang, Jianfeng Gao and 1 more

ECCV 2024 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 187 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
train LLaVA with tool-use data (LLaVA-Plus All Tools) instead of plain LLaVALLaVA · LLaVA-Bench (In-the-Wild)relative score (All): 57.1 -> 69.5Improved
train LLaVA with tool-use data on SEED-BenchLLaVA · SEED-Benchaverage accuracy: 44.45 -> 55.01Improved
use tool outputs at test time without training to use them (ablation)LLaVA · LLaVA-Bench (COCO)relative score (All): 81.2 -> 59.1Hurt
use tool outputs at test time without training on SEED-Bench (ablation)LLaVA · SEED-Benchaverage accuracy: 44.45 -> 51.54Improved
add thoughts (chain-of-thought) to the unified data format (ablation)LLaVA-Plus (Fly) · LLaVA-Bench (In-the-Wild)relative score (All): 48.7 -> 59.1Improved
add thoughts to data format on COCO set (ablation)LLaVA-Plus (Fly) · LLaVA-Bench (COCO)relative score (All): 79.4 -> 80.4Improved
use all tools as additional context rather than on the fly (ablation)LLaVA-Plus · LLaVA-Bench (Tools)All: 76.5 -> 82.3Improved
visual instruction tuning of tool use (LLaVA-Plus) vs GPT4Tools (ablation)LLaVA-Plus · LLaVA-Bench (In-the-Wild)relative score (All): 40.7 -> 59.1Improved
train open-source LMM as tool planner instead of chaining tools with GPT-4 (ablation)LLaVA-Plus · LLaVA-Bench (Tools)All: 76.9 -> 82.3Improved
LLaVA-Plus 7B with tools on MM-VetLLaVA-7B · MM-VetTotal: 24.1 -> 27.5Improved
LLaVA-Plus 7B with tools on MM-Vet mathLLaVA-7B · MM-VetMath: 8.1 -> 7.7No clear effect
LLaVA-Plus 13B vs LLaVA 13B on VisIT-BenchLLaVA-13B · VisIT-BenchELO: 1095 -> 1203Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy