Research map / Vision-language and multimodal / Multimodal LLMs
Multi-modal Agent Tuning: Building a VLM-Driven Agent for Efficient Tool Usage
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Synthesizes 20K multi-modal tool-usage trajectories (MM-Traj) with GPT-4o mini and verifiers, then LoRA-tunes VLMs (MiniCPM-V-8.5B, Qwen2-VL-7B) into the T3-Agent controller, improving on GTA and GAIA.
Problem
VLM-driven multi-modal agents have weak tool-usage reasoning, and existing tool-usage trajectory datasets are small, simple, or limited to text and images, so VLM controllers lack training data for complex multi-file tasks.
Approach
Generate queries first and then files (searched images or code-generated documents) rather than sampling files first, produce ReAct-style thought/code trajectories with a zero-shot GPT-4o mini agent, filter them with LLM query-file and trajectory verifiers, and tune a VLM controller on the resulting data.
Key results
- On GTA, tuned MiniCPM-V-8.5B reaches 52.56 AnsAcc vs 33.97 untuned, and tuned Qwen2-VL-7B reaches 53.85 vs 42.31.
- On GAIA validation, tuned Qwen2-VL-7B reaches 16.97 AnsAcc vs 9.70 untuned, and tuned MiniCPM-V-8.5B reaches 15.15 vs 7.27.
- Removing the two verifiers drops GTA AnsAcc from 52.56 to 50.00 and GAIA from 15.15 to 13.33.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Trajectory-tune MiniCPM-V-8.5B on MM-Traj | MiniCPM-V-8.5B · GTA | AnsAcc: 33.97 -> 52.56 | Helped |
| Trajectory-tune MiniCPM-V-8.5B on MM-Traj | MiniCPM-V-8.5B · GTA | ToolAcc: 36.59 -> 65.85 | Helped |
| Trajectory-tune MiniCPM-V-8.5B on MM-Traj | MiniCPM-V-8.5B · GTA | CodeExec: 56.10 -> 80.49 | Helped |
| Trajectory-tune Qwen2-VL-7B on MM-Traj | Qwen2-VL-7B · GTA | AnsAcc: 42.31 -> 53.85 | Helped |
| Trajectory-tune Qwen2-VL-7B on MM-Traj | Qwen2-VL-7B · GTA | ToolAcc: 44.85 -> 64.63 | Helped |
| Trajectory-tune Qwen2-VL-7B on MM-Traj | Qwen2-VL-7B · GTA | CodeExec: 65.19 -> 84.32 | Helped |
| Trajectory-tune MiniCPM-V-8.5B on MM-Traj | MiniCPM-V-8.5B · GAIA validation set | AnsAcc: 7.27 -> 15.15 | Helped |
| Trajectory-tune Qwen2-VL-7B on MM-Traj | Qwen2-VL-7B · GAIA validation set | AnsAcc: 9.70 -> 16.97 | Helped |
| Use both data verifiers instead of removing them (ablation) | T3-Agent · GTA | AnsAcc: 50.00 -> 52.56 | Helped |
| Use both data verifiers instead of removing them (ablation) | T3-Agent · GAIA validation set | AnsAcc: 13.33 -> 15.15 | Helped |
| Add image modality (ablation) | T3-Agent · GTA | AnsAcc: 10.67 -> 52.56 | Helped |
| Add image modality (ablation) | T3-Agent · GTA | ToolAcc: 25.32 -> 65.85 | Helped |
Surprising findings
- Removing image input from the tuned agent drops GTA AnsAcc to 10.67, indicating heavy reliance on visual input.
- Tuned T3-Agent with open VLMs still trails GPT-4-based agents on GAIA, which the authors attribute to model size and training data.
Limitations
- Only multi-modal information in queries is used; intermediate multi-modal observations in trajectories are not exploited.
- Still behind closed-source GPT-4-based agents on GAIA.
- Code execution rate and answer accuracy remain modest on GAIA (about 15-17% AnsAcc).
Experimental setup
Datasets: MM-Traj ChartQA COCO LLaVA SAM TextVQA Web-Celebrity Web-Landmark WikiArt Cauldron open-LLaVa-NeXT
Benchmarks: GTA GAIA
Metrics: AnsAcc ToolAcc CodeExec Level 1/2/3 accuracy
Baselines: Lego Agent Sibyl Agent Warm-up Act Agent HF Agent GPT-4o GPT-4o mini Qwen2-VL-7B MiniCPM-V-8.5B InternVL2-8B LLaVA-NeXT-8B
Builds on: ReAct LLaVA-Plus Lumos DEDER
Closest papers by meaning
- AutoTool: Efficient Tool Selection for Large Language Model Agents (AAAI 2026 · similarity 0.88)
- Iterative Tool Usage Exploration for Multimodal Agents via Step-wise Preference Tuning (NeurIPS 2025 · similarity 0.87)
- ToolTree: Efficient LLM Tool Planning via Dual-Feedback Monte Carlo Tree Search and Bidirectional Pruning (ICLR 2026 · similarity 0.87)
- Towards Robust Multi-Modal Reasoning via Model Selection (ICLR 2024 · similarity 0.87)
- Joint Optimization of Tool Creation and Use for Large Language Model Agents (NeurIPS 2026 · similarity 0.87)
- GTA: A Benchmark for General Tool Agents (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.