Research map / Multi-agent systems and games / LLM agents
Iterative Tool Usage Exploration for Multimodal Agents via Step-wise Preference Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SPORT, an annotation-free self-exploration framework in which a VLM tool-using agent synthesizes multimodal tasks, alternately samples and AI-verifies steps, and is refined with step-wise DPO.
Problem
Training multimodal tool-using agents usually requires costly human-annotated answers and tool trajectories, which are hard to obtain for complex multimodal tasks and may generalize poorly.
Approach
Alternate step sampling (n candidate thought+code actions executed per step) with an LLM/VLM step verifier that picks the best action, yielding step-level preference pairs without ground truth; then update the controller with DPO iteratively starting from an SFT (MAT) model.
Key results
- On GTA, SPORT agent reaches 60.26% AnsAcc vs 53.85% for MAT-SFT controller (Qwen2-VL-7B), with ToolAcc 72.41% and CodeExec 91.87%.
- On GAIA validation, SPORT agent reaches 20.61% AnsAcc vs 16.97% for MAT-Qwen2-VL-7B and 9.70% for HF Agent with Qwen2-VL-7B.
- Human study reports 82% agreement between verifier and human step preferences; applying SPORT directly to Qwen2-VL-7B raises GTA AnsAcc from 42.31% to 55.13%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply SPORT preference tuning after MAT SFT (ablation) | MAT-Qwen2-VL-7B · GTA | AnsAcc: 53.85 -> 60.26; +5.13% over MAT alone | Helped |
| apply SPORT preference tuning after MAT SFT (ablation) | MAT-Qwen2-VL-7B · GTA | ToolAcc: +5.11 | Helped |
| apply SPORT preference tuning after MAT SFT (ablation) | MAT-Qwen2-VL-7B · GTA | CodeExec: +5.97 | Helped |
| apply DPO to static MM-Traj preference data (ablation) | MAT-SFT · GTA | AnsAcc: +1.28 | Helped |
| set iteration step size d to 500 rather than d=200 or d=1000 (ablation) | SPORT · GTA | AnsAcc: 56.41 -> 57.69; 57.69 -> 57.05 | Mixed |
| reduce in-context examples from 20 to 5 and task seeds from 425 to 100 (ablation) | SPORT · GTA | AnsAcc: 60.26 -> 58.33 | Hurt |
| apply SPORT directly to the untuned base model (ablation) | Qwen2-VL-7B · GTA | AnsAcc: 42.31% to 55.13% (+12.82%) | Helped |
| use online self-exploration and step-wise preference tuning instead of the SFT agent | MAT-Qwen2-VL-7B · GAIA | AnsAcc: 16.97 -> 20.61 | Helped |
| use online self-exploration and step-wise preference tuning instead of the SFT agent | MAT-Qwen2-VL-7B · GTA | AnsAcc: 53.85 -> 60.26 | Helped |
Surprising findings
- Naively applying DPO on static MM-Traj GPT-4o data gives only modest gains (MAT-SFT-DPO 55.13% AnsAcc), far below SPORT's online self-exploration.
- Reported compute savings: SPORT total 36.56h vs MAT 173.75h, with about 33% lower GPU-hour cost.
Limitations
- Verifier relies on human-designed rules and prompts, generalizing poorly to outliers.
- Gap to closed-source GPT-4-based agents on GAIA remains, attributed to model size and data.
- No theoretical guarantee for the verifier.
Experimental setup
Datasets: MAT seed queries and SFT trajectories (20K) MM-Traj GTA GAIA
Benchmarks: GTA GAIA
Metrics: AnsAcc ToolAcc CodeExec GAIA Level 1/2/3 AnsAcc BLEU human ratings
Baselines: MAT-Qwen2-VL-7B MAT-MiniCPM-V-8.5B HF Agent Lego Agent Warm-up Act Agent MAT-SFT-DPO
Builds on: ReAct DPO MAT LoRA STEP-DPO
Closest papers by meaning
- DeepEyesV2: Toward Agentic Multimodal Model (ICLR 2026 · similarity 0.87)
- Multi-modal Agent Tuning: Building a VLM-Driven Agent for Efficient Tool Usage (ICLR 2025 · similarity 0.87)
- ToolSearcher: Optimizing Tool Selection at Scale via Reinforcement Learning (NeurIPS 2026 · similarity 0.86)
- EgoBench: An Interactive Egocentric Multimodal Benchmark for Tool-Using Agents (NeurIPS 2026 · similarity 0.86)
- AgentBrew: Offline Tool-Use Agent Learning from Raw Real-World Trajectories (NeurIPS 2026 · similarity 0.86)
- Scaling Autonomous Agents via Automatic Reward Modeling And Planning (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.