Research map / Multi-agent systems and games / LLM agents
InfantAgent-Next: A Multimodal Generalist Agent for Automated Computer Interaction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents InfantAgent-Next, a modular multimodal generalist computer-use agent that routes planning, tool selection, execution, and visual grounding to different specialist models, evaluated on OSWorld, SWE-Bench, and GAIA.
Problem
Existing computer-use agents either rely on a single large model with predefined tools (limited generality) or on pure vision GUI control (weak execution precision on clicks and file edits); a hybrid, generalist agent is needed.
Approach
A highly modular workflow with separate Planner, Tool Selector, and Executor models over a unified tagged memory cache, dynamic toolkit selection, iterative region-cropping visual grounding for mouse clicks, and a fallback fuzzy-matching file-edit procedure.
Key results
- On OSWorld with 50 steps, InfantAgent-Next (Claude-3.7-Sonnet + UI-TARS-1.5-7B) reaches 35.3% accuracy vs. 26.0% for Claude Computer Use at 50 steps, a gain of about 9 points (the abstract reports a 7.27% gain over Claude-Computer-Use).
- On SWE-Bench-Verified (50-case subset), the agent reaches 66% with Claude-3.7-Sonnet, behind only SWE-agent + Claude 3.7 Sonnet w/ Review Heavy (72%) and Openhands_04_15 (68%) among listed entries.
- On GAIA validation, the agent scores 56.97% average and 62.79% on Level 2, ranking second among open-source agents behind OWL-Roleplaying (58.18%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use InfantAgent-Next instead of Claude Computer Use | Claude-3.7-Sonnet computer interaction on OSWorld · OSWorld | accuracy: $26.0$ -> $35.3$ | Helped |
| Use InfantAgent-Next instead of Agent S2 | Claude-3.7-Sonnet computer interaction · OSWorld | accuracy: $34.5$ -> $35.3$ | Helped |
| Use InfantAgent-Next instead of UI-TARS-1.5-72B | OSWorld computer interaction at 50 steps · OSWorld | accuracy: $38.0$ -> $35.3$ | Hurt |
| Use InfantAgent-Next instead of SWE-agent + Claude 3.7 Sonnet w/ Review Heavy | SWE-Bench-Verified · SWE-Bench-Verified (50 cases) | accuracy: $72$ -> $66$ | Hurt |
| Use InfantAgent-Next instead of Agentless-1.5 | SWE-Bench-Lite with GPT-4o · SWE-Bench-Lite | accuracy: $32.00$ -> $31.67$ | Hurt |
| Use InfantAgent-Next instead of OWL-Roleplaying | GAIA validation · GAIA validation | Average: $58.18$ -> $56.97$ | Hurt |
| Use InfantAgent-Next instead of Langfun Agent v2.1 | GAIA validation · GAIA validation Level 2 | score: $68.60$ -> $62.79$ | Hurt |
| Fix cropped-region width and vary its height (ablation) | Iterative Region Cropping · ScreenSpot-Pro | click accuracy | Helped |
| Use two iterations instead of continuing to the third iteration (ablation) | Iterative Region Cropping · ScreenSpot-Pro | click accuracy | No clear effect |
Surprising findings
- On OSWorld, the agent with Claude-3.7-Sonnet as planner (35.3%) falls below the UI-TARS-1.5-72B 100-step result (42.5%) and the OpenAI CUA 200-step result (38.1%), so its advantage depends on the step budget.
- GPT-4o reportedly attains only 0.8% on ScreenSpot-Pro, highlighting weak grounding of frontier models.
Limitations
- Current work is confined to the reasoning phase and relies on manually engineered prompts rather than trained tool invocation.
- SWE-Bench-Verified and file-editing evaluations use random or uniform 50-case and 10% subsets, limiting statistical confidence.
- Heavy reliance on prompt engineering for tool routing.
Experimental setup
Datasets: OSWorld SWE-Bench-Verified SWE-Bench-Lite GAIA ScreenSpot-Pro
Benchmarks: OSWorld SWE-Bench-Verified SWE-Bench-Lite GAIA ScreenSpot-Pro
Metrics: accuracy repair success rate overall repair rate
Baselines: Claude Computer Use OpenAI CUA UI-TARS Agent S2 OWL Agentless OpenHands + CodeAct v1.8 SWE-agent + Claude 3.7 Sonnet
Builds on: OpenHands SWE-Agent Agent S2 AutoGPT
Closest papers by meaning
- OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments (NeurIPS 2024 · similarity 0.88)
- Agent S: An Open Agentic Framework that Uses Computers Like a Human (ICLR 2025 · similarity 0.87)
- VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agents (ICLR 2025 · similarity 0.87)
- Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale (ICML 2025 · similarity 0.87)
- AgentVista: Evaluating Multimodal Agent in Ultra-Challenging Realistic Visual Scenarios (NeurIPS 2026 · similarity 0.86)
- EgoBench: An Interactive Egocentric Multimodal Benchmark for Tool-Using Agents (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.