Research map / Multi-agent systems and games / Human-AI and goal-conditioned
VIKI‑R: Coordinating Embodied Multi-Agent Cooperation via Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VIKI-Bench, a three-level hierarchical benchmark for heterogeneous embodied multi-agent cooperation (agent activation, task planning, trajectory perception), and VIKI-R, an SFT-then-GRPO training framework for Qwen2.5-VL that outperforms zero-shot and SFT-only baselines.
Problem
Vision-language models lack support for diverse robot embodiments and visually grounded reasoning in heterogeneous multi-agent cooperation, including which agents to activate, how to plan parallel actions, and how to predict trajectories.
Approach
A hierarchical benchmark with three visual reasoning levels over heterogeneous robots and multi-view observations, plus a two-stage recipe (CoT SFT warm-up followed by GRPO with format, accuracy, and step-penalty rewards) that trains VLMs on it.
Key results
- VIKI-R on Qwen2.5-VL-7B reaches 93.00% on VIKI-L1 and 69.25% average on VIKI-L2, versus 17.50% for GPT-4o on L2 and 18.40% on L1.
- On VIKI-L3, VIKI-R (7B) achieves the lowest average trajectory error of 77.82 versus 115.73 for GPT-4o.
- Step penalty in a 1,000-task challenging subset raises VIKI-R accuracy from 7.1% to 46.8% on OOD-H and from 8.0% to 96.0% on ID-H.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VIKI-R instead of zero-shot inference | Qwen2.5-VL-7B-Instruct · VIKI-L1 | accuracy: 4.26 -> 93.00 | Helped |
| Use VIKI-R instead of answer-only SFT | Qwen2.5-VL-7B-Instruct · VIKI-L2 OOD | accuracy: 25.62 -> 33.25 | Helped |
| Use VIKI-R instead of answer-only SFT | Qwen2.5-VL-7B-Instruct · VIKI-L2 ID | accuracy: 96.89 -> 95.22 | Hurt |
| Add feedback after each failed planning attempt | GPT-4o · VIKI-L2 planning | pass@3: 18.7 -> 20.6 | Helped |
| Add constraint-based step penalty (ablation) | VIKI-R · VIKI-L2 OOD-H | accuracy: 7.1% -> 46.8% (+39.7%) | Helped |
| Add step-level reward (ablation) | Qwen2.5-VL-7B VIKI-R · VIKI-L2 ID | accuracy: 92.06 -> 95.22 | Helped |
| Add step-level reward (ablation) | Qwen2.5-VL-7B VIKI-R · VIKI-L2 OOD | accuracy: 29.51 -> 33.25 | Helped |
| Scale the backbone from 3B to 7B (ablation) | VIKI-R · VIKI-L1 | accuracy: 74.10 -> 93.00 | Helped |
| Scale the backbone from 7B to 32B (ablation) | VIKI-R · VIKI-L2 | accuracy: 69.25 -> 74.74 | Helped |
| Remove reasoning guidance prompt (ablation) | Qwen2.5-VL-3B VIKI-R · VIKI-L1 | accuracy: 74.10 -> 73.47 | No clear effect |
| Replace GRPO with DAPO (ablation) | Qwen2.5-VL-3B VIKI-R · VIKI-L2 ID | accuracy: 93.61 -> 93.22 | No clear effect |
| Replace GRPO with DAPO (ablation) | Qwen2.5-VL-7B VIKI-R · VIKI-L2 OOD | accuracy: 33.25 -> 33.58 | No clear effect |
Surprising findings
- RL from the base model without SFT warm-up (VIKI-R-Zero) yields near-zero planning accuracy, since the zero-shot model produces almost no valid plans.
- Zero-shot Gemini and Claude show almost no trajectory-prediction capability, while the 72B open model matches or beats some closed models on perception metrics.
Limitations
- Evaluation is limited to simulated environments; real-world case study uses only 100 VIKI-L1 tasks and shows low accuracy.
- Hierarchical fixed task levels may not capture dynamic real-world conditions, obstacles, or adaptive agents.
- Multi-level RL fine-tuning incurs substantial computational cost.
Experimental setup
Datasets: VIKI-Bench
Benchmarks: VIKI-Bench L1 VIKI-Bench L2 VIKI-Bench L3
Metrics: accuracy ACC_ID ACC_OOD RMSE Hausdorff distance discrete Fréchet distance pass@k
Baselines: GPT-4o Claude-3.7-Sonnet Gemini-2.5-Flash-preview Qwen2.5-VL-72B-Instruct Qwen2.5-VL-32B-Instruct Llama-3.2-11B-Vision Ans-SFT VIKI-R-Zero ReAct MAD (2 agents) DAPO
Builds on: GRPO DeepSeek-R1 LLaVA-CoT Qwen2.5-VL
Closest papers by meaning
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.89)
- Scaling Agentic Reinforcement Learning for Tool-Integrated Reasoning in VLMs (CVPR 2026 · similarity 0.89)
- Robot-R1: Reinforcement Learning for Enhanced Embodied Reasoning in Robotics (NeurIPS 2025 · similarity 0.89)
- PhysVLM-AVR: Active Visual Reasoning for Multimodal Large Language Models in Physical Environments (NeurIPS 2025 · similarity 0.88)
- VS-Bench: Evaluating VLMs for Strategic Abilities in Multi-Agent Environments (CVPR 2026 · similarity 0.88)
- VLMimic: Vision Language Models are Visual Imitation Learner for Fine-grained Actions (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.