Research map / Multi-agent systems and games / LLM agents
StarDojo: Benchmarking Open-Ended Behaviors of Agentic Multimodal LLMs in Production–Living Simulations with Stardew Valley
Weihao Tan, Changjiu Jiang, Yu Duan, Mingcong Lei, Li JiaGeng, Yitian Hong, Xinrun Wang, Bo An
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use closed-source GPT-4.1 instead of the best open-source model (Llama 4 Maverick) | MLLM agent · StarDojo-Lite | success rate: 7.7 -> 12.7 | Improved |
| increase task difficulty from easy to medium | MLLM agents (GPT-4.1) · StarDojo-Lite | success rate: 21.4 -> 2.7 | Hurt |
| increase task difficulty to hard | all MLLM agents · StarDojo-Lite | success rate: 0.0 | Hurt |
| use a larger model (GPT-4.1) instead of GPT-4.1 mini | MLLM agent · StarDojo-Lite Crafting | success rate: 2.4 -> 26.2 | Improved |
| remove textual state observation (image only) (ablation) | GPT-4.1 agent · five basic tasks | success rate | Hurt |
| remove visual input (text only) (ablation) | GPT-4.1 agent · Ship 1 Parsnip / Go to Bus Stop | success rate | Hurt |
| disable game pausing during inference (real-time) (ablation) | GPT-4.1 agent · five basic tasks | success rate | Hurt |
| compare failure sources of GPT-4.1 agent | GPT-4.1 agent · StarDojo-Lite error analysis | share of errors: visual 42%, reasoning 21%, planning 21%, control 16% | Mixed |
Closest papers by meaning
- UltraHorizon: Benchmarking LLM-Agent Capabilities in Ultra Long-Horizon Scenarios (ICML 2026 · similarity 0.86)
- CivBench: A Long-Horizon Benchmark for Tool-Mediated Agents in Civilization VI (NeurIPS 2026 · similarity 0.86)
- CrafterDojo: A Suite of Foundation Models for Building Open-Ended Embodied Agents in Crafter (NeurIPS 2026 · similarity 0.86)
- AgentStudio: A Toolkit for Building General Virtual Agents (ICLR 2025 · similarity 0.86)
- MineAnyBuild: Benchmarking Spatial Planning for Open-world AI Agents (NeurIPS 2025 · similarity 0.86)
- VitaBench: Benchmarking LLM Agents with Versatile Interactive Tasks in Real-world Applications (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
