Research map / Large language models / Agents, code and math

BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games

ICLR 2025 · Large language models › Agents, code and math · approach: Agent instruction-following benchmarks · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces BALROG, a benchmark of six procedurally generated game environments (BabyAI, Crafter, TextWorld, Baba Is AI, MiniHack, NetHack) for zero-shot evaluation of agentic LLMs and VLMs, finding low performance on hard games and degraded performance with visual input.

Problem

Existing agentic benchmarks involve only a few dozen interaction rounds, so there is no effective way to evaluate long-horizon decision-making, planning, and exploration of LLM and VLM agents in complex dynamic environments.

Approach

Aggregates existing RL game environments of widely varying difficulty into a unified, decoupled framework with a fine-grained 0-100 progression metric, including a data-informed progression system for NetHack, and evaluates models zero-shot in both language-only and vision-language observation formats.

Key results

• Claude 3.5 Sonnet achieved the highest language-only average progress at 32.64%, closely followed by GPT-4o at 32.34%.
• In the vision-language format Claude 3.5 Sonnet reached 35.48% average progress, while GPT-4o dropped to 22.56%, showing visual input hurts several models.
• All models are far from solving NetHack; the best, o1-preview, reached only about 1.5% average game progression.

Measured effects reported in the paper

Surprising findings

• Several models perform worse when images of the environment are added to the text observation, e.g., GPT-4o drops from 32.34% to 22.56% average progress.
• Models often know that an action is harmful (e.g., eating rotten food or exiting the dungeon early in NetHack) yet still take it, a 'knowing-doing gap'.

Limitations

• Evaluation is zero-shot only; inference-time prompting strategies such as few-shot or chain-of-thought are not benchmarked in the paper.
• o1-preview was evaluated only on NetHack due to budget constraints, and only one image per observation was used because of image-processing cost.
• Gemini models returned 'unsafe' flags on TextWorld, leaving their TextWorld results at 0% and limiting comparability.

Experimental setup

Datasets: BabyAI-Text NetHack Learning Environment MiniHack Crafter TextWorld Baba Is AI

Benchmarks: BALROG BabyAI Crafter TextWorld Baba Is AI MiniHack NetHack Learning Environment

Metrics: average progress (0-100) success rate achievement proportion

Baselines: GPT-4o GPT-4o-mini Claude 3.5 Sonnet Claude 3.5 Haiku Gemini-1.5-Pro Gemini-1.5-Flash Llama 3.1 70B Llama 3.2 90B o1-preview

Builds on: BabyAI NetHack Learning Environment MiniHack Crafter TextWorld

Closest papers by meaning

• VS-Bench: Evaluating VLMs for Strategic Abilities in Multi-Agent Environments (CVPR 2026 · similarity 0.90)
• LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Models (ICML 2025 · similarity 0.90)
• KORGym: A Dynamic Game Platform for LLM Reasoning Evaluation (NeurIPS 2025 · similarity 0.89)
• MARSHAL: Incentivizing Multi-Agent Reasoning via Self-Play with Strategic LLMs (ICLR 2026 · similarity 0.89)
• Synthesizing Multimodal Verifiable Game Data to Boost VLMs' General Reasoning (ICLR 2026 · similarity 0.89)
• TowerMind: A Tower Defence Game Learning Environment and Benchmark for LLM as Agents (AAAI 2026 · similarity 0.89)