Research map / Large language models / Agents, code and math
TALES: Text-Adventure Learning Environment Suite
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces TALES, a unified suite of text-adventure environments (Jericho, ALFWorld, ScienceWorld, TextWorld, TextWorldExpress) plus a new Simon Says task, and evaluates 34 LLM agents zero-shot, finding that even top models fail to exceed 15% on human-written games.
Problem
Assessing baseline agentic reasoning of LLMs in grounded, natural-language text environments, where failures in interface interpretation are not confounded with reasoning limitations; existing agent benchmarks rely on structured interfaces or add heavy expert scaffolding.
Approach
Unify five existing text-game frameworks in canonical form under minimal scaffolding (a single generic system prompt, no expert hints), organize them by four reasoning skills (spatial, deductive, inductive, grounded), and add a Simon Says gating task whose success predicts progress on the harder games.
Key results
- Top zero-shot model Claude-3.5-Sonnet averages 50.4 across frameworks; Claude-3.7-Sonnet averages 52.5 with 12.5 on Jericho, and no agent reaches 15% on Jericho.
- Simon Says with Memory success strongly predicts progress on TALES (Pearson r = 0.83).
- Even the top models fail to exceed about 50% of available score on Zork1 within 100 steps (top agents below 0.291 normalized score).
Surprising findings
- Despite likely training contamination on Zork1 playthroughs, top agents fail to leverage parametric knowledge to progress.
- o1 fails on the most difficult Simon Says mode while nearly all other selected LLMs achieve near-maximum scores.
Limitations
- Results are zero-shot with a 100-step cap due to compute and cost limits.
- Gemini-2.5-pro-preview and Gemini-2.0-flash scores are over one seed only.
- Qualitative analysis covers only a subset of models and games; Claude-3.7-Sonnet reasoning analysis is preliminary.
Experimental setup
Datasets: Jericho ALFWorld ScienceWorld TextWorld TextWorldExpress
Benchmarks: TALES Jericho ALFWorld ScienceWorld TextWorld TextWorldExpress Simon Says
Metrics: normalized game score (0-100) token usage score standard deviation across seeds
Baselines: Claude-3.7-Sonnet Claude-3.5-Sonnet gemini-2.5-pro-preview o1 gpt-4o DeepSeek-R1 o3-mini Llama-3.1-405B-Instruct
Builds on: Jericho ALFWorld ScienceWorld TextWorld TextWorldExpress ECBD
Closest papers by meaning
- CivBench: A Long-Horizon Benchmark for Tool-Mediated Agents in Civilization VI (NeurIPS 2026 · similarity 0.87)
- BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games (ICLR 2025 · similarity 0.87)
- Estimating the Empowerment of Language Model Agents (ICML 2026 · similarity 0.86)
- AgentOdyssey: Open-Ended Long-Horizon Text Game Generation for Test-Time Continual Learning Agents (NeurIPS 2026 · similarity 0.86)
- SmartPlay : A Benchmark for LLMs as Intelligent Agents (ICLR 2024 · similarity 0.86)
- Evaluating LLMs in Open-Source Games (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.