Research map / Multi-agent systems and games / LLM agents
EcoGym: Evaluating LLMs for Long-Horizon Plan-and-Execute in Interactive Economies
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces EcoGym, an open benchmark of three interactive economic environments (Vending, Freelance, Operation) for evaluating LLM agents on long-horizon plan-and-execute decision making over up to 365 simulated days.
Problem
Existing long-horizon agent benchmarks are episodic, domain-specific, or closed, and do not capture persistent economic dynamics with hidden mechanics and partial observability.
Approach
A unified, budgeted-action decision framework with three economic environments and hidden latent dynamics (demand elasticity/seasonality, burnout thresholds, DAU decay), scored by business outcomes (net worth, income, DAU) over an effectively unbounded horizon; open-sources an adapted Vending-Bench and adds Freelance and Operation environments.
Key results
- Gemini-3-Pro leads Vending (11272.47 net worth) and Freelance (2696.58 income), while Claude-Sonnet-4.5 leads Operation (1595.00 DAU); no single model dominates across the three environments.
- Claude-Sonnet-4.5 surpasses the human expert baseline of 1404 average DAU in Operation.
- Adding memory modules helps Gemini-3-Pro in Vending (up to 18939 with episodic memory) but not uniformly; Gemini-3-Flash benefits most from working memory (10099).
Surprising findings
- Expanding context window beyond 128 steps does not give consistent gains; Gemini-3-Pro degrades at k=1024.
- Gemini-3-Pro fails to exploit larger inventory in Medium and Large Vending settings, while Gemini-3-Flash scales with complexity.
Limitations
- Relies on rule-based abstractions that may not capture irrational human behavior.
- Studies only single-agent interactions; multi-agent market dynamics not included.
- High variance in Vending requires multiple runs; reduced run counts for Freelance and Operation to save cost.
Experimental setup
Datasets: LiveCodeBench SWE-bench FinQA BizFinBench GSM8K SciQ LEXam Pile-of-law
Benchmarks: EcoGym Vending EcoGym Freelance EcoGym Operation
Metrics: net worth income DAU
Baselines: Claude-Sonnet-4.5 Gemini-3-Pro Gemini-3-Flash GPT-5.2 Kimi-K2.6 Qwen3.5-397b-A17b DeepSeek-v3.2 GLM-5 MiniMax-M2.7 Human experts
Builds on: Vending-Bench
Closest papers by meaning
- UltraHorizon: Benchmarking LLM-Agent Capabilities in Ultra Long-Horizon Scenarios (ICML 2026 · similarity 0.87)
- STEER: Assessing the Economic Rationality of Large Language Models (ICML 2024 · similarity 0.87)
- EconGym: A Scalable AI Testbed with Diverse Economic Tasks (NeurIPS 2025 · similarity 0.87)
- Autoregressive Diffusion World Models for Off-Policy Evaluation of LLM Agents (NeurIPS 2026 · similarity 0.86)
- Selective Critique for Cost-Aware LLM Agents in Long-Horizon Decision Making (NeurIPS 2026 · similarity 0.86)
- Think Twice, Act Once: A Co-Evolution Framework of LLM and RL for Large-Scale Decision Making (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.