Research map / Large language models / Agents, code and math
Factorio Learning Environment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the Factorio Learning Environment, an open-ended Python-REPL benchmark built on Factorio with lab-play (24 tasks) and open-play (build largest factory) settings, evaluating frontier LLMs on planning, spatial reasoning, and error correction.
Problem
Existing LLM agent benchmarks saturate because they have natural completion states, so new open-ended evaluations are needed that can differentiate increasingly capable models over long horizons.
Approach
A Factorio-based environment where agents write Python programs against a typed API through a persistent REPL, with an unbounded production-score reward and milestones, so there is no natural completion state and difficulty scales exponentially with the tech tree.
Key results
- Claude-3.5-Sonnet achieves the highest lab-play task success rate at 21.9 ± 1.3%, while GPT-4o-Mini reaches 5.2 ± 0.6%.
- In open-play, Claude reaches median production score 293,206 and 28 milestones, versus 54,998 PS and 26 milestones for Llama-3.3-70B.
- Agents in lab-play had 56% of successful-task steps with execution errors, and open-play error-containing steps ranged from 29.7% to 76.4%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Claude 3.5-Sonnet rather than the evaluated weaker coding models | FLE agents · Lab-play | task success rate: Claude: 21.9 $\pm$ 1.3; GPT-4o-Mini: 5.2 $\pm$ 0.6 | Helped |
| Invest in technological research and deploy electric mining drills | Claude 3.5-Sonnet · Open-play | production score (PS): from 200k to 300k; 1.5x | Helped |
| Focus on short-sighted objectives instead of long-term research and scaling production | Gemini-2 and Deepseek-v3 agents · Open-play | production score | Hurt |
Surprising findings
- Agents fall into degenerate debug loops; one GPT-4o run repeated the same incorrect API call for 78 contiguous steps.
- GPT-4o-Mini gave up in one open-play run and repeatedly asked to be reset (234 appeals).
Limitations
- Some mid and late-game entities (trains, logistics robots, circuit networks) are not modelled.
- No human baseline; unclear whether end-game goals are achievable through the Python API.
- Agents occasionally triggered game-state resets, a possible reward-hacking vector.
Experimental setup
Benchmarks: FLE lab-play (24 tasks) FLE open-play
Metrics: task success rate production score milestones assert success rate error rate
Baselines: Claude-3.5-Sonnet GPT-4o GPT-4o-Mini DeepSeek-v3 Gemini-2-Flash Llama-3.3-70B-Instruct
Builds on: ReAct-style agent scaffolding Minecraft benchmarks (MineDojo) Jericho
Closest papers by meaning
- Can large language models explore in-context? (NeurIPS 2024 · similarity 0.85)
- Scaling Prompt Synthesis for Large Language Model Reasoning (ICML 2026 · similarity 0.85)
- Large Language Models Think Too Fast To Explore Effectively (NeurIPS 2025 · similarity 0.85)
- AgentTTS: Large Language Model Agent for Test-time Compute-optimal Scaling Strategy in Complex Tasks (NeurIPS 2025 · similarity 0.85)
- AutoManual: Constructing Instruction Manuals by LLM Agents via Interactive Environmental Learning (NeurIPS 2024 · similarity 0.85)
- Joint Optimization of Tool Creation and Use for Large Language Model Agents (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.