Research map / Large language models / Agents, code and math
PoE-World: Compositional World Modeling with Products of Programmatic Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PoE-World, a world model built as an exponentially weighted product of many small LLM-synthesized Python programmatic experts, fit by L-BFGS, which learns Atari Pong and Montezuma's Revenge dynamics from a brief demonstration and supports planning.
Problem
Symbolic, program-structured world models have been limited to text and gridworld domains; a single monolithic program does not scale to complex, stochastic, partially observed Atari games with few demonstrations.
Approach
Decompose a world program into hundreds of small LLM-synthesized deterministic programs treated as experts, combine them via a product of experts with learned exponential weights (pruning low-weight experts), add hard constraints for sharpening, and use a hierarchical TAMP-inspired planner over object-contact abstract states.
Key results
- PoE-World + Planner reaches 100 ± 0.00 on Montezuma's Revenge and Montezuma's Revenge-Alt with at most 3k agent steps, while PPO at 100k env steps scores 0 on both and WorldCoder + Planner scores 0.
- On Pong, PoE-World + Planner scores -12.33 ± 0.88 vs WorldCoder + Planner -17.00 ± 3.00 (PPO @100k: -21.00).
- PoE-World's world models have 4000+ lines of code for Montezuma's Revenge versus under 100 lines for WorldCoder; next-observation accuracy on MR test frames is 0.31 vs 0.10 for WorldCoder.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PoE-World world modeling with the planner instead of PPO, ReAct, or WorldCoder + Planner | Atari agents · Pong, Pong-Alt, MR, and MR-Alt | score | Helped |
| Use PoE-World instead of WorldCoder | Symbolic world modeling · Pong and MR training demonstrations | next observation prediction accuracy: Pong: 0.33 ± 0.01 -> 0.51 ± 0.01; MR: 0.36 ± 0.02 -> 0.75 ± 0.00 | Helped |
| Use PoE-World instead of WorldCoder | Symbolic world modeling · Pong test (1000 random frames) | next observation prediction accuracy: 0.06 ± 0.01 vs 0.06 ± 0.01 | No clear effect |
| Use PoE-World instead of WorldCoder | Symbolic world modeling · Pong-Alt, MR, and MR-Alt test (1000 random frames) | next observation prediction accuracy: Pong-Alt: 0.01 ± 0.01 -> 0.03 ± 0.00; MR: 0.10 ± 0.01 -> 0.31 ± 0.00; MR-Alt: 0.12 ± 0.03 -> 0.43 ± 0.00 | Helped |
| Use PoE-World instead of WorldCoder | Symbolic world modeling · Pong and MR training demonstrations; Pong-Alt, MR, and MR-Alt test | object attributes prediction accuracy: Pong train: 0.82 ± 0.00 -> 0.88 ± 0.00; MR train: 0.75 ± 0.01 -> 0.93 ± 0.00; Pong-Alt test: 0.67 ± 0.02 -> 0.76 ± 0.00; | Helped |
| Use PoE-World instead of WorldCoder | Symbolic world modeling · Pong test (1000 random frames) | object attributes prediction accuracy: 0.66 ± 0.01 vs 0.66 ± 0.00 | No clear effect |
| Add hard constraints (ablation) | PoE-World + Planner · MR and MR-Alt | planning successes: MR: 2/9 -> 5/9; MR-Alt: 2/9 -> 4/9 | Helped |
| Add hard constraints (ablation) | PoE-World · MR and MR-Alt test (1000 random frames) | next observation prediction accuracy: MR: 0.30 ± 0.01 -> 0.31 ± 0.00; MR-Alt: 0.43 ± 0.00 -> 0.43 ± 0.00 | No clear effect |
| Pretrain PPO inside the PoE-World world model, then fine-tune in Atari | PPO · Atari environment | score | Helped |
Surprising findings
- Removing hard constraints drops planning successes on MR from 5/9 to 2/9 even though next-observation prediction accuracy barely changes, which the authors interpret as damage control for long-horizon planning.
- PPO pretrained for 20M steps on base Montezuma's Revenge still never achieves a positive score on the game, while PoE-World's planner does.
Limitations
- Relies on symbolic object-centric inputs from manually patched OCAtari; does not learn from pixels and is infeasible to apply across the full Atari suite.
- Does not address exploration, reward learning, or planning efficiency; only evaluated on two Atari games.
Experimental setup
Datasets: Atari Pong demonstration Atari Montezuma's Revenge demonstration Pong-Alt (constructed) Montezuma's Revenge-Alt (constructed)
Benchmarks: Pong Pong-Alt Montezuma's Revenge Montezuma's Revenge-Alt
Metrics: game score planning successes next observation prediction accuracy next observation object attribute prediction accuracy
Baselines: PPO LLM as Agent (ReAct) WorldCoder WorldCoder + Planner Diamond
Builds on: WorldCoder REx ReAct PPO OCAtari
Closest papers by meaning
- WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environment (NeurIPS 2024 · similarity 0.89)
- Reinforcement World Model Learning for LLM-based Agents (NeurIPS 2026 · similarity 0.88)
- WALL-E: World Alignment by NeuroSymbolic Learning improves World Model-based LLM Agents (NeurIPS 2025 · similarity 0.88)
- Do LLMs Build World Representations? Probing Through the Lens of State Abstraction (NeurIPS 2024 · similarity 0.87)
- Generating Code World Models with Large Language Models Guided by Monte Carlo Tree Search (NeurIPS 2024 · similarity 0.87)
- Agent2World: Learning to Generate Symbolic World Models via Adaptive Multi-Agent Feedback (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.