Research map / Large language models / Language, safety and interpretability
FactorSim: Generative Simulation via Factorized Representation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces FactorSim, which generates full coded simulations from long text specs by decomposing into CoT steps and using a factored POMDP representation with model-view-controller prompts to limit context per step, evaluated on a new 8-game generative simulation benchmark and GenSim robotics tasks.
Problem
Generating complete simulations in code (game logic, dynamics, rendering) from detailed natural-language specifications for training RL agents; LLMs ignore or violate parts of long prompts when generating large codebases.
Approach
Model the simulation as a factored POMDP and generate it step by step, selecting only the relevant state variables and factors (scope sets) as context for each step, and split each step into controller, model, and view prompts from the model-view-controller pattern.
Key results
- GPT-4 w/ FactorSim achieves 78% on Flappy Bird and 81% on Puckworld system tests versus 35% and 34% for GPT-4 CoT w/ self-debug baseline in those games' comparisons; it outperforms all baselines in 7 of 8 games.
- On GenSim robotics tasks, FactorSim reaches 92% syntax pass, 58% runtime pass, and 25% human pass rate versus 75%/8%/0% for GenSim vanilla.
- Llama-3 w/ FactorSim reaches 50% on Snake and 58% on Puckworld system tests, improving over Llama-3 baselines substantially.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FactorSim instead of GPT-4 baseline generation | GPT-4 simulation generation · Flappy Bird | system-test pass rate: 0.35 -> 0.78 | Helped |
| Use FactorSim instead of GPT-4 self-debug generation | GPT-4 simulation generation · Catcher | system-test pass rate: 0.53 -> 0.66 | Helped |
| Use FactorSim instead of GPT-4 self-debug generation | GPT-4 simulation generation · Snake | system-test pass rate: 0.43 -> 0.44 | Helped |
| Use FactorSim instead of GPT-4 CoT self-debug generation | GPT-4 simulation generation · Pixelcopter | system-test pass rate: 0.53 -> 0.78 | Helped |
| Use FactorSim instead of GPT-4 self-debug generation | GPT-4 simulation generation · Pong | system-test pass rate: 0.75 -> 0.61 | Hurt |
| Use FactorSim instead of Llama-3 CoT self-debug generation | Puckworld simulation generation · Puckworld | system-test pass rate: 0.50 -> 0.81 | Helped |
| Use FactorSim instead of GPT-4 CoT self-debug generation | GPT-4 simulation generation · Waterworld | system-test pass rate: 0.50 -> 0.62 | Helped |
| Use FactorSim instead of Llama-3 CoT self-debug generation | Monster Kong simulation generation · Monster Kong | system-test pass rate: 0.35 -> 0.44 | Helped |
| Use FactorSim instead of GenSim vanilla generation | Robotics task generation · GenSim assembly tasks | runtime pass rate: 8% -> 58% | Helped |
| Use FactorSim instead of GenSim vanilla generation | Robotics task generation · GenSim assembly tasks | human pass rate: 0% -> 25% | Helped |
| Use FactorSim instead of GenSim vanilla generation | Robotics task generation · GenSim assembly tasks | syntax pass rate: 75% -> 92% | Helped |
| Use FactorSim instead of GenSim Chain of Thought (topdown) | Robotics task generation · GenSim assembly tasks | task completion pass rate: 8% -> 33% | Helped |
Surprising findings
- Llama-3 vanilla and self-debug produce highly similar incorrect implementations, suggesting mode collapse toward priors for games like Pong and Flappy Bird.
- AgentCoder performs poorly because its test designer agent writes incorrect or infeasible tests on complex specifications.
Limitations
- Only single-agent 2D games and robotics tasks evaluated; no 3D games or multi-agent settings.
- Controller reference supplied manually to all methods, and game prompts were manually supplemented with missing details.
- Robotics task completion rates are low due to limitations of the oracle agent.
Experimental setup
Datasets: PyGame Learning Environment (8 2D games) GenSim 50-task robotics benchmark CLIPort
Benchmarks: Generative Simulation benchmark (8 PyGame games) GenSim 50-task robotics benchmark
Metrics: percent of system tests passed zero-shot transfer reward (normalized to random and PPO grou human playability rating (1-4) syntax, runtime, task completion, and human pass rates for r token usage
Baselines: Mistral-7B-Instruct Llama-3 vanilla Llama-3 self-debug Llama-3 CoT w/ self-debug GPT-3.5 GPT-4 GPT-4 w/ self-debug GPT-4 w/ AgentCoder GPT-4 CoT w/ self-debug GenSim vanilla GenSim chain of thought (topdown) GenSim chain of thought (bottomup)
Builds on: chain-of-thought prompting self-debugging AgentCoder GenSim PPO
Closest papers by meaning
- Learning Interactive Real-World Simulators (ICLR 2024 · similarity 0.87)
- RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation (ICML 2024 · similarity 0.87)
- Grounded Answers for Multi-agent Decision-making Problem through Generative World Model (NeurIPS 2024 · similarity 0.87)
- GenSim: Generating Robotic Simulation Tasks via Large Language Models (ICLR 2024 · similarity 0.86)
- ReGen: Generative Robot Simulation via Inverse Design (ICLR 2025 · similarity 0.86)
- Learning to Cooperate with Humans using Generative Agents (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.