Research map / Large language models / Retrieval and knowledge
Mars: Situated Inductive Reasoning in an Open-World Environment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Mars, a counter-commonsense Crafter-based interactive environment for situated inductive reasoning, and proposes Induction from Reflection (IfR), an LLM pipeline that induces game rules from trajectories; RL and LLM agents struggle.
Problem
Existing benchmarks evaluate inductive reasoning on pre-defined evidence or evaluate knowledge application without active interaction; LLMs and RL agents lack situated inductive reasoning, i.e., inducing new rules from interaction in an environment and applying them in context.
Approach
Modify Crafter's terrain, survival settings, and task dependencies with counter-commonsense rules under principles (resource balance, supply exceeds demand, achievability), and have an LLM controller reflect on its trajectory after each subgoal to induce rules stored in a rule library.
Key results
- Induction from Reflection (IfR) achieves reward 9.0 on the Default world and 6.9 on Terr. Task., outperforming ReAct, Reflexion and Skill Library on most worlds.
- DreamerV3 reaches only about 4% score in the All three world, while the best LLM method reaches 0.6% in the same setting.
- Recall of ground-truth rules by induced rule library is only about 28%, indicating large room for improvement.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add induction from reflection to Skill Library | GPT-4-0125-preview / Skill Library · Mars worlds | score (%): Default: 8.3^{\pm1.3} -> 13.0^{\pm2.1}; Terrain: 11.9^{\pm3.4} -> 11.8^{\pm2.9}; Survival: 9.7^{\pm2.0} -> 11.0^{\pm3.7} | Mixed |
| Add induction from reflection to Skill Library | GPT-4-0125-preview / Skill Library · Mars worlds | reward: Default: 8.0^{\pm2.1} -> 9.0^{\pm2.3}; Terrain: 9.5^{\pm2.9} -> 8.0^{\pm3.7}; Survival: 7.9^{\pm2.9} -> 7.7^{\pm3.7}; Ta | Mixed |
| Provide game rules in context | ReAct · Default | score (%): 8.0% -> 11.6% | Helped |
| Provide game rules in context | ReAct · Surv. Task. | score (%): 1.3% -> 9.2% | Helped |
| Provide Crafter knowledge in context (ablation) | ReAct · Default | reward: 7.7^{\pm1.6} -> 7.9^{\pm2.7} | No clear effect |
| Provide Crafter knowledge in context (ablation) | ReAct · Task. Dep. | reward: 5.0^{\pm2.1} -> 1.8^{\pm0.5} | Hurt |
| Provide Crafter knowledge in context (ablation) | ReAct · Mars worlds | reward: Terrain: 7.4^{\pm2.7} -> 7.8^{\pm3.1}; Survival: 6.4^{\pm3.7} -> 7.0^{\pm4.1}; Terr. Surv.: 6.7^{\pm2.5} -> 6.8\pm1.7; T | Mixed |
Surprising findings
- Providing Crafter's game knowledge in-context gives minimal improvement and hurts performance on the Task. Dep world.
- Giving ReAct the ground-truth rules in context improves it only modestly and rules-based 'Surv. Task. w/ rules' still lags the Default world.
Limitations
- Overall performance of IfR remains suboptimal, with limited exploration time (five episodes).
- Rule recall is only about 28%, limiting the induction capability.
- Induction is enforced from historical trajectories rather than triggered automatically.
Experimental setup
Datasets: Crafter
Benchmarks: Mars (7 worlds: Default, Terrain, Survival, Task. Dep, Terr. Surv., Terr. Task., Surv. Task., All three)
Metrics: reward success rate overall score (log-space) rule precision rule recall
Baselines: PPO DreamerV3 ReAct Reflexion Skill Library
Builds on: Crafter Voyager Reflexion ReAct JARVIS-1
Closest papers by meaning
- Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement (ICLR 2024 · similarity 0.87)
- LeAct: Learning to Reason from Expert Actions (NeurIPS 2026 · similarity 0.86)
- Hypothesis Search: Inductive Reasoning with Language Models (ICLR 2024 · similarity 0.86)
- From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information? (ICML 2025 · similarity 0.86)
- Real-Time Reasoning Agents in Evolving Environments (ICLR 2026 · similarity 0.86)
- MIRAGE: Evaluating and Explaining Inductive Reasoning Process in Language Models (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.