Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Efficient Exploration and Discriminative World Model Learning with an Object-Centric Abstraction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MEAD, a fully model-based method that learns a discriminative world model over an object-centric abstract MDP (Ab-MDP) of items and attributes, exploring with count-based MCTS and planning to goals with Dijkstra.
Problem
Hard exploration in RL with sparse rewards (2D crafting, MiniHack), where low-level pixel-based learning is sample-inefficient; the paper asks whether an object-centric item-attribute abstraction enables more efficient exploration and world-model learning.
Approach
Model the success probability of single item-attribute changes (discriminative world model) rather than a generative distribution over next abstract states, and explore with MCTS under a count-based intrinsic reward inside a learned abstract model, with goal planning via Dijkstra over negative-log success probabilities.
Key results
- MEAD learns the Ab-MDP with about an order of magnitude better sample efficiency than Dreamer-v3 and about two orders of magnitude fewer samples than PPO (hundreds of thousands vs 10 million steps).
- In the compositional planning environment, MEAD is the only method that finds the correct solution; all other methods remain stuck at zero reward.
- A learned object-centric encoder reaches planning performance similar to the ground-truth mapping despite imperfect abstract-state prediction.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use a discriminative success-probability model instead of a generative next-state model (ablation) | MEAD world-model learning · Expert-data planning experiment | planning performance | Helped |
| use count-based intrinsic reward with MCTS instead of random exploration (ablation) | MEAD exploration · Freeze-Wand | valid object-attribute transitions discovered | Helped |
| use a parametric forward model instead of empirical count estimates (ablation) | MEAD world-model learning · Model-learning comparison | sample efficiency | Helped |
| use k=16 rather than k=8 (ablation) | MEAD · Ab-MDP | performance | No clear effect |
| replace the hand-crafted object mapping with a supervised learned mapping (ablation) | MEAD · Freeze-Horn | planning performance | No clear effect |
| use PPO-learned low-level behaviour policies instead of the assumed behaviours (ablation) | MEAD · MiniHack single-task environments | performance | No clear effect |
| use two-item attribute changes instead of single-item changes (ablation) | Discriminative planning · Adversarial crafting task | planning optimality | Helped |
| pretrain on Freeze objects before the compositional task (ablation) | MEAD · Compositional MiniHack environment | reward | Mixed |
| transfer the sandbox-trained world model without task-specific reward pretraining | Dreamer-v3 · MiniHack transfer environments | performance | Hurt |
| transfer the sandbox-trained policy | PPO · MiniHack transfer environments | performance | Hurt |
| pretrain Dreamer-v3 with privileged reward information matching the downstream task | Dreamer-v3 · MiniHack transfer environments | sample efficiency | Helped |
| use count-based MCTS exploration instead of random exploration (ablation) | MEAD · World-model exploration | goal achievement | Helped |
Surprising findings
- Dreamer-v3 with count- and disagreement-based intrinsic rewards fails to get meaningful reward during sandbox fine-tuning, indicating negative transfer; PPO also shows negative transfer.
- Dreamer-v3 pretrained with privileged task-relevant reward info shows positive transfer, hinting at its representation reliance on the reward function.
Limitations
- Requires a pre-existing object-centric mapping and competent object-perturbing behaviours; discovering them automatically is left open.
- Only discrete attributes are modelled; single-item change assumption can produce incorrect plans in cases like resource consumption or multi-item dependencies.
Experimental setup
Datasets: MiniHack 2D crafting environments 100k random-policy transitions in MiniHack Freeze-Horn
Benchmarks: MiniHack 2D crafting
Metrics: episode reward vs low-level steps success probability zero-shot and few-shot transfer performance
Baselines: PPO Dreamer-v3 MuZero IMPALA RND ICM E3B
Builds on: Dreamer-v3 MuZero Options framework OO-MDP Factored MDP
Closest papers by meaning
- Learning Dynamic Attribute-factored World Models for Efficient Multi-object Reinforcement Learning (NeurIPS 2023 · similarity 0.89)
- Curious Exploration via Structured World Models Yields Zero-Shot Object Manipulation (NeurIPS 2022 · similarity 0.89)
- Model Learning for Look-Ahead Exploration in Continuous Control (AAAI 2019 · similarity 0.89)
- Kinematic State Abstraction and Provably Efficient Rich-Observation Reinforcement Learning (ICML 2020 · similarity 0.89)
- Explicit Explore-Exploit Algorithms in Continuous State Spaces (NeurIPS 2019 · similarity 0.89)
- Learning Interactive World Model for Object-Centric Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.