Research map / Multi-agent systems and games / Human-AI and goal-conditioned
MARPLE: A Benchmark for Long-Horizon Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MARPLE, a whodunit-style benchmark for long-horizon multimodal (vision, language, audio) inference in procedurally generated household Gridworlds, comparing MCTS with learned agent models, GPT-4, and human experts.
Problem
Evaluating whether AI models can perform long-horizon, multimodal event reconstruction, i.e., infer which agent caused a state change from step-by-step evidence, which prior short-horizon or single-modality benchmarks do not cover.
Approach
Procedurally generated household Gridworld (extending Mini-BEHAVIOR) with hierarchical planners producing two-agent trajectories and synchronized vision, language (intent) and audio evidence; the task asks which agent is responsible for a query state, with accuracy tracked as evidence is revealed over time.
Key results
- Humans reach 0.8 accuracy with about 48% of evidence, requiring 10% less evidence than the best in-distribution MCTS variant and 47% less than GPT-4.
- Out-of-distribution, the best simulation model (vision+audio+language) needs 81% of evidence to reach 0.8 accuracy versus 48% for humans.
- GPT-4 fails to converge on the 'Who turned on the shower?' and 'Who turned on the laundry?' scenarios, biased toward agent-state changes rather than environment-state changes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use human participants instead of the strongest in-distribution MCTS variant | MARPLE inference · MARPLE, all five scenarios, in-distribution | evidence fraction to reach 0.8 inference accuracy: 0.58 -> 0.48 | Helped |
| Use human participants instead of the strongest out-of-distribution MCTS variant | MARPLE inference · MARPLE, all five scenarios, out-of-distribution | evidence fraction to reach 0.8 inference accuracy: 0.81 -> 0.48 | Helped |
| Train and test in unseen procedurally generated environments instead of the same environments | Vision + Audio + Language MCTS · MARPLE, all five scenarios | evidence fraction to reach 0.8 inference accuracy: 0.58 -> 0.81 | Hurt |
| Add language conditioning to the vision-only policy (ablation) | Simulation policy · Inference trajectories | action prediction accuracy: 0.86 -> 0.92 | Helped |
| Add audio evidence to the vision-only model (ablation) | Simulation baseline · MARPLE multimodal inference | inference accuracy | Helped |
| Add language evidence to the vision-only model (ablation) | Simulation baseline · MARPLE multimodal inference | inference accuracy | Helped |
| Use vision, audio, and language rather than fewer modalities (ablation) | Simulation baseline · Who turned on the shower? | inference accuracy | Helped |
| Increase agent mission preference similarity from 1.0/0.0 to 0.6/0.4 (ablation) | Vision-Only and Vision + Audio models · Who turned on the shower? | evidence fraction to reach 0.6 inference accuracy: around 40% -> 85%; around 40% -> 70% | Hurt |
| Add in-context examples to GPT-4 zero-shot prompting (ablation) | GPT-4-0613 · Who turned on the shower?; Who turned on the laundry? | inference accuracy | Helped |
Surprising findings
- GPT-4 fails to converge in two scenarios because the query state is an environment change rather than an agent-state change, and in-context examples only partially help.
- Language evidence (intent) contributed more than audio to simulation-model performance.
Limitations
- Only two agents and whodunit scenarios where the query state is caused by exactly one agent.
- Gridworld abstraction lacks physical realism and photorealistic or realistic audio.
- Human baseline uses only two experts.
Experimental setup
Datasets: MARPLE training set MARPLE test set
Benchmarks: MARPLE
Metrics: inference accuracy over time fraction of evidence needed to reach 0.8 accuracy
Baselines: GPT-4-0613 zero-shot GPT-4 in-context learning Vision-only MCTS simulation Vision + Audio + Language MCTS simulation Llama-3.1-8B-Instruct Qwen2-7B-Instruct human experts
Builds on: Mini-BEHAVIOR Monte Carlo tree search chain-of-thought prompting
Closest papers by meaning
- AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications (ICML 2026 · similarity 0.86)
- Evolving Minds: Logic-Informed Inference from Temporal Action Patterns (ICML 2025 · similarity 0.86)
- WorldMemArena: Evaluating Multimodal Agent Memory Through Action–World Interaction (NeurIPS 2026 · similarity 0.85)
- ACED-Bench: Evaluating How LLM Agents Acquire and Act on Causal Evidence (NeurIPS 2026 · similarity 0.85)
- Episodic Memories Generation and Evaluation Benchmark for Large Language Models (ICLR 2025 · similarity 0.85)
- Reflection-Bench: Evaluating Epistemic Agency in Large Language Models (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.