Research map / Multi-agent systems and games / LLM agents

FutureSim: Replaying World Events to Evaluate Adaptive Agents

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds FutureSim, a chronological news-replay environment where agents forecast 330 real-world events (Jan–Mar 2026) and are scored with a multi-outcome Brier skill score, finding GPT 5.5 leads while open-weight models often have negative BSS.

Problem

Existing agent benchmarks rarely measure long-horizon adaptation to new information in open-ended real-world settings; forecasting beyond a model's knowledge cutoff offers a grounded testbed, but live prediction markets are hard to reproduce and ablate.

Approach

Replay real-world events chronologically using dated Common Crawl News snapshots and auto-generated short-answer questions, letting agents submit multi-outcome probability distributions and update them daily, scored by a proper multi-category Brier skill score that rewards abstention at zero.

Key results

• GPT 5.5 is the best agent, with top-1 accuracy of about 25% and the highest Brier skill score across the 90-day simulation.
• Open-weight frontier models have negative Brier skill scores in the final results, worse than abstaining.
• Sequential updating reaches 24.8% accuracy versus 31.2% when GPT 5.5 forecasts each question directly with full information; removing daily corpus updates drops accuracy to 17.9%.

Surprising findings

• Open-weight frontier models end with negative Brier skill score, worse than making no prediction.
• When started from the weakest agent's initial forecasts, all agents fail to recover to a Brier skill score of 0 by the end of the simulation.

Limitations

• Results depend on harness engineering; reported scores are a lower bound on capabilities.
• Purely predictive setting; cannot model agent actions that change the environment.
• Question set limited to 330 questions from Al Jazeera, with topic skew toward politics, conflict, and sports.

Experimental setup

Datasets: Common Crawl News Al Jazeera articles (question source) Polymarket markets (comparison)

Benchmarks: FutureSim

Metrics: accuracy (top-1) Brier skill score

Baselines: Claude Opus 4.6 in Claude Code DeepSeek V4 Pro GLM 5.1 Qwen3.6 Plus in OpenCode single semantic search query

Builds on: ForecastBench OpenForesight Vending-Bench SWE-Evo

Closest papers by meaning

• ForecastCompass: Guiding Agentic Forecasting with Adaptive Factor Memory (NeurIPS 2026 · similarity 0.86)
• Dyna-Mind: Learning to Simulate from Experience for Better AI Agents (ICLR 2026 · similarity 0.85)
• AdaWorld: Learning Adaptable World Models with Latent Actions (ICML 2025 · similarity 0.84)
• Forecasting in Offline Reinforcement Learning for Non-stationary Environments (NeurIPS 2025 · similarity 0.84)
• Human-Centric Open-Future Task Discovery: Formulation, Benchmark, and Scalable Tree-Based Search (AAAI 2026 · similarity 0.84)
• AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications (ICML 2026 · similarity 0.84)