Research map / Large language models / Language, safety and interpretability
Curating the Future: A Scalable Recipe for Training Open-Ended Forecasters
Nikhil Chandak, Shashwat Goel, Ameya Prabhu, Moritz Hardt, Jonas Geiping
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add leakage-removal filtering to generated questions (ablation) | RL on OpenForesight, Qwen3-8B · Validation set | accuracy: worsens without removal; improves after leakage removal | Improved |
| add all filtering stages on top of leakage removal (ablation) | RL on OpenForesight, Qwen3-8B · Validation set | accuracy, Brier score: 3x less data and steps | Improved |
| add accuracy term to Brier-only reward (ablation) | GRPO reward · Guardian validation set | accuracy, Brier score: 'Unknown' answers ~40% -> ~4% | Improved |
| add Brier term to accuracy-only reward (ablation) | GRPO reward · Guardian validation set | Brier score: negative Brier scores, worse than constant 0 | Improved |
| add retrieval of 5 news chunks (ablation) | forecasters across model families · Guardian validation set | accuracy: 9-18% accuracy gain | Improved |
| retrieve more than 5 chunks (ablation) | Qwen3-8B, gpt-oss-120b · Guardian validation set | accuracy: plateau after five chunks | No clear effect |
| train in temporal order instead of shuffled (ablation) | RL training · Validation set | accuracy | No clear effect |
| train on free-form questions instead of binary-only (ablation) | Qwen3-8B RL · Free-form validation set | free-form Brier score: 0.004 -> 0.086 | Improved |
| train on free-form only versus mix of binary and free-form (ablation) | Qwen3-8B RL · Free-form validation set | free-form Brier score: 0.086 vs 0.065 | Mixed |
| add binary questions to free-form training data (ablation) | Qwen3-8B RL · Metaculus binary questions | accuracy, binary Brier: binary-only and mix improve; free-form only does not | Improved |
| RL training on OpenForesight | Qwen3-8B · Guardian validation set, world news | Avg@3 accuracy (%): 21.7 -> 33.3 | Improved |
| RL training on OpenForesight | Qwen3-8B · Guardian validation set, sport | Avg@3 accuracy (%): 43.2 -> 43.2 | No clear effect |
Closest papers by meaning
- Approaching Human-Level Forecasting with Language Models (NeurIPS 2024 · similarity 0.89)
- Rethinking Post-Training Recipes for Multimodal Time-Series Forecasting (NeurIPS 2026 · similarity 0.87)
- Introducing FOReCAst: The Future Outcome Reasoning and Confidence Assessment Benchmark (NeurIPS 2025 · similarity 0.87)
- ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities (ICLR 2025 · similarity 0.87)
- Context is Key: A Benchmark for Forecasting with Essential Textual Information (ICML 2025 · similarity 0.87)
- LangTime: A Language-Guided Unified Model for Time Series Forecasting with Proximal Policy Optimization (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
