Research map / Large language models / Retrieval and knowledge

CruxBench: A Benchmark of Information Discovery

NeurIPS 2026 · Large language models › Retrieval and knowledge · approach: Explanatory and multiple-choice QA benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces CruxBench, a contamination-resistant benchmark scoring LLM-generated forecasting subquestions ("cruxes") by market-based value of information (KL divergence of Polymarket price updates), across 8 models on 293 questions.

Problem

Existing LLM benchmarks evaluate answer accuracy against fixed labels, not the ability to identify which subquestions (cruxes) are worth asking to inform a difficult forecasting target; such evaluation is also vulnerable to contamination.

Approach

Grade LLM-proposed cruxes by the realized belief update (KL divergence of prediction-market prices before and after the crux resolves) as an unbiased estimator of value of information, using future real-world events so ground truth cannot be contaminated.

Key results

• Claude-Opus-4.6 achieves the highest VOI (0.016 bits, 95% CI [0.009, 0.023]), only narrowly above the random-timing baseline (0.007) and far below the hindsight oracle (0.129).
• VOI correlates with Epoch Capability Index at r=0.90 (p<0.01) across eight generators.
• Cruxes from higher-VOI generators improve downstream forecaster Brier score, e.g., Claude-Opus-4.6 cruxes lower Brier from 0.198 to 0.176 and raise AUC from 0.690 to 0.730.

Surprising findings

• Only the three frontier models significantly outperform a random-timing baseline; the remaining models are not distinguishable from it.
• Semantically similar cruxes (cosine ~0.92) can receive very different VOI scores because of differences in resolution criteria, such as arrest versus arrest-and-charge.

Limitations

• Per-crux VOI is not identifiable, so only population-level estimates are reported.
• Market price updates may be confounded by simultaneous information arrivals and market inefficiencies.
• Retrospective rather than live evaluation, limiting testing of the latest models due to contamination concerns.

Experimental setup

Datasets: Polymarket prediction market price histories AskNews news articles

Benchmarks: CruxBench

Metrics: VOI (KL divergence, bits) resolvable rate Brier score AUC Pearson correlation with ECI

Baselines: random-timing baseline hindsight oracle GPT-3.5-Turbo Llama-3.1-8B-Instruct Llama-3.3-70B DeepSeek-V3.1 Qwen3-235B-A22B Gemini-3.1-Pro-Preview

Builds on: expected information gain Epoch Capability Index

Closest papers by meaning

• AutoBencher: Towards Declarative Benchmark Construction (ICLR 2025 · similarity 0.87)
• DiscoveryBench: Towards Data-Driven Discovery with Large Language Models (ICLR 2025 · similarity 0.86)
• metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.86)
• Evaluating language models as risk scores (NeurIPS 2024 · similarity 0.86)
• AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions (NeurIPS 2025 · similarity 0.86)
• Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle (ICML 2025 · similarity 0.85)