Research map / Large language models / Language, safety and interpretability

Efficient Prediction of Pass@k Scaling in Large Language Models

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: LLM pruning and scaling data · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a beta-binomial fit of per-problem difficulty plus a dynamic (hardest-first) sampling allocation to predict pass@k at large k from small sampling budgets, outperforming log-log regression and discretized-beta fitting.

Problem

Predicting pass@k for very large k (repeated-sampling success or jailbreak rates) from a small sampling budget, where existing power-law regression and discretized-beta fits are statistically flawed and biased.

Approach

Model per-problem success probabilities with a beta distribution fit directly by beta-binomial maximum likelihood on variable per-problem counts (instead of discretized bins or log-log regression), and allocate samples adaptively to the hardest problems (fewest successes, then fewest attempts).

Key results

• Proposed beta-binomial plus dynamic sampling method achieves lower MSE than log-log regression and discretized beta across AdvBench, MATH, and Code Contests for budgets 10^1 to 10^4 (paper claims closest predictions to ground truth virtually everywhere).
• Log-log regression often predicts impossible pass rates above 1 at high k; discretized beta consistently underestimates pass@k at large k.
• Paper reports that for some models its MSE is more than 10x lower than counterparts.

Surprising findings

• Using the estimated scale parameter theta does not improve predictions empirically, despite the theory suggesting it could help.

Limitations

• Dynamic sampling benefits are hard to isolate empirically on the real datasets; main support comes from synthetic experiments.
• Theoretical variance-optimal allocation is derived for a frequentist estimator, not the Bayesian multi-stage estimator actually used.
• Ground truth relies on a withheld 10,000-sample set per problem, and data comes from prior work's models rather than newly sampled models.

Experimental setup

Datasets: AdvBench MATH CodeContests

Benchmarks: AdvBench MATH CodeContests

Metrics: mean squared error squared error of pass@k prediction

Baselines: Log-Log Regression Discretized Beta

Builds on: pass@k unbiased estimator discretized beta fitting (Hughes et al.) log-log linear regression power law fitting beta-binomial model

Closest papers by meaning

• Language models scale reliably with over-training and on downstream tasks (ICLR 2025 · similarity 0.88)
• A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.88)
• Prescriptive Scaling Reveals the Evolution of Language Model Capabilities (ICML 2026 · similarity 0.88)
• Why Has Predicting Downstream Capabilities of Frontier AI Models with Scale Remained Elusive? (ICML 2025 · similarity 0.88)
• Predicting Emergent Abilities with Infinite Resolution Evaluation (ICLR 2024 · similarity 0.87)
• Statistical Estimation of Adversarial Risk in Large Language Models under Best-of-N Sampling (NeurIPS 2026 · similarity 0.87)