Research map / Large language models / Language, safety and interpretability
Position: Don't Use the CLT in LLM Evals With Fewer Than a Few Hundred Datapoints
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Position paper arguing that CLT-based error bars for LLM evals undercover on small benchmarks, and recommending Wilson, Clopper-Pearson, and Bayesian (conjugate and importance-sampled) credible intervals, with a Python library.
Problem
CLT-based confidence intervals are routinely used for LLM benchmark error bars, but with the small, specialized benchmarks (tens to hundreds of items) that are increasingly common they severely undercover and can produce invalid intervals outside [0,1] or of zero width.
Approach
Argues that CLT-based intervals should be replaced in small-data LLM evals by Wilson/Clopper-Pearson intervals for single proportions and Bayesian credible intervals (Beta-Bernoulli, importance-sampled clustered and paired models, Dirichlet for F1), validated through coverage simulations on known ground-truth parameters.
Key results
- For N=100, nominal 95% CLT-based intervals achieve only about 92.5% empirical coverage in the IID Bernoulli simulation, while Wilson and Bayesian intervals track nominal coverage.
- CLT-based and bootstrap intervals collapse to zero width or extend beyond [0,1] on small real benchmarks such as LangChain (N=20) and AIME 2025 II (N=15).
- In the clustered and paired simulations, only the clustered Bayesian (importance-sampled) and paired Bayesian methods achieve near-nominal coverage at small N, while non-Bayesian methods severely underperform.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CLT-based intervals instead of Wilson or Bayesian intervals | Single-model accuracy on IID Bernoulli questions · IID Bernoulli simulation, including N=100 | coverage: 95% nominal -> 92.5% at N=100 | Hurt |
| Use bootstrap intervals instead of Wilson or Bayesian intervals | Single-model accuracy on IID Bernoulli questions · IID Bernoulli simulation | coverage | Hurt |
| Use Wilson score intervals instead of CLT-based intervals | Single-model accuracy on IID Bernoulli questions · IID Bernoulli simulation | coverage | Helped |
| Use Beta-Bernoulli Bayesian credible intervals instead of CLT-based intervals | Single-model accuracy on IID Bernoulli questions · IID Bernoulli simulation | coverage | Helped |
| Use Clopper-Pearson intervals instead of Wilson intervals | Single-model accuracy on IID Bernoulli questions · IID Bernoulli simulation | interval width | Hurt |
| Use a clustered Bayesian model instead of simple or clustered CLT intervals | Model accuracy on clustered questions · Clustered questions simulation | coverage | Helped |
| Use CLT-based intervals instead of Bayesian intervals | Difference in independent model accuracies · Independent model comparison | coverage | Hurt |
| Use Fisher's exact test intervals instead of Bayesian intervals | Odds ratio between independent model accuracies · Independent model comparison | coverage | Mixed |
| Use Bayesian intervals instead of non-Bayesian methods | Difference in paired model accuracies · Paired model comparison | coverage | Helped |
| Account for paired response correlations with paired Bayes instead of unpaired Bayes (ablation) | Difference in paired model accuracies · Paired model comparison | interval width | Helped |
| Use Bayesian credible intervals instead of bootstrap intervals | F1 score from a confusion matrix · Nonlinear metric simulation | coverage | Helped |
| Use CLT-based intervals instead of Bayesian credible intervals | LLM evaluation metrics in small-data settings · LangChain tool-use benchmark (N=20) | interval validity | Hurt |
Surprising findings
- Bootstrap intervals also perform poorly on small samples, not just CLT-based ones.
- Fisher's exact test for odds ratios corresponds to Bayesian analysis with improper priors and is overly conservative.
Limitations
- Simulations use uniform or assumed generative priors, so results depend on prior choice; informative priors may be needed for optimal efficiency.
- Clustered and paired importance-sampling Bayesian methods are less readily available and harder to implement than conjugate methods.
Experimental setup
Datasets: LangChain tool-use benchmark (26-tool typewriter task, N=20) Math Arena AIME 2025 II (N=15)
Benchmarks: LangChain tool-use benchmark Math Arena AIME 2025 II
Metrics: coverage probability coverage error interval width
Baselines: CLT-based intervals bootstrap intervals Wilson score interval Clopper-Pearson interval Fisher's exact test
Builds on: Wilson score interval Clopper-Pearson interval delta method
Closest papers by meaning
- Benchmarking LLMs via Uncertainty Quantification (NeurIPS 2024 · similarity 0.91)
- An Empirical Analysis of Uncertainty in Large Language Model Evaluations (ICLR 2025 · similarity 0.90)
- Large Language Models Must Be Taught to Know What They Don’t Know (NeurIPS 2024 · similarity 0.88)
- Measuring what Matters: Construct Validity in Large Language Model Benchmarks (NeurIPS 2025 · similarity 0.88)
- LLMEval: A Preliminary Study on How to Evaluate Large Language Models (AAAI 2024 · similarity 0.88)
- Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.