Research map / Large language models / Language, safety and interpretability
CELEUS: Certifiable and Efficient LLM Evaluation via E-Processes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CELEUS, an e-process-based framework giving anytime-valid confidence intervals for LLM evaluation, combining uncertainty-guided sampling and surrogate-assisted approximations to reach a target precision with fewer evaluated samples.
Problem
Certifiable LLM evaluation with sequential stopping is not anytime-valid when CIs are repeatedly updated and used to decide when to stop, and evaluating samples (especially human or judge-based) is costly.
Approach
Builds a conditionally unbiased per-round signal combining observed risks, surrogate scores for unevaluated samples, and an inverse-probability-weighted surrogate-residual correction, then turns it into anytime-valid e-process CIs via test inversion with a Hoeffding correction to the population risk; sampling probabilities are proportional to proxy residual magnitudes.
Key results
- CELEUS reaches target precision 0.05 with 54%, 60%, and 62% fewer evaluated samples than EValue on SST-2, MMLU, and AG News.
- Cer-Eval miscoverage is 0.161 versus nominal 0.05, while e-process methods range 0.021-0.030.
- Surrogate-assisted CELEUS further reduces labels by 10-23% relative to the no-surrogate variant using 7-8B surrogates for 67-72B targets.
Surprising findings
- Cer-Eval, a state-of-the-art certifiable evaluation method, exceeds nominal miscoverage (0.161 vs 0.05) because its adaptive partitioning reuses data.
- Surrogate-assisted approach with a 7-8B model cuts labels needed for 67-72B targets, though the paper notes per-round runtime is higher than EValue.
Limitations
- Surrogate scoring adds computational cost and per-round runtime is higher than EValue.
- Population-level guarantee relies on i.i.d. pool assumption via Hoeffding correction; non-i.i.d. pools need dependent-data bounds.
- Coverage guarantees require bounded evaluation scores (normalization or clipping).
Experimental setup
Datasets: SST-2 AG News MMLU
Benchmarks: SST-2 MMLU AG News
Metrics: miscoverage rate number of evaluated samples to reach target precision CI width mean squared error conditional variance
Baselines: Cer-Eval EValue Celeus (w/o Surr) Oracle
Builds on: e-processes Cer-Eval LURE estimator prediction-powered inference hedged capital process
Closest papers by meaning
- Position: Don't Use the CLT in LLM Evals With Fewer Than a Few Hundred Datapoints (ICML 2025 · similarity 0.85)
- Towards Reliable LLM Evaluation: Correcting the Winner’s Curse in Adaptive Benchmarking (NeurIPS 2026 · similarity 0.85)
- Rethinking LLM Evaluation: Can We Evaluate LLMs with 200× Less Data? (ICLR 2026 · similarity 0.85)
- Efficient multi-prompt evaluation of LLMs (NeurIPS 2024 · similarity 0.85)
- Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking (NeurIPS 2026 · similarity 0.84)
- Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.