Research map / Large language models / Language, safety and interpretability
Scaling Up Active Testing to Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows active testing can scale to LLMs by using a fixed surrogate built with a single in-context learning step, a smaller surrogate than the target, and even no target-model predictions, and adds a bootstrap estimator of risk-estimation error.
Problem
Active testing for evaluating LLMs is computationally expensive because it requires repeated surrogate training, surrogate predictions, and target-model predictions over the pool set.
Approach
Replace gradient-based surrogate training in LURE-style sampling-based active testing with a fixed surrogate constructed by in-context learning on a small random initial test set; use a smaller surrogate than the target; and approximate target predictions with the surrogate's predictive entropy. Also introduces a bootstrap estimator of the MSE of the LURE risk estimate.
Key results
- Active testing (LURE-7B) lowers risk-estimation error by 32% on average (median) relative to uniform-random testing across datasets, targets and budgets.
- LURE-70B fails on 70B-few for SST-2 because about 75% of high-NLL (>5) examples are mislabelled; filtering resolves the failure.
- Bootstrap 95% confidence intervals for MSE estimation achieve around 88% coverage of the true estimation error (and about 94% coverage for K>=100 in typical runs).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use active testing (LURE) instead of uniform-random acquisition | LLM risk estimation with fixed in-context surrogates · Four core text-classification datasets and tested model configurations | risk-estimation error: 32% on average | Helped |
| Use a fixed in-context surrogate instead of a dynamically updated surrogate (ablation) | LURE · Selected datasets and target models | risk-estimation error | No clear effect |
| Use a smaller surrogate model to guide acquisition | LURE evaluating larger target models · Four core datasets | risk-estimation error | Mixed |
| Replace target-model predictions in acquisition with surrogate predictive entropy (ablation) | LURE-7B evaluating Llama 2 70B-few · Four core datasets | risk-estimation error | Helped |
| Use label-aware negative-log-likelihood acquisition for dataset curation | Active testing with already-labelled data · SST-2, Subj, HS, and FPB | risk-estimation error | Mixed |
| Use LURE instead of uniform-random testing | Active testing on the harder MMLU dataset · MMLU | risk-estimation error | Mixed |
| Filter pool examples with target-model NLL greater than 5 (ablation) | LURE-70B evaluating Llama 2 70B-few · SST-2 | risk-estimation error | Helped |
| Filter pool examples with target-model NLL greater than 3 (ablation) | LURE-70B evaluating Llama 2 70B-few · SST-2 | risk-estimation error | Helped |
| Use LURE-70B rather than LURE-7B as surrogate (ablation) | Llama 2 70B-few evaluation · SST-2 | risk-estimation error: 4.0596 -> 14.7939 | Hurt |
| Use dynamic rather than fixed LURE-7B surrogate (ablation) | Llama 2 7B-few evaluation · SST-2, FPB, HS, and Subj at step 40 | MSE (×10^4): 8.74 -> 8.62; 16.26 -> 11.53; 63.67 -> 43.83; 9.12 -> 30.94 | Mixed |
| Use ASE instead of sampling-based LURE with a fixed surrogate (ablation) | Active testing · HS with Llama 2 70B-few; other tested settings | risk-estimation error | Mixed |
| Use Gemma3-1B as the surrogate (ablation) | LURE · MMLU | risk-estimation error | Hurt |
Surprising findings
- Using the surrogate's predictive entropy in place of target-model predictions for acquisition remains surprisingly effective.
- Llama 2 7B surrogates can outperform 70B surrogates for some target models because the 70B surrogate's high-loss examples are often mislabelled.
Limitations
- Fails when the surrogate is weak (e.g., Gemma3-1B on MMLU) or uninformative (LURE-7B on 70B-few Subj).
- Bootstrap MSE estimator lacks theoretical convergence guarantees since losses are not iid; estimation fails in some runs.
- Sensitive to label noise in the pool set.
Experimental setup
Datasets: SST-2 Subj FPB HS MMLU
Benchmarks: SST-2 Subj FPB HS MMLU
Metrics: mean squared error of risk estimate relative error vs uniform-random coverage probability of bootstrap confidence intervals
Baselines: uniform-random testing ASE (active surrogate estimator) dynamic-surrogate active testing
Builds on: LURE active surrogate estimator active testing bootstrap
Closest papers by meaning
- Active Evaluation Acquisition for Efficient LLM Benchmarking (ICML 2025 · similarity 0.90)
- Efficient Evaluation of LLM Performance with Statistical Guarantees (NeurIPS 2026 · similarity 0.90)
- Efficiently Learning at Test-Time: Active Fine-Tuning of LLMs (ICLR 2025 · similarity 0.88)
- Prediction-Powered Active Testing (NeurIPS 2026 · similarity 0.88)
- Compute Aligned Training: Optimizing for Test Time Inference (NeurIPS 2026 · similarity 0.88)
- The Surprising Effectiveness of Test-Time Training for Few-Shot Learning (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.