Research map / Learning theory and algorithms / Prediction and decision losses
Query-efficient model evaluation using cached responses
Hayden Helm, Ben Johnson, Carey Priebe
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| predict benchmark score by regression on DKPS representations of cached responses instead of the sample score on the query subset | Sample Score · HELM-Lite LegalBench, 4 queries | MAE: 0.196 -> 0.068 | Improved |
| predict with DKPS regression instead of sample score at 1 query | Sample Score · HELM-Lite MATH, 1 query | MAE: 0.353 -> 0.147 | Improved |
| use DKPS regression instead of sample score at 64 queries | Sample Score · HELM-Lite MATH, 64 queries | MAE: 0.039 -> 0.070 | Hurt |
| add ensemble of sample score and DKPS prediction with weight alpha=m/M (ablation) | DKPS · HELM-Lite MATH, 64 queries | MAE: 0.070 -> 0.061 | Improved |
| add IRT-selected combination to the ensemble (Ens(DKPS+IRT)) (ablation) | Ens(DKPS) · HELM-Lite MATH, 64 queries | MAE: 0.061 -> 0.042 | Improved |
| add IRT-selected combination to the ensemble (Ens(DKPS+IRT)) (ablation) | Ens(DKPS) · HELM-Lite LegalBench, 4 queries | MAE: 0.068 -> 0.070 | Hurt |
| use more reference models to build DKPS (ablation) | DKPS regression · HELM-Lite subtasks | MAE | Improved |
| switch embedding from all-minilm-l6-v2 to gemini-embedding-001 (ablation) | DKPS · MATH counting and probability, 1 query | MAE: 0.15 -> 0.12 | Improved |
| larger embedding model (ablation) | DKPS · MATH counting and probability | MAE | No clear effect |
| offline query selection maximizing R2 on reference models instead of random queries | DKPS with random query sets · MATH counting and probability, m=8 | MAE | Improved |
| offline R2 query selection at larger query budgets | DKPS · MATH counting and probability | MAE | No clear effect |
| use OLS regression instead of k-NN in DKPS (ablation) | DKPS · HELM-Lite tasks | MAE | Improved |
Closest papers by meaning
- How Benchmark Prediction from Fewer Data Misses the Mark (NeurIPS 2025 · similarity 0.86)
- Rethinking LLM Evaluation: Can We Evaluate LLMs with 200× Less Data? (ICLR 2026 · similarity 0.85)
- Randomization Boosts KV Caching, Learning Balances Query Load: A Joint Perspective (ICLR 2026 · similarity 0.84)
- PatchKV: Weight Space Compensation of KV Cache (NeurIPS 2026 · similarity 0.84)
- BEST-Route: Adaptive LLM Routing with Test-Time Optimal Compute (ICML 2025 · similarity 0.84)
- Towards Establishing Guaranteed Error for Learned Database Operations (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
