AI Paper Atlas

Research map / Learning theory and algorithms / Prediction and decision losses

Query-efficient model evaluation using cached responses

Hayden Helm, Ben Johnson, Carey Priebe

ICML 2026 · Learning theory and algorithms › Prediction and decision losses · approach: Adaptive testing and experiments · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
predict benchmark score by regression on DKPS representations of cached responses instead of the sample score on the query subsetSample Score · HELM-Lite LegalBench, 4 queriesMAE: 0.196 -> 0.068Improved
predict with DKPS regression instead of sample score at 1 querySample Score · HELM-Lite MATH, 1 queryMAE: 0.353 -> 0.147Improved
use DKPS regression instead of sample score at 64 queriesSample Score · HELM-Lite MATH, 64 queriesMAE: 0.039 -> 0.070Hurt
add ensemble of sample score and DKPS prediction with weight alpha=m/M (ablation)DKPS · HELM-Lite MATH, 64 queriesMAE: 0.070 -> 0.061Improved
add IRT-selected combination to the ensemble (Ens(DKPS+IRT)) (ablation)Ens(DKPS) · HELM-Lite MATH, 64 queriesMAE: 0.061 -> 0.042Improved
add IRT-selected combination to the ensemble (Ens(DKPS+IRT)) (ablation)Ens(DKPS) · HELM-Lite LegalBench, 4 queriesMAE: 0.068 -> 0.070Hurt
use more reference models to build DKPS (ablation)DKPS regression · HELM-Lite subtasksMAEImproved
switch embedding from all-minilm-l6-v2 to gemini-embedding-001 (ablation)DKPS · MATH counting and probability, 1 queryMAE: 0.15 -> 0.12Improved
larger embedding model (ablation)DKPS · MATH counting and probabilityMAENo clear effect
offline query selection maximizing R2 on reference models instead of random queriesDKPS with random query sets · MATH counting and probability, m=8MAEImproved
offline R2 query selection at larger query budgetsDKPS · MATH counting and probabilityMAENo clear effect
use OLS regression instead of k-NN in DKPS (ablation)DKPS · HELM-Lite tasksMAEImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy