Research map / Learning theory and algorithms / Prediction and decision losses

Efficient Lifelong Model Evaluation in an Era of Rapid Progress

NeurIPS 2024 · Learning theory and algorithms › Prediction and decision losses · approach: Active learning and subset selection

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Sort & Search, a dynamic-programming method that ranks test samples by difficulty using cached per-model correctness and sub-samples a few samples to efficiently estimate sample-level accuracy of new models, evaluated on new Lifelong-CIFAR10 and Lifelong-ImageNet benchmarks.

Problem

Evaluating a growing number of models on an ever-expanding pool of test samples is prohibitively expensive (O(|D||M|) cost), so efficient instance-level performance prediction is needed for lifelong benchmarks.

Approach

Sort test samples by difficulty using a cache of prior model correctness (sort-by-sum, with a DP-based threshold search), then evaluate new models only on a small uniformly sampled subset and extrapolate a single threshold-based prediction vector, achieving O(|D| log|D|) cost and minimal memory.

Key results

• Sort & Search reduces evaluation compute from 180 GPU days to 5 GPU hours (~1000x) on a single A100 with MAE around 0.15-0.17 using ~100 samples in the sample-insertion setting.
• MAE decays exponentially with sampling budget n' (fitted decay coefficients b of 0.04 and 0.02).
• Sort by sum reaches MAE < 0.15 on sample difficulty estimation with only 64 of 31,250 models evaluated (~10^4x savings).

Measured effects reported in the paper

Surprising findings

• Overall-accuracy prediction metric can be optimal (zero error) while sample-level predictions are 40-50% wrong, showing aggregate accuracy error is an unreliable metric.
• All model pairs on Lifelong-ImageNet show positive prediction correlations with no anti-correlated cliques, supporting a single global ranking.

Limitations

• Relies on a single global ranking of samples, which is noisy and creates an irreducible aleatoric error.
• Evaluated only on image classification with trained classifiers, not foundation models.
• Metric-based analysis relies on MAE which can be misleading; ranking-level validation shows only moderate Spearman correlation (~0.5).

Experimental setup

Datasets: CIFAR10 CIFAR10.1 CIFAR10.2 CINIC10 CIFAR10-W CIFAR10-C ImageNet ImageNet-A ObjectNet ImageNet-V2 ImageNet-R ImageNet-Sketch

Benchmarks: Lifelong-CIFAR10 Lifelong-ImageNet

Metrics: mean absolute error compute savings Spearman correlation memory cost

Baselines: CopyNearest&Expand Anchor Point Sampling IRT-Clustering Sorting by Confidence Sum Random sampling

Builds on: computerized adaptive testing Model Spider NATS-Bench

Closest papers by meaning

• How Benchmark Prediction from Fewer Data Misses the Mark (NeurIPS 2025 · similarity 0.86)
• Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors (ICLR 2024 · similarity 0.85)
• LL-Bench: Rethinking Low-Level Vision Evaluation in the Era of Large-Scale Generative Models (NeurIPS 2026 · similarity 0.85)
• Power Lines: Scaling laws for weight decay and batch size in LLM pre-training (NeurIPS 2025 · similarity 0.85)
• Benchmarking Optimizers for Large Language Model Pretraining (NeurIPS 2026 · similarity 0.84)
• Kinetics: Rethinking Test-Time Scaling Law (NeurIPS 2025 · similarity 0.84)