Research map / Label-efficient and robust learning / Recognition and long-tail
Energy-based Automated Model Evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Meta-Distribution Energy (MDE), a training-free AutoEval measure that normalizes per-sample energies over an unlabeled test set and linearly regresses accuracy from it, outperforming prior AutoEval baselines on vision and NLI benchmarks.
Problem
Estimating a classifier's accuracy on unlabeled out-of-distribution test sets (automated model evaluation) without the overconfidence, storage and compute overhead of prior methods.
Approach
Treat the classifier as an energy-based model with per-sample free energy from logits, then re-normalize the energies over the unlabeled dataset into a meta-distribution statistic (MDE) and fit a linear regressor from MDE to accuracy on synthetic shifted sets.
Key results
- MDE achieves average R^2 of 0.973 on CIFAR-10 and 0.977 on TinyImageNet synthetic shifts, beating ConfScore, Entropy, Frechet and ATC.
- On natural shifts, MDE reduces average MAE to about 40% below NuclearNorm (from 5.25 to 3.14 averaged across seven datasets, per the paper).
- MDE reaches Spearman rho above 0.981 for vision and 0.846 for text (MNLI) on synthetic shifts.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MDE instead of NuclearNorm for accuracy prediction | AutoEval · CIFAR-10, CIFAR-100, TinyImageNet, MNLI, and ImageNet synthetic shifts | R²: 0.935 -> 0.973 (CIFAR-10); 0.950 -> 0.966 (CIFAR-100); 0.969 -> 0.977 (TinyImageNet); 0.496 -> 0.680 (MNLI); 0.955 -> 0. | Mixed |
| Use MDE instead of AvgEnergy | Energy-based AutoEval · CIFAR-10, CIFAR-100, TinyImageNet, MNLI, and ImageNet synthetic shifts | R²: 0.957 -> 0.973 (CIFAR-10); 0.963 -> 0.966 (CIFAR-100); 0.964 -> 0.977 (TinyImageNet); 0.595 -> 0.680 (MNLI); 0.953 -> 0. | Helped |
| Use MDE instead of NuclearNorm for natural-shift accuracy prediction | AutoEval · CIFAR-10, TinyImageNet, MNLI, and ImageNet natural shifts | MAE: 3.42 -> 1.78 (CIFAR-10); 3.84 -> 2.15 (TinyImageNet); 8.44 -> 5.76 (MNLI); 8.63 -> 5.48 (ImageNet) | Helped |
| Use MDE instead of NuclearNorm | WILDS accuracy prediction · Camelyon17, RxRx1, and FMoW natural shifts | MAE: 4.26 -> 2.93 (Camelyon17); 3.67 -> 1.62 (RxRx1); 4.49 -> 2.24 (FMoW) | Helped |
| Increase the MDE temperature above T=1 (ablation) | MDE · CIFAR-10 with VGG-11 | correlation and MAE performance | Hurt |
| Vary the training random seed (ablation) | MDE AutoEval · CIFAR-10 with VGG-11 | performance | No clear effect |
| Apply new transformations to naturally shifted test sets (ablation) | MDE · CIFAR-10.1-A/B, CIFAR-10.2-A/B, CINIC-10-A/B, and STL-10.1-A/B | MAE: MAE<5.92 | Mixed |
| Create long-tail test sets with imbalance ratio r≥0.4 (ablation) | MDE · CIFAR-10-C, severity level 2 | accuracy prediction performance | Helped |
| Create severely imbalanced test sets with r≤0.2 (ablation) | MDE · CIFAR-10-C, severity level 2 | accuracy prediction performance | Hurt |
| Compute MDE from shallow-layer features instead of classification-head features (ablation) | MDE · CIFAR-10 ID and OOD test sets | discriminability | Hurt |
Surprising findings
- Shallow-layer features yield MDE scores far less discriminative than the classification-head features, so the method relies on the final representation.
Limitations
- Performance degrades under severe class imbalance (imbalance ratio <= 0.2), as label shift affects the measure.
- Requires a set of labeled synthetic shifted sets to fit the linear regressor.
- Theoretical link to accuracy relies on a T->0 limit and assumptions that do not fully hold at T=1.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 TinyImageNet ImageNet-1K MNLI WILDS (Camelyon17, RxRx1, FMoW) CIFAR-10-C CIFAR-100-C ImageNet-C TinyImageNet-C
Benchmarks: CIFAR-10-C CIFAR-100-C ImageNet-C TinyImageNet-C CIFAR-10.1 CIFAR-10.2 CINIC-10 STL-10 ImageNet-V2 ImageNet-Sketch ImageNet-Rendition ImageNet-Adversarial
Metrics: R^2 Pearson r Spearman rho mean absolute error
Baselines: ConfScore Entropy Frechet AgreeScore ATC COT AvgEnergy ProjNorm NuclearNorm
Builds on: energy-based model (JEM) Average Thresholded Confidence Nuclear norm AutoEval ProjNorm
Closest papers by meaning
- AutoEval Done Right: Using Synthetic Data for Model Evaluation (ICML 2025 · similarity 0.85)
- Adaptive Prediction-Powered AutoEval with Reliability and Efficiency Guarantees (NeurIPS 2025 · similarity 0.85)
- Divide-and-Aggregate Learning for Evaluating Performance on Unlabeled Data (AAAI 2024 · similarity 0.84)
- TEA: Test-time Energy Adaptation (CVPR 2024 · similarity 0.84)
- Limits to scalable evaluation at the frontier: LLM as judge won’t beat twice the data (ICLR 2025 · similarity 0.83)
- From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Set (ICML 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.