Research map / Label-efficient and robust learning / Recognition and long-tail
Divide-and-Aggregate Learning for Evaluating Performance on Unlabeled Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Divide-and-Aggregate Learning (DAL), which bins model confidence or feature distributions into buckets, aggregates symmetric KL-style shifts, and regresses accuracy for label-free model evaluation.
Problem
Predicting the accuracy of a trained model on unlabeled test data (automatic/label-free model evaluation), where global-only or single-signal methods lose local sample information.
Approach
Divide the sorted output distribution into k equal-frequency or equal-width buckets, convert each to proportions, compute a symmetrized KL divergence (Jeffreys-style) between test and source distributions as the feature, and regress accuracy with a LinearRegression model; works on both feature and confidence outputs.
Key results
- On CIFAR-10 with ResNet-56, DAL-f reaches overall RMSE 3.570 vs. 4.985 for FID (best baseline).
- On MNIST with AlexNet, DAL-c reaches RMSE 7.060, beating KCFCA (8.851) and FID (10.588).
- On TinyImageNet with ResNet-18, DAL-f reaches RMSE 1.344, slightly better than KCFCA (1.352); on ImageNet, DAL-f reaches 2.022 vs. 2.663 for KCFCA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DAL-f instead of ConfScore | ResNet-56 evaluation · CIFAR-10.1 | RMSE: 2.190 -> 1.948 | Helped |
| Use DAL-f instead of FID | ResNet-56 evaluation · CIFAR-10.1-C | RMSE: 5.145 -> 2.854 | Helped |
| Use DAL-c instead of KCFCA | ResNet-56 evaluation · CIFAR-10-F | RMSE: 3.905 -> 2.391 | Helped |
| Use DAL-f instead of FID | ResNet-56 evaluation · Overall | RMSE: 4.985 -> 3.570 | Helped |
| Use DAL-f instead of ConfScore | RepVGG-A0 evaluation · CIFAR-10.1 | RMSE: 5.470 -> 0.015 | Helped |
| Use DAL-f instead of FID | RepVGG-A0 evaluation · CIFAR-10.1-C | RMSE: 6.318 -> 4.395 | Helped |
| Use DAL-f instead of FID | RepVGG-A0 evaluation · Overall | RMSE: 5.966 -> 4.570 | Helped |
| Use DAL-c instead of ConfScore | AlexNet evaluation · MNIST | RMSE: 8.178 -> 7.060 | Helped |
| Use DAL-f instead of KCFCA | ResNet-18 evaluation · TinyImageNet | RMSE: 1.352 -> 1.344 | Helped |
| Use DAL-f instead of KCFCA | ResNet-18 evaluation · ImageNet | RMSE: 2.663 -> 2.022 | Helped |
| Use equal-frequency rather than equal-width binning (ablation) | DAL-c, ResNet-56 evaluation · CIFAR-10 | RMSE: 6.629 -> 5.306 | Helped |
| Use equal-frequency rather than equal-width binning (ablation) | DAL-f, ResNet-56 evaluation · CIFAR-10 | RMSE: 5.138 -> 4.157 | Helped |
Surprising findings
- Equal-width binning outperforms equal-frequency binning for DAL-f on MNIST, contrary to the general trend.
Limitations
- Performance is not uniformly best; DAL-c underperforms on some settings (e.g., CIFAR-10-F, MNIST DAL-f vs. FID-style baselines).
- Relies on constructing meta-datasets via transformations of source data, so generalization to real distribution shifts depends on that proxy.
- Bucket count and binning choice are hyperparameters with inconsistent effects across settings.
Experimental setup
Datasets: CIFAR-10 CIFAR-10.1 CIFAR-10.1-C CIFAR-10-F MNIST TinyImageNet ImageNet
Benchmarks: CIFAR-10.1 CIFAR-10.1-C CIFAR-10-F MNIST TinyImageNet ImageNet
Metrics: RMSE
Baselines: ConfScore Entropy Rotation FID ATC KCFCA
Builds on: AutoEval (Deng and Zheng 2021) KCFCA (Miao et al. 2023) ATC (Garg et al. 2022) Outlier Exposure
Closest papers by meaning
- MixEval-X: Any-to-any Evaluations from Real-world Data Mixture (ICLR 2025 · similarity 0.84)
- SureMap: Simultaneous mean estimation for single-task and multi-task disaggregated evaluation (NeurIPS 2024 · similarity 0.84)
- Limits to scalable evaluation at the frontier: LLM as judge won’t beat twice the data (ICLR 2025 · similarity 0.84)
- Energy-based Automated Model Evaluation (ICLR 2024 · similarity 0.84)
- Evaluating multiple models using labeled and unlabeled data (NeurIPS 2025 · similarity 0.83)
- A hierarchical decomposition for explaining ML performance discrepancies (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.