Research map / Label-efficient and robust learning / Recognition and long-tail
Active Bayesian Assessment of Black-Box Classifiers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use informative model-score priors instead of uninformative priors for classwise accuracy estimation | Bayesian assessment · CIFAR-100, ImageNet, SVHN, 20 News, DBpedia | classwise accuracy RMSE: 30.7 -> 15.0; 29.4 -> 13.2; 13.7 -> 5.1; 23.9 -> 12.3; 14.9 -> 2.0 | Helped |
| Add active Thompson sampling to informative-prior accuracy assessment | IPrior classwise accuracy estimation · CIFAR-100, ImageNet, SVHN, 20 News, DBpedia | classwise accuracy RMSE: 15.0 -> 15.3; 13.2 -> 13.2; 5.1 -> 3.4; 12.3 -> 11.7; 2.0 -> 1.5 | Mixed |
| Use model-score informative priors with active sampling for ECE estimation | ECE estimation · CIFAR-100, ImageNet, SVHN, 20 News, DBpedia, N=20 | ECE estimation error (%): 76.7 -> 28.7; 198.7 -> 36.4; 383.6 -> 49.7; 54.0 -> 46.1; 900.3 -> 93.1 | Helped |
| Actively sample groups using Thompson sampling to identify the least accurate class | Classwise accuracy identification · CIFAR-100, ImageNet, SVHN, 20 News | labeled samples needed (%): 81.1 -> 24.9; 96.9 -> 9.3; 90.5 -> 82.8; 53.9 -> 16.9 | Helped |
| Use active Thompson sampling rather than non-active IPrior for least-accurate-class identification | IPrior · DBpedia, least accurate top-1 class | labeled samples needed (%): 7.6 -> 11.6 | Hurt |
| Use informative priors and active allocation for groupwise accuracy comparison | Accuracy comparison · CIFAR-100 superclass pairs | labels required: 203.5 -> 121.9 | Helped |
| Use Thompson sampling instead of epsilon-greedy or Bayesian UCB for identification | Least-accurate and top-m least-accurate class identification · Five datasets | MRR | Helped |
| Replace TS with top-two Thompson sampling (ablation) | Least-accurate-class identification · Five datasets | MRR | No clear effect |
| Use an informative prior rather than an uninformative prior during active identification (ablation) | Active Thompson sampling · ImageNet and SVHN classwise accuracy identification | MRR | Mixed |
| Increase prior strength (ablation) | Bayesian assessment · Reliability diagrams and extreme-class identification | labeled data needed | Hurt |
Closest papers by meaning
- Estimating Accuracy from Unlabeled Data: A Bayesian Approach (ICML 2016 · similarity 0.87)
- Active Statistical Inference (ICML 2024 · similarity 0.86)
- Bayesian Inference for Correlated Human Experts and Classifiers (ICML 2025 · similarity 0.85)
- Uncertainty Herding: One Active Learning Method for All Label Budgets (ICLR 2025 · similarity 0.85)
- SimPro: A Simple Probabilistic Framework Towards Realistic Long-Tailed Semi-Supervised Learning (ICML 2024 · similarity 0.85)
- Bayesian Exploration of Pre-trained Models for Low-shot Image Classification (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.