Research map / Label-efficient and robust learning / OOD and anomaly detection
Detecting Errors and Estimating Accuracy on Unlabeled Data with Self-training Ensembles
Jiefeng Chen, Frederick Liu, Besim Avci, Xi Wu, Yingyu Liang, Somesh Jha
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace Proxy Risk with self-training ensemble (TRM) for accuracy estimation | Proxy Risk · Digits | absolute estimation error: 0.085±0.142 -> 0.023±0.020 | Improved |
| replace Proxy Risk with TRM for error detection | Proxy Risk · Digits | F1 score: 0.844±0.118 -> 0.881±0.084 | Improved |
| replace Proxy Risk with TRM for error detection | Proxy Risk · Office-31 | F1 score: 0.605±0.177 -> 0.767±0.052 | Improved |
| use TRI ensemble instead of TRM on imbalanced iWildCam | Ours · iWildCam | absolute estimation error: 0.035±0.022 -> 0.015±0.008 | Improved |
| use TRI instead of TRM on Digits | Ours · Digits | absolute estimation error: 0.023±0.020 -> 0.164±0.218 | Hurt |
| use TRI instead of TRM on Amazon Review | Ours · Amazon Review | F1 score: 0.426±0.036 -> 0.384±0.032 | Hurt |
| replace Proxy Risk with TRM on Office-31 accuracy estimation | Proxy Risk · Office-31 | absolute estimation error: 0.033±0.012 -> 0.029±0.021 | No clear effect |
| replace Proxy Risk with TRM on iWildCam accuracy estimation | Proxy Risk · iWildCam | absolute estimation error: 0.119±0.043 -> 0.035±0.022 | Hurt |
| replace average confidence with TRM on CIFAR10-C | Avg Conf · CIFAR10-C | absolute estimation error: 0.353±0.175 -> 0.022±0.009 | Improved |
| use DANN-arch instead of typical DNN as model f | Ours (RM) · Digits | F1 score: 0.881±0.084 -> 0.841±0.112 | Hurt |
| increase ensemble size N (N=1 means no ensemble) (ablation) | Ours (TRM) · Digits | average F1 score | Improved |
| increase self-training iterations T (T=1 means no self-training) (ablation) | Ours (TRM) · Digits | average F1 score | Improved |
Closest papers by meaning
- Estimating Accuracy from Unlabeled Data: A Bayesian Approach (ICML 2016 · similarity 0.84)
- Leveraging unlabeled data to predict out-of-distribution performance (ICLR 2022 · similarity 0.84)
- Estimating Accuracy from Unlabeled Data: A Probabilistic Logic Approach (NeurIPS 2017 · similarity 0.84)
- Confidence Estimation Using Unlabeled Data (ICLR 2023 · similarity 0.84)
- Detecting Misclassification Errors in Neural Networks with a Gaussian Process Model (AAAI 2022 · similarity 0.84)
- A Simple Remedy for Dataset Bias via Self-Influence: A Mislabeled Sample Perspective (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
