Research map / Label-efficient and robust learning / Recognition and long-tail
Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use soft label-classifier outputs as the representation with univariate tests instead of the alternatives | Shift detection on MNIST and CIFAR-10 simulated shifts · MNIST and CIFAR-10 simulated shifts | detection accuracy: 0.47 vs 0.38 | Helped |
| Use UAE representations with multivariate tests rather than other multivariate DR methods | Shift detection on MNIST and CIFAR-10 simulated shifts · MNIST and CIFAR-10 simulated shifts | detection accuracy: 0.43 vs 0.38 | Helped |
| Use domain-discriminating classification instead of BBSDs in the low-sample regime | Shift detection on MNIST and CIFAR-10 simulated shifts · MNIST and CIFAR-10 simulated shifts, 100 samples | detection accuracy: 0.47 vs 0.21 | Hurt |
| Use univariate tests with Bonferroni aggregation instead of multivariate kernel tests | Shift detection on MNIST and CIFAR-10 simulated shifts · MNIST and CIFAR-10 simulated shifts | detection accuracy | No clear effect |
| Use no dimensionality reduction with multivariate testing | Shift detection on MNIST and CIFAR-10 simulated shifts · MNIST and CIFAR-10 simulated shifts, 100 samples | detection accuracy: 0.28 vs 0.43 | Hurt |
| Increase the number of target samples | BBSDs shift detection · MNIST and CIFAR-10 simulated shifts | detection accuracy: 0.19 -> 0.79 | Helped |
| Increase shift intensity from small to large | BBSDs shift detection · MNIST and CIFAR-10 simulated shifts, 50 samples | detection accuracy: 0.14 -> 0.78 | Helped |
| Increase the perturbed fraction from 10% to 50% | BBSDs shift detection · MNIST and CIFAR-10 simulated shifts, 50 samples | detection accuracy: 0.24 -> 0.52 | Helped |
| Apply a small Gaussian-noise shift rather than no shift | BBSDs shift detection · MNIST and CIFAR-10, s_gn | detection accuracy: 0.10 | Hurt |
| Use BBSDs rather than UAE for detecting shifts with univariate versus multivariate tests | Shift detection on MNIST and CIFAR-10 · MNIST and CIFAR-10 simulated shifts | detection accuracy: 0.47 vs 0.43 | Mixed |
| Use domain classifier confidence to select likely target examples for qualitative characterization | Shift analysis · MNIST medium image shift and COIL-100 angle partition | shift characterization | Helped |
| Evaluate the classifier on labeled examples most confidently assigned to the target domain | Shift malignancy assessment · MNIST medium image shift and COIL-100 angle partition | malignancy identification | Helped |
Closest papers by meaning
- "What is Different Between These Datasets?" A Framework for Explaining Data Distribution Shifts (NeurIPS 2026 · similarity 0.89)
- A Learning Based Hypothesis Test for Harmful Covariate Shift (ICLR 2023 · similarity 0.88)
- A Fine-Grained Analysis on Distribution Shift (ICLR 2022 · similarity 0.88)
- Sequential Harmful Shift Detection Without Labels (NeurIPS 2024 · similarity 0.86)
- Generalization vs Specialization under Concept Shift (NeurIPS 2025 · similarity 0.85)
- Optimal Aggregation of Prediction Intervals under Unsupervised Domain Shift (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.