Research map / Label-efficient and robust learning / OOD and anomaly detection
Open Set Label Shift with Test Time Out-of-Distribution Reference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a training-free method for open set label shift estimation and correction that uses a source ID classifier plus an off-the-shelf ID/OOD classifier and a test-time OOD reference set, combining three estimation stages with an EM algorithm.
Problem
Open set label shift: the label distribution shifts from source to target and the target contains an extra OOD class; the goal is to estimate the target label distribution and correct a source classifier without retraining.
Approach
Reformulates open set label shift as a closed set label shift problem with K+1 classes by using an ID classifier and an ID/OOD classifier, estimates source ID ratio from an OOD reference set, runs an EM MLE/MAP estimator for the target label distribution, and applies a correction for imperfect OOD classifiers, with concentration-based error bounds.
Key results
- On CIFAR100 and ImageNet-200 under LT and Dirichlet shifts, the OSLS-MAP model outperforms closed set label shift baselines (MLLS, BBSE, RLLS, MAPLS) and the uniform baseline in most settings; Table 3 reports it ahead of all baselines in 100% (36/36) of CIFAR10 LT settings and 83.3% (10/12) of Diric
- The target ID ratio correction brings predicted rho_t closer to ground truth across OOD classifiers (Figure 4).
- Calibration of the ResNet18 ID classifiers (ECE about 0.013 to 0.063) supports the calibration assumption.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OSLS estimation with MLS as the OOD classifier instead of the closed-set baseline | OSLS-MAP · CIFAR100 LT10 Forward, Near OOD, r=1.0 | estimation error (w −ˆw)2/K: 0.426±0.000 -> 0.323±0.020 | Helped |
| Use OSLS estimation with MLS as the OOD classifier instead of the closed-set baseline | OSLS-MAP · CIFAR100 LT10 Forward, Far OOD, r=1.0 | estimation error (w −ˆw)2/K: 0.426±0.000 -> 1.289±0.337 | Hurt |
| Use OSLS estimation with MLS as the OOD classifier instead of the closed-set baseline | OSLS-MAP · ImageNet-200 LT10 Forward, Near OOD, r=1.0 | estimation error (w −ˆw)2/K: 0.436±0.000 -> 0.194±0.011 | Helped |
| Use OSLS estimation with MLS as the OOD classifier instead of the closed-set baseline | OSLS-MAP · ImageNet-200 LT10 Forward, Far OOD, r=1.0 | estimation error (w −ˆw)2/K: 0.436±0.000 -> 0.118±0.024 | Helped |
| Add the target ID/OOD ratio correction model (ablation) | OSLS-MLE/MAP · CIFAR10/100 with Dirichlet shift | target ID data ratio estimation | Helped |
Surprising findings
- RLLS has nearly constant estimation error across OOD/ID ratios, and closed set methods improve when the target has fewer OOD samples, which the authors note.
Limitations
- Relies on an OOD reference set; pseudo-OOD from Gaussian-noise mixing is an approximation to true OOD.
- Assumption 3.3B (calibrated ID/OOD classifier) may fail; correction needs condition (11) to hold.
- Evaluated only on image classification benchmarks with ResNet18 classifiers.
Experimental setup
Datasets: CIFAR10 CIFAR100 ImageNet-200 SVHN Places OpenImage-O NINCO TinyImageNet iNaturalist SSB MNIST Texture
Benchmarks: CIFAR10 CIFAR100 ImageNet-200
Metrics: label shift estimation error (w - ŵ)^2/K ID/OOD ratio estimation error expected calibration error
Baselines: MLLS BBSE RLLS MAPLS Baseline (uniform target label distribution, r=1)
Builds on: MLLS MAPLS BBSE OpenOOD
Closest papers by meaning
- When and How Does In-Distribution Label Help Out-of-Distribution Detection? (ICML 2024 · similarity 0.85)
- A Noisy Elephant in the Room: Is Your Out-of-Distribution Detector Robust to Label Noise? (CVPR 2024 · similarity 0.84)
- ImageNet-OOD: Deciphering Modern Out-of-Distribution Detection Algorithms (ICLR 2024 · similarity 0.83)
- Can We Ignore Labels in Out of Distribution Detection? (ICLR 2025 · similarity 0.83)
- Rethinking the Evaluation of Out-of-Distribution Detection: A Sorites Paradox (NeurIPS 2024 · similarity 0.83)
- EDGE: Unknown-aware Multi-label Learning by Energy Distribution Gap Expansion (AAAI 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.