Research map / Label-efficient and robust learning / Test-time adaptation
Revealing Distribution Discrepancy by Sampling Transfer in Unlabeled Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Importance Divergence (I-Div), which estimates train-test distribution discrepancy for a fixed trained hypothesis using only labeled training samples and unlabeled test inputs, via hypothesis-oriented density ratio and adaptive likelihood ratio.
Problem
Quantifying whether a hypothesis selected on labeled training data remains applicable to unlabeled test data, i.e., measuring train-test distribution discrepancy without test labels.
Approach
Estimates the expected-risk difference between training and test distributions via importance sampling, transferring test sampling to the training distribution using a hypothesis-oriented density ratio (learned by minimizing KL divergences with a neural adapter on the hypothesis backbone) and an adaptive likelihood ratio derived from a generalization-error bound.
Key results
- On CIFAR10 class-split, I-Div reaches 100.0 AUROC and AUPR on all 10 target classes, matching MSP and R-Div and beating NNBD and MMD-D on some classes.
- On CIFAR10-trained ResNet18, I-Div gives AUROC 37.2 on STL10 and 43.4 on CIFAR10.1 (low, indicating transferable hypothesis), while MSP, NNBD, MMD-D and R-Div give near 100 AUROC on these sets.
- Under adversarial perturbations, I-Div keeps a consistently low AUROC even as standard network accuracy falls, which the authors interpret as matching human perception.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use I-Div instead of HDR alone for assessing hypothesis applicability on noisy test data (ablation) | CIFAR10 corruption experiments · CIFAR10 with Gaussian, Salt & Pepper, Uniform, and Speckle noise | AUROC | Helped |
| Use I-Div instead of HDR alone for adversarial samples (ablation) | CIFAR10 adversarial-data experiments · CIFAR10 adversarial samples | AUROC | Mixed |
Surprising findings
- Under adversarial perturbations, I-Div keeps a low AUROC even as classifier accuracy drops sharply, so it does not flag adversarial test sets as shifted.
- The density ratio alone (HDR) is not as consistent as the full I-Div; R-Div separates clean from corrupted data even at minimal noise, which the authors say does not reflect hypothesis applicability.
Limitations
- The likelihood ratio cannot be estimated precisely without test labels; the paper uses an approximate adaptive likelihood ratio.
- Relies on the assumption of a fixed hypothesis and bounded loss; evaluations use only image classification benchmarks.
- Evaluation uses synthetic corruptions and adversarial perturbations rather than real-world shifts in most experiments.
Experimental setup
Datasets: CIFAR10 SVHN PACS Office-Home ImageNet CIFAR100 CIFAR10.1 STL10 DTD Flowers102 OxfordIIITPet SEMEION
Benchmarks: CIFAR10 SVHN PACS Office-Home ImageNet OOD test sets CIFAR10.1 STL10
Metrics: AUROC AUPR ACC
Baselines: MSP NNBD MMD-D R-Div H-Div HDR
Builds on: H-Divergence R-Divergence KLIEP Maximum Mean Discrepancy with deep kernels
Closest papers by meaning
- Efficient Discrepancy Testing for Learning with Distribution Shift (NeurIPS 2024 · similarity 0.84)
- The Polynomial Stein Discrepancy for Assessing Moment Convergence (ICML 2025 · similarity 0.83)
- Boosting Test Performance with Importance Sampling--a Subpopulation Perspective (AAAI 2025 · similarity 0.83)
- How Does Unlabeled Data Provably Help Out-of-Distribution Detection? (ICLR 2024 · similarity 0.83)
- Stability Evaluation through Distributional Perturbation Analysis (ICML 2024 · similarity 0.83)
- Optimal Algorithms for Augmented Testing of Discrete Distributions (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.