Research map / Label-efficient and robust learning / Recognition and long-tail

Positive-unlabeled AUC Maximization under Covariate Shift

ICML 2025 · Label-efficient and robust learning › Recognition and long-tail · approach: Group-robust classification

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a dynamic importance-weighting method that maximizes test AUC under covariate shift using only positive-unlabeled training data and unlabeled test data, without needing negative labels or class priors.

Problem

Maximizing AUC on a test distribution under covariate shift when only positive and unlabeled training data (no labeled negatives) and unlabeled test data are available, with class-priors unknown.

Approach

Derives two importance-weighted estimators of the test smoothed AUC risk, one from importance-weighted PU training pairs and one from weighted positive training data paired with unlabeled test data, combines them with weight beta, and learns the relative importance weights dynamically with a shared feature extractor.

Key results

• On the four class-imbalanced datasets with positive class-prior 0.01-0.1, the proposed method was best or statistically comparable in 12 of 12 settings' rows, e.g., CIFAR10 at pi=0.01 reached 0.896 test AUC versus 0.874 for trAUC and 0.456 for trPU.
• Compared with the two-step importance weighting variant, the dynamic approach improved average test AUC on SVHN (0.690 vs 0.608) and CIFAR10 (0.886 vs 0.876).
• Under class-prior shift (pi_tr=0.01), the proposed method achieved 0.938 on MNIST at pi_te=0.01 and was the best or comparable in 12 of 12 MNIST/FashionMNIST/SVHN/CIFAR10 rows with 0.983 on FashionMNIST at pi_te=0.05.

Measured effects reported in the paper

Surprising findings

• Two-step importance weighting variant of the method did not consistently beat the dynamic version; on SVHN it was substantially worse (0.608 vs 0.690).
• Baseline PAUC performed very poorly (e.g., 0.236-0.504 on SVHN), attributed to its reliance on positive-distribution-shift assumptions.

Limitations

• Assumes covariate shift; not guaranteed to work under other distribution shifts.
• Experiments use only image datasets plus two tabular datasets in appendix; the paper notes importance weight estimation is hard for complex models.
• Requires hyperparameters alpha and beta selected via validation with few positive samples.

Experimental setup

Datasets: MNIST FashionMNIST SVHN CIFAR10 epsilon Hreadmission

Benchmarks: MNIST FashionMNIST SVHN CIFAR10

Metrics: test AUC

Baselines: trPU tePU trAUC teAUC trteAUC UDAUC PAUC CPU Ours w/ 2step

Builds on: AUC maximization with sigmoid surrogate non-negative PU learning relative density-ratio estimation (Yamada et al., 2013) dynamic importance weighting (Fang et al., 2020) Sakai & Shimizu 2019 covariate shift PU

Closest papers by meaning

• AUC Maximization under Positive Distribution Shift (NeurIPS 2024 · similarity 0.94)
• Interplay of ROC and Precision-Recall AUCs: Theoretical Limits and Practical Implications in Binary Classification (ICML 2024 · similarity 0.88)
• Covariate Shift Adaptation on Learning from Positive and Unlabeled Data (AAAI 2019 · similarity 0.87)
• A Closer Look at AUROC and AUPRC under Class Imbalance (NeurIPS 2024 · similarity 0.85)
• A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimators (ICML 2025 · similarity 0.84)
• AUC Optimization from Multiple Unlabeled Datasets (AAAI 2024 · similarity 0.84)