Research map / Label-efficient and robust learning / Semi-supervised learning
Out-Of-Domain Unlabeled Data Improves Generalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Robust Self-Supervised (RSS) training that combines distributionally robust optimization with self-training on out-of-domain unlabeled data, with non-asymptotic generalization bounds for Gaussian mixtures and small experiments on simulated and histopathology data.
Problem
Semi-supervised classification where unlabeled data may be slightly out-of-domain (small Wasserstein shift); the labeled-vs-unlabeled sample trade-off and generalization bounds for robust and non-robust losses are not well understood.
Approach
Add a DRO-based robust loss on labeled data plus a self-training penalty on unlabeled out-of-domain samples (pseudo-labeled by the current classifier) that discourages decision boundaries crossing crowded regions; the penalty is solvable in polynomial time for convex settings and yields explicit bounds for two-component Gaussian mixtures.
Key results
- On simulated d=200 data, 10 labeled samples with 10,000 unlabeled samples reach 0.82 accuracy versus 0.59 with labeled data alone (same distribution), and 0.81 under distribution shift.
- On NCT-CRC-HE-100K, 48 labeled samples with 2,000 same-distribution unlabeled samples reach 0.82 accuracy versus 0.65 with labeled data alone.
- Theory claims the RSS bound beats ERM when alpha <= O(d/m) and n >= Omega(m^2/d).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add 10,000 unlabeled samples | RSS, simulated data · Same distribution; labeled size 10 | accuracy: 0.59 -> 0.82 | Helped |
| Add 10,000 unlabeled samples | RSS, simulated data · Same distribution; labeled size 20 | accuracy: 0.62 -> 0.82 | Helped |
| Add 10,000 unlabeled samples | RSS, simulated data · Same distribution; labeled size 40 | accuracy: 0.65 -> 0.82 | Helped |
| Add 10,000 out-of-domain unlabeled samples | RSS, simulated data · Different distribution; labeled size 10 | accuracy: 0.59 -> 0.81 | Helped |
| Add 10,000 out-of-domain unlabeled samples | RSS, simulated data · Different distribution; labeled size 20 | accuracy: 0.62 -> 0.80 | Helped |
| Add 10,000 out-of-domain unlabeled samples | RSS, simulated data · Different distribution; labeled size 40 | accuracy: 0.65 -> 0.80 | Helped |
| Add 2,000 unlabeled samples | RSS, histopathology classifier · NCT-CRC-HE-100K, same distribution; labeled size 48 | accuracy: 0.65 -> 0.82 | Helped |
| Add 4,000 unlabeled samples | RSS, histopathology classifier · NCT-CRC-HE-100K, same distribution; labeled size 240 | accuracy: 0.77 -> 0.83 | Helped |
| Add 20,000 unlabeled samples | RSS, histopathology classifier · NCT-CRC-HE-100K, same distribution; labeled size 1040 | accuracy: 0.83 -> 0.91 | Helped |
| Add 2,000 out-of-domain unlabeled samples | RSS, histopathology classifier · NCT-CRC-HE-100K labels and PatchCamelyon unlabeled; labeled size 25 | accuracy: 0.78 -> 0.81 | Helped |
| Add 3,000 out-of-domain unlabeled samples | RSS, histopathology classifier · NCT-CRC-HE-100K labels and PatchCamelyon unlabeled; labeled size 50 | accuracy: 0.82 -> 0.87 | Helped |
| Add 8,000 out-of-domain unlabeled samples | RSS, histopathology classifier · NCT-CRC-HE-100K labels and PatchCamelyon unlabeled; labeled size 300 | accuracy: 0.87 -> 0.90 | Helped |
Surprising findings
- Out-of-domain unlabeled data (PatchCamelyon) gave accuracy comparable to or better than in-domain unlabeled data in some settings, e.g., 25 labeled samples with 400 out-of-domain unlabeled reach 0.79 versus 0.80 in-domain with 700.
Limitations
- Theoretical guarantees are restricted to two-component Gaussian mixtures with linear classifiers and zero-one loss.
- Requires the distribution shift to be small (Wasserstein distance alpha controlled).
- Real-data experiments use frozen ImageNet ResNet50 embeddings and a small MLP, with no comparison to standard SSL baselines.
Experimental setup
Datasets: NCT-CRC-HE-100K PatchCamelyon ImageNet synthetic Gaussian mixture (d=200)
Benchmarks: NCT-CRC-HE-100K PatchCamelyon simulated Gaussian mixture
Metrics: accuracy
Baselines: ERM with labeled data only
Builds on: distributionally robust learning self-training Wasserstein distance-based DRO
Closest papers by meaning
- Unlocking the Potential of Unlabeled Data in Semi-Supervised Domain Generalization (CVPR 2025 · similarity 0.91)
- Semi-Supervised Sparse Gaussian Classification: Provable Benefits of Unlabeled Data (NeurIPS 2024 · similarity 0.88)
- Understanding Domain Generalization: A Noise Robustness Perspective (ICLR 2024 · similarity 0.88)
- Scaling Up Semi-supervised Learning with Unconstrained Unlabelled Data (AAAI 2024 · similarity 0.88)
- Robust Domain Generalization under Divergent Marginal and Conditional Distributions (NeurIPS 2026 · similarity 0.87)
- How Does Unlabeled Data Provably Help Out-of-Distribution Detection? (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.