Research map / Label-efficient and robust learning / Recognition and long-tail
Rethinking Guidance Information to Utilize Unlabeled Samples: A Label Encoding Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Label-Encoding Risk Minimization (LERM), which estimates label encodings via prediction means of unlabeled samples and aligns them to one-hot encodings, as a plug-in to SSL, UDA, and SHDA methods.
Problem
Utilizing unlabeled samples under label-insufficient scenarios, where entropy minimization emphasizes discriminability but neglects prediction diversity and is biased toward majority categories.
Approach
Treat one-hot label encodings as guidance for unlabeled samples: estimate each category's label encoding by the weighted prediction mean of unlabeled samples and minimize its L1 distance to the ground-truth one-hot encoding, a category-wise risk that generalizes ERM and upper-bounds EntMin.
Key results
- On CIFAR-100 with 4 labels per class, ERM+LERM reaches 75.57% top-1 on CIFAR-10 (4 labels) versus 57.04% for ERM, and 60.19% on CIFAR-100 top-1 versus 47.18%.
- On Office-Home UDA, ERM+LERM averages 69.18% versus 59.32% for ERM, 65.75% for ERM+EntMin, and 68.71% for ERM+BNM.
- SDAT+LERM reaches 72.66% average on Office-Home and 90.37% average on Office-31, the best in each comparison group.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add LERM to DST | DST · CIFAR-100, 1 label/category | Top-1 accuracy: 32.92 -> 43.78 | Helped |
| Add LERM to FlexMatch | FlexMatch · CIFAR-10, 1 label/category | Top-1 accuracy: +12.83% | Helped |
| Add LERM to ERM | ERM · Office-Home UDA | Average accuracy: 59.32 -> 69.18 | Helped |
| Add LERM to ERM | ERM · Office-31 UDA | Average accuracy: 80.29 -> 88.22 | Helped |
| Add LERM to ERM | ERM · VisDA UDA | Average accuracy: 51.30 -> 80.72 | Helped |
| Add LERM to ERM | ERM · NUS-WIDE → ImageNet SHDA | Accuracy: 67.42 -> 79.04 | Helped |
| Add LERM to ERM | ERM · CIFAR-10, class-imbalanced SSL | Average classification accuracy: 79.47% -> 92.97% | Helped |
| Add LERM to ERM | ERM · CIFAR-10 SSL | Prediction entropy: 0.3832 -> 0.0440 | Helped |
| Replace L1 loss with KL loss (ablation) | LERM · CIFAR-10, 1 label/category | Top-1 accuracy: 38.22 -> 35.23 | Hurt |
| Replace L1 loss with L2 loss (ablation) | LERM · CIFAR-10, 1 label/category | Top-1 accuracy: 38.22 -> 37.08 | Hurt |
| Set λ to 10 instead of 50 (ablation) | ERM + LERM · CIFAR-100, 1 label/category | Top-1 accuracy: 30.15 -> 31.84 | Helped |
| Set λ to 10 rather than 0 (ablation) | ERM + LERM · Office-31 UDA | Average accuracy: 80.29 -> 88.22 | Helped |
Surprising findings
- EntMin degrades below ERM in several one-label settings (e.g., CIFAR-100 1-label top-1 15.32% vs ERM 23.58%), while LERM improves over ERM there.
- The KL loss performed worse than L1 and L2 losses for the label-encoding risk, which the authors attribute to KL not constraining non-target probabilities.
Limitations
- Only one hyperparameter lambda, but its best value varies widely across datasets and methods (e.g., 0.01 to 2000 in the tables).
- Theoretical relations to ERM and EntMin rely on extra assumptions about loss convexity and class-balance inequalities.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 DTD ImageNet-1K Office-31 Office-Home VisDA-2017 ImageNet-R Multilingual Reuters Collection NUS-WIDE DeCAF6 features
Benchmarks: CIFAR-10 CIFAR-100 DTD ImageNet-1K Office-31 Office-Home VisDA-2017 ImageNet-R Multilingual Reuters Collection NUS-WIDE to ImageNet
Metrics: top-1 accuracy top-5 accuracy average entropy F1 score
Baselines: ERM EntMin BNM DANN AFN CDAN SDAT FlexMatch DST KPG
Builds on: entropy minimization FlexMatch DST CDAN SDAT KPG BNM
Closest papers by meaning
- Label Noise: Ignorance Is Bliss (NeurIPS 2024 · similarity 0.86)
- Imprecise Label Learning: A Unified Framework for Learning with Various Imprecise Label Configurations (NeurIPS 2024 · similarity 0.85)
- Predicting Label Distribution from Ternary Labels (NeurIPS 2024 · similarity 0.85)
- Learning with Partial-Label and Unlabeled Data: A Uniform Treatment for Supervision Redundancy and Insufficiency (ICML 2024 · similarity 0.84)
- Understanding Domain Generalization: A Noise Robustness Perspective (ICLR 2024 · similarity 0.84)
- Learning with Complementary Labels Revisited: The Selected-Completely-at-Random Setting Is More Practical (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.