Research map / Label-efficient and robust learning / Recognition and long-tail
Distilling Reliable Knowledge for Instance-Dependent Partial Label Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DIRK, a self-distillation framework for instance-dependent partial label learning that rectifies teacher confidences so candidate labels outrank non-candidates, plus a representation refinement module (DIRK-REF).
Problem
Instance-dependent partial label learning (IDPLL), where candidate label sets depend on instance features, and distillation from a teacher can violate the partial label knowledge that candidate labels should have higher confidence than non-candidates.
Approach
Decompose teacher outputs into candidate and non-candidate parts and rescale them with an instance-adaptive factor (eta = 1/p_c) so that every candidate label's confidence exceeds every non-candidate's, then distill the rectified confidence; add an instance-wise weighted contrastive loss that transfers label-space similarity into feature space.
Key results
- DIRK achieves the best accuracy in Table 1 across benchmark datasets, e.g. 91.48% on Fashion-MNIST and 44.03% on CUB-200 (per the extracted table), and outperforms all compared deep PLL methods.
- On real-world datasets DIRK significantly outperforms baselines on Lost, BirdSong, and Yahoo! News (Table 2).
- DIRK-REF with lambda>0 consistently outperforms DIRK across all benchmark datasets (Table 3), and DIRK has the lowest ECE among compared methods on CIFAR-10 (1.42 teacher, 1.98 student).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DIRK instead of the strongest listed baseline | partial-label classifier · CUB-200 | accuracy | Helped |
| use DIRK instead of the strongest listed baseline | partial-label classifier · Flower | accuracy | Helped |
| use DIRK instead of the strongest listed baseline | partial-label classifier · Oxford-IIIT Pet | accuracy | Helped |
| use DIRK instead of the strongest listed baseline | partial-label classifier · CIFAR-100 | accuracy | Helped |
| use DIRK instead of the strongest listed baseline | partial-label classifier · MSRCv2 | accuracy | Hurt |
| use DIRK instead of the strongest listed baseline | partial-label classifier · Soccer Player | accuracy | Hurt |
| add knowledge-based representation refinement with λ = 0.1 (ablation) | DIRK · Fashion-MNIST | accuracy: 91.48 ± 0.21% -> 92.01 ± 0.24% | Helped |
| add knowledge-based representation refinement with λ = 0.3 (ablation) | DIRK · CIFAR-10 | accuracy | Helped |
| add knowledge-based representation refinement with λ = 0.7 (ablation) | DIRK · CUB-200 | accuracy | Helped |
| vary the momentum hyperparameter m across {0.1, 0.3, 0.5, 0.7, 0.9, 0.99} (ablation) | DIRK · benchmark datasets | accuracy | No clear effect |
| set the representation-refinement weight to different λ values (ablation) | DIRK · benchmark datasets | accuracy | Helped |
Surprising findings
- Direct application of vanilla self-distillation destroys partial label knowledge: the proportion of instances whose teacher confidences satisfy the candidate-over-non-candidate condition remains low, even at the best alpha.
Limitations
- Gains on real-world PLL datasets are insignificant on MSRCv2 and Soccer Player, attributed to semantic sparsity and limited training data.
- Real-world experiments use a linear model backbone and set gamma to 1, a simplification.
Experimental setup
Datasets: Fashion-MNIST Kuzushiji-MNIST CIFAR-10 CIFAR-100 CUB-200 Flower Oxford-IIIT Pet Lost Soccer Player Yahoo! News MSRCv2 BirdSong
Benchmarks: Fashion-MNIST Kuzushiji-MNIST CIFAR-10 CIFAR-100 CUB-200 Flower Oxford-IIIT Pet Lost BirdSong MSRCv2 Soccer Player Yahoo!News
Metrics: classification accuracy expected calibration error
Baselines: POP IDGP ABLE VALEN PICO CAVL CR-DPLL LWS RC CC PRODEN
Builds on: Born-Again Networks PICO ABLE VALEN PRODEN
Closest papers by meaning
- Reduction-based Pseudo-label Generation for Instance-dependent Partial Label Learning (NeurIPS 2025 · similarity 0.91)
- Partial Label Causal Representation Learning for Instance-Dependent Supervision and Domain Generalization (AAAI 2025 · similarity 0.90)
- What Makes Partial-Label Learning Algorithms Effective? (NeurIPS 2024 · similarity 0.90)
- Partial Label Learning with a Partner (AAAI 2024 · similarity 0.89)
- Navigating Real-World Partial Label Learning: Unveiling Fine-Grained Images with Attributes (AAAI 2024 · similarity 0.89)
- Controller-Guided Partial Label Consistency Regularization with Unlabeled Data (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.