Research map / Label-efficient and robust learning / Knowledge distillation
Towards the Fundamental Limits of Knowledge Transfer over Finite Domains
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Characterizes minimax rates of knowledge transfer from a probabilistic teacher to a student over finite input/label spaces under three levels of teacher information (hard labels, partial soft labels, full soft labels), proposing an empirical squared error logit loss to reach the optimal rate.
Problem
How statistically efficient is knowledge transfer from a teacher classifier to a student when only limited privileged information (hard labels, teacher probabilities of sampled labels, or full logits) is accessible, and what is the fundamental limit?
Approach
Decomposes the transfer set into hard labels, partial soft labels (teacher probability of each sampled label), and soft labels (full teacher distribution per sampled input), proving matching minimax rates at each level; shows naive empirical cross-entropy with partial soft labels is asymptotically biased, and proposes an empirical squared error logit loss that attains the n^-1 rate.
Key results
- Hard-label MLE attains minimax rate sqrt(|S||A|/n) with matching lower bound up to log factors.
- With partial soft labels, naive empirical cross-entropy is asymptotically biased (converges to a squared-probability distortion), while the empirical SEL loss attains rate |S||A|/n up to logs, matching the lower bound Omega(|S||A|/n).
- With full soft labels, KL minimization attains O~(|S|/n), matching the lower bound Omega(|S|/n) independent of |A|.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Provide teacher probabilities for sampled labels in addition to hard labels | Knowledge transfer · Finite domains, partial soft labels | minimax expected total variation risk lower bound: \Omega\left(\sqrt{\nicefrac{|\cS||\cA|}{n}}\right) -> \Omega\left(\nicefrac{|\cS||\cA|}{n}\right) | Helped |
| Use empirical cross-entropy instead of empirical squared error logit loss | Knowledge transfer with partial soft labels · Finite domains, partial soft labels | expected total variation risk: \Omega(1) vs. \tilde{O}\left(\nicefrac{|\cS||\cA|}{n}\right) | Hurt |
| Use empirical squared error logit loss instead of empirical cross-entropy | Knowledge transfer with partial soft labels · Finite domains, partial soft labels | expected total variation risk: \Omega(1) -> \tilde{O}\left(\nicefrac{|\cS||\cA|}{n}\right) | Helped |
| Provide complete teacher soft labels for sampled inputs instead of only probabilities of sampled labels | Knowledge transfer · Finite domains, soft labels | minimax expected total variation risk lower bound: \Omega\left(\nicefrac{|\cS||\cA|}{n}\right) -> \Omega\left(\nicefrac{|\cS|}{n}\right) | Helped |
| Use complete teacher soft labels rather than hard labels | Knowledge transfer · Finite domains | expected total variation risk: \tilde{O}\left(\sqrt{\nicefrac{|\cS||\cA|}{n}}\right) -> \tilde{O}\left(\nicefrac{|\cS|}{n}\right) | Helped |
| Use full KL minimization with complete teacher logits rather than empirical squared error logit loss with partial soft labels | Knowledge transfer · Finite domains | expected total variation risk: \tilde{O}\left(\nicefrac{|\cS||\cA|}{n}\right) -> \tilde{O}\left(\nicefrac{|\cS|}{n}\right) | Helped |
| Replace hard-label maximum likelihood with partial-label empirical cross-entropy | Knowledge transfer with partial soft labels · Finite domains, partial soft labels | asymptotic bias | Hurt |
| Use empirical cross-entropy with partial soft labels rather than maximum likelihood | Knowledge transfer with partial soft labels · Classic regime, Instance 0 | estimated expected total variation risk | Hurt |
| Use empirical squared error logit loss instead of maximum likelihood | Knowledge transfer · Non-asymptotic worst-case instances | expected total variation risk rate: n^{-0.5} -> n^{-1} | Helped |
| Use full KL minimization instead of maximum likelihood | Knowledge transfer · Non-asymptotic worst-case instances | expected total variation risk rate: n^{-0.5} -> n^{-1} | Helped |
| Use full KL minimization rather than empirical squared error logit loss | Knowledge transfer · Classic regime, large |A| | estimated expected total variation risk | Helped |
| Use maximum likelihood when the teacher policy is close to deterministic | Knowledge transfer with hard labels · Instances 2 and 3 | expected total variation risk rate: n^{-0.5} -> n^{-1} | Helped |
Surprising findings
- Naive cross-entropy adaptation with partial soft labels is asymptotically biased and does not converge to the teacher even with privileged information.
- Hard-label MLE can achieve n^-1 rates in benign cases where teacher distributions are near vertices of the simplex.
Limitations
- Analysis is restricted to tabular (finite S and A) settings without function approximation.
- Upper bounds do not adapt to student initialization for unseen inputs or labels.
- Empirical SEL analysis uses normalized logits; practical unnormalized-logit versions are not analyzed.
Experimental setup
Metrics: expected total variation conditioned on input distribution log-log regression slope of risk vs n
Baselines: maximum likelihood estimation empirical cross-entropy loss KL divergence minimization
Builds on: knowledge distillation learning using privileged information Assouad's method Paninski's construction
Closest papers by meaning
- What Makes a Strong Model? A Unified Spectral Analysis of Knowledge Transfer over High-dimensional Linear Regression (ICML 2026 · similarity 0.87)
- Wasserstein Transfer Learning (NeurIPS 2025 · similarity 0.86)
- ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via $\alpha$-$\beta$-Divergence (ICML 2025 · similarity 0.86)
- Universal Rates for Active Learning (NeurIPS 2024 · similarity 0.85)
- PLD: A Choice-Theoretic List-Wise Knowledge Distillation (NeurIPS 2025 · similarity 0.84)
- Multiclass Transductive Online Learning (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.