Research map / Label-efficient and robust learning / Knowledge distillation
Aligning Logits Generatively for Principled Black-Box Knowledge Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Mapping-Emulation KD (MEKD), a black-box KD method that trains a GAN generator as an inverse mapping of the teacher, then distills a student by aligning teacher and student logits through the frozen generator's image space, without ground-truth labels.
Problem
Black-box knowledge distillation from a cloud-hosted teacher to an edge student, where the teacher's internals are hidden, only soft or hard responses are returned, query budgets are limited, and local data are unlabeled and distributionally different.
Approach
Train a DCGAN generator with an information-maximization loss so it approximates the inverse of the teacher; then freeze it and align teacher and student logits by minimizing the L1 distance between generator outputs of the two logit vectors, optionally combined with a logit-level KL loss.
Key results
- On CIFAR-100 with ResNet56-MobileNet, MEKD (soft) reaches 67.07% and MEKD (hard) 67.36%, versus 57.85% best prior-style baseline among listed rows (KD 54.14 / AM 62.17 in comparable settings), i.e. roughly 5-10 points over other B2KD methods as claimed.
- On ImageNet-1K (100K subset), MEKD (hard) gets 59.89% (RN50-RN34) and 61.21% (RX101-RX50), exceeding DB3KD (58.61%) and AM (56.92%).
- On out-of-domain SVHN transfer from Syn.Digits, MEKD reaches 86.45-89.21% but DB3KD is higher (90.15-91.14%) at much greater data-exchange cost (~120 MB vs ~20.8 GB).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MEKD instead of DB3KD | ResNet50 → ResNet34 · ImageNet-1K | top-1 accuracy (%): 58.61 -> 59.89 | Helped |
| use MEKD instead of DB3KD | ResNeXt101 → ResNeXt50 · ImageNet-1K | top-1 accuracy (%): 59.90 -> 61.21 | Helped |
| add the information maximization loss during deprivatization (ablation) | MEKD · CIFAR-100 | top-1 accuracy (%): 60.87 -> 67.07 | Helped |
| add the KLD loss to the distillation objective (ablation) | MEKD · CIFAR-100 | top-1 accuracy (%): 56.28 -> 67.07 | Helped |
| use L2 instead of L1 for the image-space distance (ablation) | MEKD (soft) · CIFAR-10 | top-1 accuracy (%): 86.85 / 86.63 | No clear effect |
| use L2 instead of L1 for the image-space distance (ablation) | MEKD (soft) · CIFAR-100 | top-1 accuracy (%): 67.07 / 66.95 | No clear effect |
| increase the training data size from 0.1K to 50K (ablation) | MEKD (soft) · CIFAR-10 | top-1 accuracy (%): 49.04 -> 93.48 | Helped |
| use MEKD instead of DB3KD | ResNet56 → MobileNet · SVHN | top-1 accuracy (%): 90.15 -> 86.45 | Hurt |
Surprising findings
- Hard-response MEKD performs comparably to soft-response MEKD, and the paper argues the method does not differentiate soft from hard responses.
- The information-maximization term alone gives little benefit without the KL logit loss, per the query-sample curves.
Limitations
- Quality of the distillation depends on a well-trained generator; GANs suffer from mode collapse on complex datasets such as CIFAR-100 and Tiny ImageNet.
- Parameter size and structural limits prevent the student from fully mimicking the teacher.
- Theoretical results rely on assumptions (e.g., Wasserstein-zero conditions, Kolmogorov-based expressiveness) that are not empirically verified in full.
Experimental setup
Datasets: MNIST CIFAR-10 CIFAR-100 Tiny ImageNet ImageNet-1K Syn.Digits SVHN
Benchmarks: MNIST CIFAR-10 CIFAR-100 Tiny ImageNet ImageNet-1K SVHN
Metrics: top-1 accuracy
Baselines: KD ML AL DKD DAFL KN AM DB3KD
Builds on: Knowledge distillation (Hinton et al.) DAFL DCGAN DB3KD Wasserstein GAN
Closest papers by meaning
- All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.86)
- Enhancing Logits Distillation with Plug&Play Kendall's $\tau$ Ranking Loss (ICML 2025 · similarity 0.85)
- Gap Preserving Distillation by Building Bidirectional Mappings with A Dynamic Teacher (ICLR 2025 · similarity 0.85)
- Knowledge Distillation with Multi-granularity Mixture of Priors for Image Super-Resolution (ICLR 2025 · similarity 0.85)
- Small Scale Data-Free Knowledge Distillation (CVPR 2024 · similarity 0.85)
- EPSD: Early Pruning with Self-Distillation for Efficient Model Compression (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.