AI Paper Atlas

Research map / Label-efficient and robust learning / Knowledge distillation

Mining Data Impressions from Deep Models as Substitute for the Unavailable Training Data

Gaurav Kumar Nayak, Konda Reddy Mopuri, Saksham Jain, Anirban Chakraborty

TPAMI 2021 · Label-efficient and robust learning › Knowledge distillation · approach: Few-shot and metric distillation · 13 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use Dirichlet-modelled Data Impressions instead of data-free distillation from few original samplesZSKD, LeNet-5-Half student · MNISTaccuracy: 86.70 -> 98.77Improved
use Dirichlet-modelled Data Impressions instead of 200 original samplesZSKD, LeNet-5-Half student · Fashion-MNISTaccuracy: 72.50 -> 79.62Improved
replace original training data by Data Impressions for distillationAlexNet-Half student · CIFAR-10 (AlexNet)accuracy: 80.08 -> 69.56Hurt
replace original training data by Data Impressions for distillation to a smaller ResNetResnet-18-half student · CIFAR-10 (Resnet-18 teacher)accuracy: 86.31 -> 81.10Hurt
replace original training data by Data Impressions for VGG-11 studentVGG-11 student · CIFAR-10 (VGG-19 teacher)accuracy: 84.93 -> 74.10Hurt
use Dirichlet-modelled Data Impressions instead of one-hot Class Impressions (ablation)ZSKD · MNIST, Fashion-MNIST, CIFAR-10accuracyImproved
increase the number of Data Impressions in the transfer set (ablation)ZSKD · MNIST, Fashion-MNIST, CIFAR-10accuracyImproved
distill from an adversarially trained teacher instead of a normal teacher using DIs (PGD attack) (ablation)ZSKD student · MNIST PGD attackadversarial accuracy: 3.06 -> 54.38Improved
distill from an adversarially trained teacher instead of a normal teacher using DIs (PGD attack) (ablation)ZSKD student · CIFAR-10 PGD attackadversarial accuracy: 15.24 -> 24.92Improved
distill from a robust teacher using DIs, natural accuracy cost (ablation)ZSKD student · CIFAR-10 naturalaccuracy: 69.56 -> 54.38Hurt
use a lower beta mixture {0.01, 0.1} instead of {0.1, 1.0} for DI generation (ablation)Source-free domain adaptation, SVHN to MNIST · SVHN to MNIST domain adaptationaccuracy: 0.690 -> 0.866Improved
use a lower beta mixture {0.01, 0.1} instead of {0.1, 1.0} for DI generation (ablation)Source-free domain adaptation, USPS to MNIST · USPS to MNIST domain adaptationaccuracy: 0.7505 -> 0.8915Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy