Research map / Federated learning and privacy / Privacy attacks and secure learning
Students Parrot Their Teachers: Membership Inference on Model Distillation
Matthew Jagielski, Milad Nasr, Katherine Lee, Christopher A. Choquette-Choo, Nicholas Carlini, Florian Tramèr
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Distill the teacher into a student rather than attack the teacher directly | Teacher training examples · CIFAR-10, WikiText103, Purchase-100, Texas-100 | Membership inference vulnerability: 5% of examples' vulnerability drop by less than 8 percentage points on CIFAR-10, 5 points on Purchase-100, and 4 points | Mixed |
| Use student queries to infer teacher-example membership without querying on the target | Teacher training examples · CIFAR-10 | Membership inference accuracy: 73% | Improved |
| Use teacher-label logits instead of student-label logits (ablation) | Indirect student-query attack · Teacher-example membership inference | Attack performance | Improved |
| Filter student queries to the most informative examples (ablation) | Indirect student-query attack · Teacher-example membership inference | Attack performance | Improved |
| Use teacher-label logits and filter student queries (ablation) | Indirect student-query attack · Teacher-example membership inference | TPR at FPR: TPR of $10^{-3}$ at a FPR of $10^{-4}$ | Improved |
| Use neither teacher-label scoring nor query filtering (ablation) | Indirect student-query attack · Teacher-example membership inference | Attack performance | Hurt |
| Allow near-duplicate examples across teacher and student datasets | Indirect student-query attack · CIFAR-10 teacher examples with student-set duplicates | Membership inference vulnerability: $p<10^{-15}$ | Hurt |
| Increase the number of poisoned teacher examples | Teacher-set membership inference · CIFAR-10 | Membership inference accuracy | Hurt |
| Reduce distillation temperature from 4 to 0.1 | Student model · Teacher-example membership inference | TPR at FPR: reducing TPR by a factor of roughly 4 at an FPR of $10^{-3}$ | Improved |
| Give the attacker less knowledge of the teacher model | Private-student membership inference · CIFAR-10 | Attack success: a TPR as large as $10^{-2}$ at a FPR of $10^{-3}$ | Improved |
| Increase the distillation-loss weight alpha in self-distillation | Self-distilled student model · Self-distillation | Membership inference attack success | Hurt |
| Train a student model from student queries rather than attack the queries alone | Teacher-example membership inference · Teacher training examples | Attack success | Improved |
Closest papers by meaning
- Privacy Backdoors: Enhancing Membership Inference through Poisoning Pre-trained Models (NeurIPS 2024 · similarity 0.88)
- Efficient Verified Unlearning For Distillation (NeurIPS 2025 · similarity 0.87)
- Mitigating Privacy Risk in Membership Inference by Convex-Concave Loss (ICML 2024 · similarity 0.87)
- Assessing Per-Sample Membership Inference Vulnerability without Retraining (NeurIPS 2026 · similarity 0.87)
- Sequential Membership Inference Attacks (NeurIPS 2026 · similarity 0.87)
- PANORAMIA: Privacy Auditing of Machine Learning Models without Retraining (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
