Research map / Image generation / Text-to-image diffusion
Knowledge Distillation Detection for Open-weights Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes knowledge distillation detection: given an open-weight student and API access to candidate teachers, it synthesizes inputs with a data-free generator and scores student-teacher alignment to identify the source teacher, for classifiers and text-to-image models.
Problem
Determining whether an open-weight student model was distilled from a given teacher when only the student's weights and the teacher's API are available, without training data or teacher weights.
Approach
A model-agnostic three-stage pipeline: data-free synthetic input construction (a label-conditioned generator trained with BN-statistics and confidence losses for classifiers; empty-prompt generations for diffusion models), followed by point-wise (KL or LPIPS) or set-level (ACS or CKA) alignment scores, and argmax selection over candidate teachers.
Key results
- On CIFAR-10 with N=1, KL-based method reaches 0.62 accuracy and 0.75 AUC, and the average accuracy is 0.83 (ACS) across N, claimed 59.6% improvement over strongest baselines.
- On ImageNet, the KL variant averages 0.65 accuracy / 0.86 AUC, and ACS averages 0.67 accuracy, versus baselines around 0.33-0.36 accuracy.
- On text-to-image, LPIPS-based scoring achieves 0.89 accuracy and 1.00 AUC with a single input and 1.00 accuracy from N=50 onward, versus about 0.80 accuracy for the best baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use synthetic inputs and KL scoring instead of the strongest reported baseline | Distillation detection · CIFAR-10 | accuracy: 0.51 -> 0.81 | Helped |
| use synthetic inputs and KL scoring instead of the strongest reported baseline | Distillation detection · ImageNet | accuracy: 0.33 -> 0.65 | Helped |
| use LPIPS point-wise scoring with the proposed input construction | Distillation detection · Text-to-image generation | accuracy: 0.80 -> 0.96 | Helped |
| replace synthetic inputs with OOD-filtered inputs (ablation) | ACS classification detector · CIFAR-10 | accuracy: 0.83 -> 0.56 | Hurt |
| replace synthetic inputs with OOD-filtered inputs (ablation) | ACS classification detector · ImageNet | accuracy: 0.67 -> 0.37 | Hurt |
| replace ACS with CKA while retaining synthetic inputs (ablation) | Classification detector · CIFAR-10 | accuracy: 0.83 -> 0.82 | No clear effect |
| replace ACS with CKA while retaining synthetic inputs (ablation) | Classification detector · ImageNet | accuracy: 0.67 -> 0.62 | Hurt |
| increase the number of synthesized inputs | Ours (KL) · CIFAR-10 | accuracy: 0.62 -> 0.87 | Helped |
| increase the number of synthesized inputs | Ours (KL) · ImageNet | accuracy: 0.47 -> 0.75 | Helped |
| increase the number of inputs | Ours (LPIPS) · Text-to-image generation | accuracy: 0.89 -> 1.00 | Helped |
| increase the distillation weight lambda (ablation) | Student model and detection method · Knowledge distillation detection | detection performance | Helped |
| use a low distillation weight (lambda < 0.5) (ablation) | Student model and detection method · Knowledge distillation detection | detection performance | Hurt |
Surprising findings
- Detection can fail when distillation weight lambda is small (<0.5), where student-teacher scores are comparable to those with an independently trained same-architecture model.
- On ImageNet, the RN18 students distilled from RN50 or RNXt50 are misidentified as RN18, suggesting architectural mismatch makes detection harder.
Limitations
- Detection is less reliable when the student inherits only limited influence from the teacher (small distillation weight).
- Experiments are restricted to image classifiers and text-to-image models; free-form inputs for diffusion and LLMs are left to future work.
- Multiple-choice setup assumes the true teacher is among the candidate set; pairwise binary detection is only briefly tested (30 pairs).
Experimental setup
Datasets: CIFAR-10 ImageNet ImageNet-100
Benchmarks: CIFAR-10 ImageNet text-to-image distilled models (AMD, DMD2, SDXL-Lightning, BK-SDM-v2)
Metrics: accuracy AUC F1
Baselines: MIA Filter + KL OOD Filter + KL MIA Filter + CKA OOD Filter + CKA MMD-FUSE BLIP-Base captioning + CLIP/DINO/ACS/MMD GPT-2 captioning + CLIP/DINO/ACS/MMD Oracle
Builds on: knowledge distillation data-free knowledge distillation membership inference attack MMD-FUSE Aligned Cosine Similarity
Closest papers by meaning
- Small Scale Data-Free Knowledge Distillation (CVPR 2024 · similarity 0.88)
- Knowledge Distillation from A Stronger Teacher (NeurIPS 2022 · similarity 0.88)
- Debiased Distillation for Consistency Regularization (AAAI 2025 · similarity 0.87)
- Hybrid Data-Free Knowledge Distillation (AAAI 2025 · similarity 0.86)
- Impartial Adversarial Distillation: Addressing Biased Data-Free Knowledge Distillation via Adaptive Constrained Optimization (AAAI 2024 · similarity 0.86)
- Progressively Knowledge Distillation via Re-parameterizing Diffusion Reverse Process (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.