Research map / Label-efficient and robust learning / Knowledge distillation
Knowledge Distillation as Decontamination? Revisiting the “Data Laundering” Concern
Hengyu Luo, Raúl Vázquez, Timothee Mickus, Filip Ginter, Jörg Tiedemann
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| distill through a student instead of training directly on contaminated data | DistilBERT · tweet sentiment | accuracy gain from test contamination: 25.66% -> 3.25% | Improved |
| distill through a student instead of training directly on contaminated data | DistilBERT · 20newsgroups | accuracy gain from test contamination: 11.91% -> 1.42% | Improved |
| contaminated teacher distilled into clean student (laundering) | DistilBERT student · AGNews | accuracy gain: 0.65% gain, significant | Mixed |
| contaminated teacher distilled into clean student (laundering) | DistilBERT student · banking77, rotten tomatoes, SNLI | accuracy gain | No clear effect |
| replace BERT teacher with Llama3.2-1B (ablation) | clean student · AGNews | accuracy: 91.59 -> 91.08 (-0.51) | No clear effect |
| replace BERT teacher with Qwen3-0.6B (ablation) | clean student · rotten tomatoes | accuracy: 83.77 -> 84.22 (+0.45) | No clear effect |
| use stronger LLM teacher with contaminated data (ablation) | DistilBERT student · tweet sentiment | student accuracy gain: teacher gains 25.80%, student gain limited to 1.91% | No clear effect |
| use mixed objectives or soft labels instead of pure hard distillation (ablation) | DistilBERT student · eight benchmarks | laundering gain | Mixed |
| compare laundering with contamination at sample level | sample-level effects · agnews and tweet sentiment | Pearson correlation: 0.32 and 0.31 | No clear effect |
| compare laundering with contamination at sample level | sample-level effects · SNLI | Pearson correlation: -0.03 | No clear effect |
| widen train-test distributional gap (Level 1 to 5) (ablation) | student laundering gain · rotten tomatoes | accuracy gain: 0.4 -> 6.5 | Improved |
| widen train-test distributional gap (Level 1 to 5) (ablation) | student laundering gain · emotion | statistical significance: p-values 0.1455 to 0.0001 range across levels | Improved |
Closest papers by meaning
- Efficient Verified Unlearning For Distillation (NeurIPS 2025 · similarity 0.85)
- Time Travel in LLMs: Tracing Data Contamination in Large Language Models (ICLR 2024 · similarity 0.84)
- Scaling Trends for Data Poisoning in LLMs (AAAI 2025 · similarity 0.83)
- The Emperor's New Clothes in Benchmarking? A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination (ICML 2025 · similarity 0.83)
- Combating Data Laundering in LLM Training (NeurIPS 2026 · similarity 0.82)
- Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination’s Impact on Machine Translation (ICML 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
