Research map / Label-efficient and robust learning / Knowledge distillation
Augmentation with Projection: Towards an Effective and Efficient Data Augmentation Paradigm for Distillation
Ziqi Wang, Yuexin Wu, Frederick Liu, Daogao Liu, Le Hou, Hongkun Yu, Jing Li, Heng Ji
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add AugPro projection to MixUp (AugPro-Mix) (ablation) | KD with MixUp augmentation · GLUE CoLA / RTE / MNLI | Matthew / Acc: +5.97 CoLA, +9.02 RTE over MixUp | Improved |
| add AugPro projection to FGSM (AugPro-FGSM) (ablation) | KD with FGSM augmentation · GLUE CoLA / RTE | Matthew / Acc: +10.52 CoLA, +8.31 RTE over FGSM | Improved |
| combine AugPro-Mix and AugPro-FGSM | KD · GLUE all listed tasks | accuracy / Matthew: KD improved by roughly 2% to 14% | Improved |
| AugPro projection on MixUp/FGSM with small-data (10% labeled) setting | KD · 10% labeled GLUE (MNLI, SST-2, QNLI) | accuracy: about 1-2% on average | Improved |
| AugPro projection (vs. MixUp/FGSM) adds little time overhead | KD training cost · Time per 1000 steps on TPU v3 | minutes: AugPro-Mix 3.48 vs MixUp 3.48; AugPro-FGSM 3.24 vs FGSM 3.24 | No clear effect |
| back translation as augmentation with models | KD · Time per 1000 steps | minutes: 13.15 vs 1.68 for KD | Hurt |
| random sign in FGSM projection (AugPro-FGSMR) or opposite sign (AugPro-FGSMD) (ablation) | AugPro-FGSM · GLUE MNLI/SST-2/QNLI | accuracy: SST-2 94.27 for both variants vs 94.61 full; MNLI within about 0.5 | No clear effect |
| use original labels with AugPro-Mix/AugPro-FGSM data (no label diversity) (ablation) | AugPro data fine-tuning · Data size 20%-100% (SST-2, MNLI-M) | accuracy: Finetune 84.69/84.26 vs AugPro-Mix 84.53/83.92 at 100% | Hurt |
| epsilon too large in AugPro-FGSM (100 vs 35) (ablation) | AugPro-FGSM · SST-2 | accuracy: 94.27 at eps=35 vs 91.97 at eps=100 | Hurt |
| random token concatenation augmentation (Random) (ablation) | KD · GLUE CoLA / SST-2 / RTE | Matthew / Acc: CoLA 45.89 vs KD 45.56; RTE 61.37 vs 61.37 | No clear effect |
| add MixUp, FGSM, or TMix/KNN augmentation to KD | KD · GLUE CoLA and RTE | Matthew / Acc: KNN RTE 66.79 vs KD 61.37 but TMix CoLA 44.42 vs 45.56 KD | Mixed |
Closest papers by meaning
- Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversity (NeurIPS 2025 · similarity 0.87)
- Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximation (ICLR 2025 · similarity 0.86)
- TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models (ICLR 2025 · similarity 0.85)
- Understanding when Dynamics-Invariant Data Augmentations Benefit Model-free Reinforcement Learning Updates (ICLR 2024 · similarity 0.85)
- DA-KD: Difficulty-Aware Knowledge Distillation for Efficient Large Language Models (ICML 2025 · similarity 0.85)
- DDK: Distilling Domain Knowledge for Efficient Large Language Models (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
