AI Paper Atlas

Research map / Label-efficient and robust learning / Knowledge distillation

Augmentation with Projection: Towards an Effective and Efficient Data Augmentation Paradigm for Distillation

Ziqi Wang, Yuexin Wu, Frederick Liu, Daogao Liu, Le Hou, Hongkun Yu, Jing Li, Heng Ji

ICLR 2023 · Label-efficient and robust learning › Knowledge distillation · approach: Self and data-free distillation · 2 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add AugPro projection to MixUp (AugPro-Mix) (ablation)KD with MixUp augmentation · GLUE CoLA / RTE / MNLIMatthew / Acc: +5.97 CoLA, +9.02 RTE over MixUpImproved
add AugPro projection to FGSM (AugPro-FGSM) (ablation)KD with FGSM augmentation · GLUE CoLA / RTEMatthew / Acc: +10.52 CoLA, +8.31 RTE over FGSMImproved
combine AugPro-Mix and AugPro-FGSMKD · GLUE all listed tasksaccuracy / Matthew: KD improved by roughly 2% to 14%Improved
AugPro projection on MixUp/FGSM with small-data (10% labeled) settingKD · 10% labeled GLUE (MNLI, SST-2, QNLI)accuracy: about 1-2% on averageImproved
AugPro projection (vs. MixUp/FGSM) adds little time overheadKD training cost · Time per 1000 steps on TPU v3minutes: AugPro-Mix 3.48 vs MixUp 3.48; AugPro-FGSM 3.24 vs FGSM 3.24No clear effect
back translation as augmentation with modelsKD · Time per 1000 stepsminutes: 13.15 vs 1.68 for KDHurt
random sign in FGSM projection (AugPro-FGSMR) or opposite sign (AugPro-FGSMD) (ablation)AugPro-FGSM · GLUE MNLI/SST-2/QNLIaccuracy: SST-2 94.27 for both variants vs 94.61 full; MNLI within about 0.5No clear effect
use original labels with AugPro-Mix/AugPro-FGSM data (no label diversity) (ablation)AugPro data fine-tuning · Data size 20%-100% (SST-2, MNLI-M)accuracy: Finetune 84.69/84.26 vs AugPro-Mix 84.53/83.92 at 100%Hurt
epsilon too large in AugPro-FGSM (100 vs 35) (ablation)AugPro-FGSM · SST-2accuracy: 94.27 at eps=35 vs 91.97 at eps=100Hurt
random token concatenation augmentation (Random) (ablation)KD · GLUE CoLA / SST-2 / RTEMatthew / Acc: CoLA 45.89 vs KD 45.56; RTE 61.37 vs 61.37No clear effect
add MixUp, FGSM, or TMix/KNN augmentation to KDKD · GLUE CoLA and RTEMatthew / Acc: KNN RTE 66.79 vs KD 61.37 but TMix CoLA 44.42 vs 45.56 KDMixed

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy