Research map / Label-efficient and robust learning / Knowledge distillation
Emphasizing Discriminative Features for Dataset Distillation in Complex Scenarios
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EDF, a trajectory-matching dataset distillation method that uses Grad-CAM maps to up-weight discriminative pixel gradients and drops low-loss supervision, plus the Comp-DD benchmark of easy/hard ImageNet-1K subsets.
Problem
Dataset distillation performs well on simple datasets (CIFAR, MNIST, TinyImageNet) but degrades substantially on complex scenarios with varied object sizes and much class-irrelevant content, where discriminative features occupy a small image area.
Approach
Use Grad-CAM activation maps to rescale pixel gradients so discriminative regions receive more updates, and drop the lowest trajectory-matching-loss supervision elements that carry common patterns.
Key results
- On ImageMeow and ImageYellow at IPC300, EDF reaches 65.9% and 84.2%, with ImageYellow's full-data accuracy at 83.2%, i.e., lossless performance on these subsets.
- EDF outperforms DATM across most ImageNet-1K subset settings, e.g., ImageMeow IPC10 52.6 vs. DATM's 48.9 range in Table 1 (EDF reported as 52.6).
- On Comp-DD, EDF achieves higher accuracy than FTD and DATM on Bird, Dog, and Car categories, and easy subsets show higher recovery ratios than hard ones.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add EDF's discriminative-feature emphasis and common-pattern dropout to DATM | DATM · ImageMeow, IPC10 | accuracy: 48.9±1.1 -> 52.6±0.4 | Helped |
| Add EDF's discriminative-feature emphasis and common-pattern dropout to DATM | DATM · ImageSquawk, IPC10 | accuracy: 61.8±1.3 -> 65.4±0.8 | Helped |
| Add EDF's discriminative-feature emphasis and common-pattern dropout to DATM | DATM · ImageNet100, IPC50 | accuracy: 42.7±0.7 -> 44.2±0.7 | Helped |
| Add Discriminative Area Enhancement (DAE) (ablation) | EDF · ImageMeow, IPC10 | accuracy: 49.5 -> 52.6 | Helped |
| Add Common Pattern Dropout (CPD) (ablation) | EDF · ImageMeow, IPC10 | accuracy: 51.2 -> 52.6 | Helped |
| Set enhancement factor β to 10 instead of β=1 (ablation) | EDF · ImageMeow, IPC1 | accuracy: 34.5 -> 31.8 | Hurt |
| Set enhancement factor β to 0.5 instead of β=1 (ablation) | EDF · ImageMeow, IPC1 | accuracy: 34.5 -> 33.4 | Hurt |
| Use the dynamic mean activation threshold rather than a fixed activation threshold (ablation) | EDF · ImageMeow | accuracy: 34.2 -> 34.5; 52.3 -> 52.6; 59.0 -> 59.5 | Helped |
| Use an excessive supervision dropout ratio of 75% (ablation) | EDF · ImageFruit, IPC10 | accuracy: 46.5 -> 44.3 | Hurt |
| Increase activation-map update frequency from 1 to 200 iterations (ablation) | EDF · ImageNette, IPC1 | accuracy: 49.4 -> 51.2 | Helped |
| Use random parameter dropout instead of loss-based dropout (ablation) | EDF · Table 8 evaluation setting | accuracy: 74.8 -> 71.3 | Hurt |
Surprising findings
- Low-loss trajectory-matching supervision tends to shrink discriminative activation areas and embed common patterns, so dropping it helps.
- Applying knowledge distillation for evaluation reduces EDF's advantage at low IPC (IPC1, IPC10).
Limitations
- Built on trajectory matching; applying to other matching methods requires modifications not evaluated in the main text.
- Comp-DD full benchmark release is promised as future work; only partial results are reported.
- Knowledge-distillation-based evaluation reduces EDF's gains at small IPC.
Experimental setup
Datasets: ImageNette ImageWoof ImageMeow ImageYellow ImageFruit ImageSquawk ImageNet100 CIFAR-10 Tiny-ImageNet ImageNet-1K
Benchmarks: ImageNette ImageWoof ImageMeow ImageYellow ImageFruit ImageSquawk ImageNet100 CIFAR-10 Tiny-ImageNet Comp-DD
Metrics: test accuracy recovery ratio
Baselines: MTT FTD DATM SRe2L RDED Random
Builds on: MTT (trajectory matching) DATM Grad-CAM SRe2L RDED
Closest papers by meaning
- Dataset Distillation for Pre-Trained Self-Supervised Vision Models (NeurIPS 2025 · similarity 0.88)
- Data Distillation Can Be Like Vodka: Distilling More Times For Better Quality (ICLR 2024 · similarity 0.88)
- Efficient Dataset Distillation for Pre-Trained Self-Supervised Models via Statistical Flow Matching (NeurIPS 2026 · similarity 0.87)
- Large Scale Dataset Distillation with Domain Shift (ICML 2024 · similarity 0.87)
- Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximation (ICLR 2025 · similarity 0.87)
- Enhancing Dataset Distillation via Non-Critical Region Refinement (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.