Research map / Label-efficient and robust learning / Recognition and long-tail
DELT: A Simple Diversity-driven EarlyLate Training for Dataset Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DELT, a diversity-driven EarlyLate training scheme for batch-to-global dataset distillation that staggers optimization iterations across IPC sub-batches initialized from teacher-ranked real patches, cutting synthesis time while improving accuracy.
Problem
Batch-to-global matching dataset distillation produces low intra-class diversity because each synthetic image is optimized independently against the same global BatchNorm supervision signals.
Approach
Partition the IPC samples into sub-batches that start optimization at staggered times (early vs. late), with later sub-batches joining the ongoing recovery rather than freezing earlier ones, and initialize from real image patches ranked by a pretrained teacher near the per-class median probability.
Key results
- On ImageNet-1K with IPC 50 and ResNet-101, DELT reaches 66.1% top-1, outperforming RDED (61.2%) by 4.9%.
- DELT reduces ImageNet-1K synthesis time by 39.3% (17.6 h vs 29.0 h for SRe2L/CDA at 4K iterations).
- Across cross-architecture tests on ImageNet-1K IPC 10, DELT improves ResNet-18 accuracy from 42.3 (RDED) to 46.4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace RDED with DELT | dataset distillation · CIFAR-10, IPC 50, ResNet-101 | top-1 accuracy: 51.6 ± 0.4 -> 54.1 ± 0.8 | Helped |
| replace RDED with DELT | dataset distillation · ImageNette, IPC 50, ResNet-101 | top-1 accuracy: 75.0 ± 1.2 -> 83.3 ± 1.1 | Helped |
| replace RDED with DELT | dataset distillation · Tiny-ImageNet, IPC 50, ResNet-101 | top-1 accuracy: 41.2 ± 0.4 -> 58.5 ± 0.3 | Helped |
| replace RDED with DELT | dataset distillation · ImageNet-100, IPC 50, ResNet-101 | top-1 accuracy: 66.0 ± 0.6 -> 70.8 ± 2.3 | Helped |
| replace RDED with DELT | dataset distillation · ImageNet-1K, IPC 50, ResNet-101 | top-1 accuracy: 61.2 ± 0.4 -> 66.1 ± 0.5 | Helped |
| use 1 × 1 rather than larger image-patch mosaics for initialization (ablation) | DELT · ImageNet-1K, IPC 10, ResNet-18 | top-1 accuracy: 56.92 -> 57.57; 56.62 -> 57.57; 56.71 -> 57.57; 56.51 -> 57.57 | Helped |
| select medium-probability crops rather than lowest-probability crops for initialization (ablation) | DELT · ImageNet-1K, IPC 10, ResNet-18 | top-1 accuracy: 57.55 -> 57.67 | No clear effect |
| select medium-probability crops rather than highest-probability crops for initialization (ablation) | DELT · ImageNet-1K, IPC 10, ResNet-18 | top-1 accuracy: 57.03 -> 57.67 | No clear effect |
| set MI to 4K and RI to 500 rather than MI 4K and RI 1K (ablation) | DELT · ImageNet-1K, IPC 10, ResNet-18 | top-1 accuracy: 44.66 -> 46.42 | Helped |
| add EarlyLate training (ablation) | CDA with real image initialization · ImageNet-1K, IPC 10, ResNet-18 | top-1 accuracy: 43.5 -> 46.1 | Helped |
| add EarlyLate training (ablation) | CDA with real image initialization · Tiny-ImageNet, IPC 10, ResNet-18 | top-1 accuracy: 42.2 -> 43.0 | Helped |
| add EarlyLate training (ablation) | CDA with real image initialization · CIFAR-10, IPC 10, ResNet-18 | top-1 accuracy: 39.4 -> 43.0 | Helped |
Surprising findings
- A 1x1 (single real patch) initialization outperformed larger mosaic patterns such as 2x2 to 5x5, contrary to the RDED mosaic design.
Limitations
- IPC=1 cannot use the EarlyLate strategy, so a single image per class is optimized for a constant number of iterations.
- Gains on small-scale datasets are not uniform; e.g., on ImageNette and some ConvNet ImageNet-100 settings DELT underperforms RDED.
Experimental setup
Datasets: CIFAR-10 Tiny-ImageNet ImageNet-1K ImageNette ImageNet-100
Benchmarks: CIFAR-10 Tiny-ImageNet ImageNet-1K ImageNette ImageNet-100
Metrics: top-1 accuracy synthesis time (hours) intra-class cosine similarity
Baselines: MTT IDM TESLA SRe2L DATM RDED CDA G-VBSM MinimaxDiffusion
Builds on: SRe2L CDA RDED MTT D3
Closest papers by meaning
- Efficient Dataset Distillation for Pre-Trained Self-Supervised Models via Statistical Flow Matching (NeurIPS 2026 · similarity 0.89)
- Diversity-Driven Synthesis: Enhancing Dataset Distillation through Directed Weight Adjustment (NeurIPS 2024 · similarity 0.89)
- Dataset Distillation for Pre-Trained Self-Supervised Vision Models (NeurIPS 2025 · similarity 0.89)
- SelMatch: Effectively Scaling Up Dataset Distillation via Selection-Based Initialization and Partial Updates by Trajectory Matching (ICML 2024 · similarity 0.88)
- D^4: Dataset Distillation via Disentangled Diffusion Model (CVPR 2024 · similarity 0.88)
- MGD$^3$ : Mode-Guided Dataset Distillation using Diffusion Models (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.