Research map / Architectures and efficiency / RNNs and normalization
Elucidating the Design Space of Dataset Condensation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EDC, a dataset condensation method that replaces bi-level optimization with statistical matching plus real-image initialization, soft category-aware matching, and post-evaluation tweaks, reported on CIFAR, Tiny-ImageNet and ImageNet-1k.
Problem
Dataset condensation methods rely on costly bi-level optimization and suffer from poor performance on either small or large-scale datasets, with unclear hyperparameter choices and loss of information.
Approach
Uses generalized data synthesis via statistical matching of global statistics from a model trained on the full dataset (no inner/outer loop), combined with a series of design improvements (real image initialization, smoothing LR schedule, flatness regularization, small batch size, soft category-aware matching, weak augmentation, EMA evaluation) that generalize across small and large datasets.
Key results
- On CIFAR-10 with ResNet-18, EDC reaches 79.1% at IPC 10 and 87.0% at IPC 50, versus 37.1% and 62.1% for RDED.
- On ImageNet-1k with ResNet-18, EDC reaches 48.6% at IPC 10 and 58.0% at IPC 50, versus 42.0% and 56.5% for RDED.
- On Tiny-ImageNet under IPC 50 adversarial evaluation, EDC achieves 57.21 clean accuracy and 10.71 under PGD, versus 26.16 and 0.41 for MTT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EDC instead of RDED | ResNet-18 dataset condensation · CIFAR-10, IPC 10 | accuracy: 37.1 $\pm$ 0.3 -> 79.1 $\pm$ 0.3 | Helped |
| Use EDC instead of G-VBSM | ResNet-18 dataset condensation · CIFAR-100, IPC 10 | accuracy: 59.5 $\pm$ 0.4 -> 63.7 $\pm$ 0.3 | Helped |
| Use EDC instead of WMDD | ResNet-18 dataset condensation · Tiny-ImageNet, IPC 50 | accuracy: 59.4 $\pm$ 0.5 -> 57.2 $\pm$ 0.2 | Hurt |
| Use EDC instead of RDED | ResNet-18 dataset condensation · ImageNet-10, IPC 10 | accuracy: 53.3 $\pm$ 0.1 -> 63.4 $\pm$ 0.2 | Helped |
| Use EDC instead of RDED | ResNet-18 dataset condensation · ImageNet-1k, IPC 10 | accuracy: 42.0 $\pm$ 0.1 -> 48.6 $\pm$ 0.3 | Helped |
| Use EDC instead of GCOND | Graph dataset condensation evaluated with GCN · Cora, 1.3% | accuracy: 79.8 -> 80.1 | Helped |
| Add real image initialization to G-VBSM (ablation) | G-VBSM · Tiny-ImageNet, CONFIG A to CONFIG B | Top-1 accuracy (%): 31.4 -> 34.4 | Helped |
| Add smoothing LR schedule to real-image-initialized G-VBSM (ablation) | G-VBSM · Tiny-ImageNet, CONFIG B to CONFIG C | Top-1 accuracy (%): 34.4 -> 38.7 | Helped |
| Add flatness regularization to the configuration with real image initialization and smoothing LR schedule (ablation) | G-VBSM · Tiny-ImageNet, CONFIG C to CONFIG D | Top-1 accuracy (%): 38.7 -> 39.5 | Helped |
| Add small batch size and better backbone choice (ablation) | G-VBSM with real image initialization, smoothing LR schedule, and flatness regularization · Tiny-ImageNet, CONFIG D to CONFIG E | Top-1 accuracy (%): 39.5 -> 46.2 | Helped |
| Add soft category-aware matching (ablation) | G-VBSM configuration with preceding improvements · Tiny-ImageNet, CONFIG E to CONFIG F | Top-1 accuracy (%): 46.2 -> 48.0 | Helped |
| Add weak augmentation and EMA-based evaluation (ablation) | G-VBSM configuration with preceding improvements · Tiny-ImageNet, CONFIG F to CONFIG G | Top-1 accuracy (%): 48.0 -> 48.6 | Helped |
Surprising findings
- Authors state that a small batch size can help because condensed-dataset gradients approximate the global gradient, and that the default RandomResizedCrop scale of 0.08 was too small, with 0.5 yielding a large gain.
Limitations
- Extending to new domains requires redesigning statistical matching and hyperparameter tuning, per the authors.
- Authors acknowledge the ImageNet-21k-P experiment was incomplete in the rebuttal (marked TODO), and the statistical matching loss did not reach zero at the end of synthesis.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNet-1k ImageNet-10 ImageNet-21k-P Cora
Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNet-1k ImageNet-10 Cora
Metrics: top-1 test accuracy robust accuracy under FGSM, PGD, CW, VMI, Jitter, AutoAttack GPU memory training time
Baselines: MTT TESLA SRe2L G-VBSM CDA WMDD RDED Random Herding K-Center GCOND GCOND-X
Builds on: G-VBSM SRe2L RDED CDA
Closest papers by meaning
- Ameliorate Spurious Correlations in Dataset Condensation (ICML 2024 · similarity 0.89)
- M3D: Dataset Condensation by Minimizing Maximum Mean Discrepancy (AAAI 2024 · similarity 0.88)
- Multisize Dataset Condensation (ICLR 2024 · similarity 0.87)
- Diversity-Driven Synthesis: Enhancing Dataset Distillation through Directed Weight Adjustment (NeurIPS 2024 · similarity 0.87)
- Rethinking Dataset Distillation for Classification: Do Distilled Sets Outperform Coresets? (NeurIPS 2026 · similarity 0.86)
- CondTSF: One-line Plugin of Dataset Condensation for Time Series Forecasting (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.