Research map / Architectures and efficiency / RNNs and normalization
Embarrassingly Simple Dataset Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RaT-BPTT, a randomly placed truncated backpropagation-through-time method for bilevel dataset distillation, plus Boost-DD, a boosting scheme that yields nested distilled subsets.
Problem
Dataset distillation as bilevel optimization is hard: full BPTT through long inner training is unstable, memory-heavy and slow, while prior methods rely on surrogates (NTK, GP, random features) or proxy objectives; distilled datasets also show strong intercorrelation so subsets perform poorly.
Approach
Unroll the inner training for many steps but backpropagate only through a randomly positioned fixed-size window along the trajectory (RaT-BPTT), giving a subsampled, more stable and cheaper version of full BPTT gradients; additionally, build distilled sets in nested blocks with reduced learning rates on earlier blocks (Boost-DD) to reduce intercorrelation.
Key results
- RaT-BPTT reaches 53.2% on CIFAR-10 IPC1 and 69.4% IPC10 with narrow ConvNet, and 75.3% IPC50, state of the art at IPC10/50 on CIFAR-100, CIFAR-10 and CUB-200.
- Transferring data distilled with a narrow network to a wide network gives 54.1% (CIFAR-10 IPC1) and 71.0% (IPC10).
- Random subsamples of IPC50 distilled data from KIP or RaT-BPTT can perform worse than random real images of the same size; Boost-DD keeps nested subsets near joint-distillation performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| combine truncation with random placement of the backpropagation window (ablation) | BPTT dataset distillation · CIFAR-10 IPC10 | test accuracy during distillation | Helped |
| use a fixed truncated window instead of the full trajectory (ablation) | BPTT dataset distillation · CIFAR-10 IPC10 | test accuracy | Hurt |
| randomly place the truncated window rather than use full unrolling (ablation) | BPTT dataset distillation · CIFAR-10 IPC10 | test accuracy | Hurt |
| use random uniform window placement rather than forward- or backward-moving placement (ablation) | RaT-BPTT · CIFAR-10 IPC10 | test accuracy during distillation | Helped |
| sample the truncation window more often in the initial phase rather than uniformly (ablation) | RaT-BPTT · CIFAR-10 IPC10 | final accuracy: +0.4\% | Helped |
| increase unrolling length to 300 while keeping window size 40 (ablation) | RaT-BPTT · CIFAR-10 IPC10 | test accuracy | Hurt |
| use window size 60 or 100 rather than 40 with unrolling length 200 (ablation) | RaT-BPTT · CIFAR-10 IPC10 | test accuracy | No clear effect |
| add linear-basis data parameterization (ablation) | RaT-BPTT · CIFAR-10 IPC1 | accuracy: 53.2\% -> 68.2\% | Helped |
| add linear-basis data parameterization (ablation) | RaT-BPTT · CIFAR-10 IPC10 | accuracy: 69.4\% -> 72.8\% | Helped |
| construct the distilled dataset with Boost-DD rather than joint distillation (ablation) | RaT-BPTT · CIFAR-10 | test accuracy | Mixed |
| use weak boosting rather than strong boosting for larger IPC (ablation) | Boost-DD with RaT-BPTT · CIFAR-10 | test accuracy | Helped |
| sample IPC5 from an IPC50 distilled dataset rather than distill IPC5 directly (ablation) | RaT-BPTT · CIFAR-10 | test accuracy | Hurt |
Surprising findings
- Random subsamples of jointly distilled IPC50 data perform worse than training on randomly chosen real images of the same size.
- Randomly placed truncation windows outperform forward- or backward-moving windows, and plain BPTT gives higher accuracy than fixed truncation.
Limitations
- Memory consumption still high since intermediate parameters must be stored across the window and unrolling.
- Design choices (unroll length, window size) are tuned empirically and guided by intuition rather than theory.
- Evaluated only on image classification benchmarks with small ConvNets; not applied to larger models or other modalities.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 CUB-200 Tiny-ImageNet
Benchmarks: CIFAR-10 CIFAR-100 CUB-200 Tiny-ImageNet
Metrics: test accuracy
Baselines: BPTT KIP RFAD FRePO RCIG DSA DM MTT FTD
Builds on: backpropagation through time truncated BPTT KIP MTT LinBa
Closest papers by meaning
- Large Scale Dataset Distillation with Domain Shift (ICML 2024 · similarity 0.90)
- D^4: Dataset Distillation via Disentangled Diffusion Model (CVPR 2024 · similarity 0.89)
- Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matching (ICLR 2024 · similarity 0.89)
- Beyond Random: Automatic Inner-loop Optimization in Dataset Distillation (NeurIPS 2025 · similarity 0.89)
- Rethinking Dataset Distillation for Classification: Do Distilled Sets Outperform Coresets? (NeurIPS 2026 · similarity 0.89)
- Data Distillation Can Be Like Vodka: Distilling More Times For Better Quality (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.