Research map / Architectures and efficiency / RNNs and normalization
Beyond Random: Automatic Inner-loop Optimization in Dataset Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AT-BPTT, an inner-loop dataset distillation method that adaptively chooses truncation positions and window sizes by stage-aware gradient statistics, plus low-rank Hessian approximation and patch-wise semantic preservation.
Problem
Inner-loop dataset distillation via backpropagation through time relies on random or fixed truncation, which ignores the stage-dependent learning dynamics of neural networks and yields suboptimal distilled data at high computational cost.
Approach
Adapt truncation positions probabilistically using softmax-normalized gradient magnitudes, set per-timestep window sizes from normalized gradient variation, switch training stages via accuracy-variation thresholds, and approximate Hessians with randomized low-rank SVD.
Key results
- AT-BPTT reaches 54.4% on CIFAR-10 IPC=1 and 72.4% at IPC=10, versus 53.2% and 69.4% for RaT-BPTT.
- Achieves 30.6% on ImageNet-1K IPC=10 versus 13.0% for RaT-BPTT, and 32.7% on Tiny-ImageNet IPC=10 versus 24.4%.
- Reports 3.9x training speedup and 63% memory reduction compared with RaT-BPTT, with an average accuracy gain of 6.16%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AT-BPTT instead of RaT-BPTT | Dataset distillation with ConvNet · CIFAR-10, CIFAR-100, Tiny-ImageNet and ImageNet-1K, reported IPC settings | accuracy: average accuracy gain of 6.16% over RaT-BPTT | Helped |
| Use AT-BPTT instead of RaT-BPTT | Dataset distillation · Compared with RaT-BPTT | training speed: 3.9× | Helped |
| Use AT-BPTT instead of RaT-BPTT | Dataset distillation · Compared with RaT-BPTT | GPU memory usage: 63% reduction | Helped |
| Select preliminary timesteps in the early training stage (ablation) | RaT-BPTT on ConvNets · CIFAR-10 | validation accuracy: 2.9% | Helped |
| Choose preliminary rather than post timesteps in the middle stage (ablation) | RaT-BPTT on ConvNets · CIFAR-10 | validation accuracy: Δ<0.5% | No clear effect |
| Select post timesteps in the late training stage (ablation) | RaT-BPTT on ConvNets · CIFAR-10 | validation accuracy: 1.8% | Helped |
| Use a larger window in the early stage (ablation) | Stage-adaptive truncation · CIFAR-10 | accuracy: 2.5% | Helped |
| Vary window size in the middle or late stage (ablation) | Stage-adaptive truncation · CIFAR-10 | accuracy: Δ<0.2% | No clear effect |
| Add dynamic truncation position (DTP) alone (ablation) | AT-BPTT · CIFAR-10/100, Tiny-ImageNet and ImageNet-1K IPC=10 | accuracy: ↑1.3; ↑0.6; ↑2.9; ↑6.7 | Helped |
| Add adaptive window size (AWS) alone (ablation) | AT-BPTT · CIFAR-10/100, Tiny-ImageNet and ImageNet-1K IPC=10 | accuracy: ↑0.8; ↑0.4; ↑2.2; ↑4.9 | Helped |
| Add patch-wise semantic preservation (PSP) alone (ablation) | AT-BPTT · CIFAR-10/100, Tiny-ImageNet and ImageNet-1K IPC=10 | accuracy: ↑0.1; ↑0.3; ↑1.0; ↑4.2 | Mixed |
| Add DTP and AWS together (ablation) | AT-BPTT · CIFAR-10/100, Tiny-ImageNet and ImageNet-1K IPC=10 | accuracy: ↑2.8; ↑1.2; ↑6.7; ↑13.4 | Helped |
Surprising findings
- Middle-stage truncation position has negligible effect on accuracy, suggesting random truncation is adequate there.
Limitations
- Relies on simple CNN architectures (ConvNet) rather than modern networks.
- Inner-loop unrolling still incurs notable computational overhead.
- Many hyperparameters (M, N, X, Y, tau, d) require tuning.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNet-1K
Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNet-1K
Metrics: test accuracy GPU memory training time
Baselines: DSA CAFE MTT TESLA DATM ATT MCT NCFM BPTT FRePO RCIG RaT-BPTT
Builds on: BPTT T-BPTT RaT-BPTT MTT
Closest papers by meaning
- Embarrassingly Simple Dataset Distillation (ICLR 2024 · similarity 0.89)
- Large Scale Dataset Distillation with Domain Shift (ICML 2024 · similarity 0.86)
- Data Distillation Can Be Like Vodka: Distilling More Times For Better Quality (ICLR 2024 · similarity 0.85)
- Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximation (ICLR 2025 · similarity 0.85)
- Reducing Fine-Tuning Memory Overhead by Approximate and Memory-Sharing Backpropagation (ICML 2024 · similarity 0.84)
- Efficient Dataset Distillation for Pre-Trained Self-Supervised Models via Statistical Flow Matching (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.