Research map / Architectures and efficiency / RNNs and normalization
FADRM: Fast and Accurate Data Residual Matching for Dataset Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FADRM, a dataset distillation method that adds data-level residual connections (blending optimized images with real patches) plus mixed-precision and multi-resolution optimization to cut time and memory roughly in half while improving accuracy on ImageNet-1K and other benchmarks.
Problem
Uni-level dataset distillation suffers from information vanishing during optimization and high computational/memory cost for large-scale synthesis (e.g., EDC needs ~70 hours for 50 IPC on ImageNet-1K).
Approach
Introduces data residual matching: periodically fuses the optimized synthetic images with resized real patches via a merge ratio alpha (Adjustable Residual Connection), combined with mixed-precision training and multi-resolution optimization (downsample, optimize, upsample) to speed up generation.
Key results
- On ImageNet-1K IPC=10 with ResNet-18, FADRM+ (multi-model) reaches 50.0% vs EDC 48.6% and CV-DD 46.0%; single-model FADRM reaches 47.7% at 0.8% ratio per abstract.
- FADRM reduces time cost per image to 0.47 s vs 4.99 s for EDC-type baselines and peak memory to 2.9 GB for single-model; FADRM+ 1.09 s and 11.0 GB vs EDC 17.9 GB.
- Mixed precision training cuts peak GPU memory from 5.3 GB to 2.9 GB for single-model FADRM with unchanged ResNet-18 accuracy (47.7% vs 47.8% without MPT).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add data residual connections (ARC) to the FADRM framework (ablation) | FADRM on ImageNet-1K IPC=10 · ImageNet-1K IPC=10 | accuracy: 46.2 -> 47.7 | Helped |
| Add multi-resolution optimization (MRO) without ARC (ablation) | FADRM on ImageNet-1K IPC=10 · ImageNet-1K IPC=10 | accuracy: 46.4 -> 46.2 | Hurt |
| Use mixed-precision training (MPT) (ablation) | FADRM on ImageNet-1K IPC=10 · ImageNet-1K IPC=10 | time per image per iteration: 0.63 ms -> 0.26 ms | Helped |
| Use mixed-precision training (MPT) (ablation) | FADRM on ImageNet-1K IPC=10 · ImageNet-1K IPC=10 | peak GPU memory: 5.3 GB -> 2.9 GB | Helped |
| Use mixed-precision training (MPT) (ablation) | FADRM on ImageNet-1K IPC=10 · ImageNet-1K IPC=10 | accuracy: 47.8 % -> 47.7 % | No clear effect |
| Set ARC merge ratio alpha to 0.5 (ablation) | FADRM on ImageNet-1K IPC=10 · ImageNet-1K IPC=10 | accuracy: 46.4 -> 47.7 | Helped |
| Set ARC merge ratio alpha to 0.5 (ablation) | FADRM+ on ImageNet-1K IPC=10 · ImageNet-1K IPC=10 | accuracy: 48.7 -> 50.0 | Helped |
| Use 2x2 patches rather than 1x1 patches (ablation) | FADRM on ImageNet-1K IPC=10 · ImageNet-1K IPC=10 | accuracy: 48.4 -> 47.7 | Hurt |
| Increase ARC frequency from k=1 to k=3 (ablation) | FADRM on ImageNet-1K IPC=10 · ImageNet-1K IPC=10 | accuracy: 47.1 -> 47.8 | Helped |
| Increase ARC frequency beyond k=3 to k=6 (ablation) | FADRM on ImageNet-1K IPC=10 · ImageNet-1K IPC=10 | accuracy: 47.8 -> 47.3 | Hurt |
| Use downsampled size Dds=200 instead of Dds=224 (ablation) | FADRM on ImageNet-1K IPC=10 · ImageNet-1K IPC=10 | accuracy: 47.7 -> 47.7 | No clear effect |
| Use downsampled size Dds=200 instead of Dds=224 (ablation) | FADRM on ImageNet-1K IPC=10 · ImageNet-1K IPC=10 | time per image: 0.52 -> 0.47 | Helped |
Surprising findings
- Multi-resolution optimization alone (without residual connection) lowers accuracy slightly (46.2% vs 46.4%), and performance decreases for k beyond 3, attributed to redundant local details.
Limitations
- Relies on assumption that residual signals capture critical learning dynamics; may not generalize to non-visual modalities such as text or time series.
- Distilled data may reinforce dataset biases; fairness concerns noted by authors.
- Multi-resolution optimization is applied only to large-scale datasets since downscaling gives no efficiency gain for small inputs.
Experimental setup
Datasets: ImageNet-1K CIFAR-100 Tiny-ImageNet ImageNette ImageWoof
Benchmarks: CIFAR-100 Tiny-ImageNet ImageNet-1K ImageNette ImageWoof
Metrics: top-1 accuracy time cost per image (s) peak GPU memory (GB) continual learning accuracy
Baselines: RDED EDC CV-DD SRe2L++ G-VBSM
Builds on: EDC SRe2L RDED CV-DD
Closest papers by meaning
- D^4: Dataset Distillation via Disentangled Diffusion Model (CVPR 2024 · similarity 0.89)
- Large Scale Dataset Distillation with Domain Shift (ICML 2024 · similarity 0.89)
- Unlocking Dataset Distillation with Diffusion Models (NeurIPS 2025 · similarity 0.88)
- DELT: A Simple Diversity-driven EarlyLate Training for Dataset Distillation (CVPR 2025 · similarity 0.87)
- Dataset Distillation for Pre-Trained Self-Supervised Vision Models (NeurIPS 2025 · similarity 0.86)
- MGD$^3$ : Mode-Guided Dataset Distillation using Diffusion Models (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.