Research map / Architectures and efficiency / RNNs and normalization
Diversity-Driven Synthesis: Enhancing Dataset Distillation through Directed Weight Adjustment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Directed Weight Adjustment (DWA), which perturbs the teacher network's weights along a loss-directed direction computed on a random real subset before synthesizing each mini-batch of distilled images, to boost diversity in isolated (SRe2L-style) dataset distillation.
Problem
Per-instance (isolated) synthesis in dataset distillation, such as SRe2L, tends to produce synthetic images with low diversity, and random initialization contributes little diversity, which limits the representativeness of distilled datasets.
Approach
Analyzes the BN loss variance term as the main source of diversity in isolated synthesis, decouples its coefficient from the mean term, and injects diversity by a directed (gradient-ascent-based) weight perturbation of the teacher on a random real subset, which adds randomness without degrading the supervision signal.
Key results
- On ImageNet-1K with ipc=10 and ResNet-18, DWA reaches 37.9% vs. 21.3% for SRe2L (+16.6%).
- On CIFAR-100 with ipc=50 (ResNet-18), DWA achieves 60.9% vs. 52.2% for SRe2L; on CIFAR-10 ipc=50 DWA gets 53.1% vs. 47.5%.
- Computational overhead is 7.32% extra time per ipc on CIFAR-100 (125.12 s vs. 116.58 s for SRe2L).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DWA instead of SRe2L | Dataset distillation on ImageNet-1K · ImageNet-1K, ipc 10, ResNet-18 | Top-1 accuracy: 21.3 ± 0.6 -> 37.9 ± 0.2 | Helped |
| Use DWA instead of SRe2L | Dataset distillation on Tiny-ImageNet · Tiny-ImageNet, ipc 50, ResNet-18 | Top-1 accuracy: 41.1 ± 0.4 -> 52.8 ± 0.2 | Helped |
| Use DWA instead of SRe2L | Dataset distillation on CIFAR-100 · CIFAR-100, ipc 50, ResNet-18 | Top-1 accuracy: 52.2 ± 0.3 -> 60.9 ± 0.5 | Helped |
| Use DWA instead of SRe2L | Dataset distillation on CIFAR-10 · CIFAR-10, ipc 50, ResNet-18 | Top-1 accuracy: 47.5 ± 0.5 -> 53.1 ± 0.3 | Helped |
| Replace no weight adjustment with directed weight adjustment (ablation) | DWA on CIFAR-100 · CIFAR-100, ipc 10, ResNet-18 | Top-1 accuracy: 30.6 ± 0.7 -> 39.6 ± 0.6 | Helped |
| Use random Gaussian weight adjustment instead of no adjustment (ablation) | DWA on CIFAR-100 · CIFAR-100, ipc 10, ResNet-18 | Top-1 accuracy: 30.6 ± 0.7 -> 14.9 ± 0.1 | Hurt |
| Use random Gaussian weight adjustment instead of no adjustment (ablation) | DWA on CIFAR-100 · CIFAR-100, ipc 50, ResNet-18 | Top-1 accuracy: 56.1 ± 0.4 -> 56.2 ± 0.6 | No clear effect |
| Replace random Gaussian adjustment with directed weight adjustment (ablation) | DWA on CIFAR-100 · CIFAR-100, ipc 50, ResNet-18 | Top-1 accuracy: 56.2 ± 0.6 -> 60.3 ± 0.5 | Helped |
| Increase the variance-loss coefficient separately rather than coupling it to the BN coefficient (ablation) | SRe2L and DWA on CIFAR-100 · CIFAR-100, ResNet-18 | Top-1 accuracy | Helped |
| Use real-instance initialization instead of Gaussian pixel-space initialization (ablation) | Synthetic dataset initialization · Dataset distillation | feature-space coverage | Helped |
Surprising findings
- Random (undirected) weight perturbation degrades performance at ipc=10 on CIFAR-100 (e.g., ResNet-18 drops from 30.6% to 14.9%), so the direction of perturbation matters.
- Simply increasing the BN loss coefficient hurts diversity because the mean regularizer conflicts with the variance regularizer; decoupling the variance coefficient helps.
Limitations
- Relies on sampling a random distribution for weight perturbation; increasing its variance can introduce noise that bottlenecks performance.
- Cannot be directly applied to models without BN layers such as standard Vision Transformers; requires a ViT-BN variant.
- Random perturbation harms performance at small ipc values.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNet-1K
Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNet-1K
Metrics: Top-1 accuracy normalized feature distance generation time per ipc
Baselines: DC DM CAFE MTT TESLA SRe2L DataDAM
Builds on: SRe2L MTT Batch Normalization loss
Closest papers by meaning
- DELT: A Simple Diversity-driven EarlyLate Training for Dataset Distillation (CVPR 2025 · similarity 0.89)
- Data Distillation Can Be Like Vodka: Distilling More Times For Better Quality (ICLR 2024 · similarity 0.88)
- MGD$^3$ : Mode-Guided Dataset Distillation using Diffusion Models (ICML 2025 · similarity 0.88)
- D^4: Dataset Distillation via Disentangled Diffusion Model (CVPR 2024 · similarity 0.87)
- Large Scale Dataset Distillation with Domain Shift (ICML 2024 · similarity 0.87)
- Embarrassingly Simple Dataset Distillation (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.