Research map / Image generation / Autoregressive and latent generation
Hierarchical Features Matter: A Deep Exploration of Progressive Parameterization Method for Dataset Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes H-PD, which progressively optimizes latents across hierarchical layers of a pretrained StyleGAN-XL generator (GLaD-based) for dataset distillation, guided by a class-relevant feature distance for layer selection.
Problem
Existing parameterization-based dataset distillation methods fix a single optimization feature space (e.g., one GAN layer or DCT domain), which limits performance and requires manual selection of the feature domain for each setting and architecture.
Approach
Instead of optimizing in one fixed GAN feature layer, greedily search across all hierarchical GAN layers, passing the best latent from each layer as the initialization for the next, and select latents with a cheap class-relevant CAM-weighted feature distance rather than costly real-performance evaluation.
Key results
- On ImageNet-Subsets under IPC=1 with same architecture, H-PD improves TESLA-based distillation from 50.7 to 55.1 on ImNet-A and reaches 45.4 on ImNette vs 15.2 for D4M and 22.8 for Minimax.
- Under IPC=10 on ImageNet-1K with SRe2L, H-PD reaches 23.5% vs 21.9% for GLaD (ResNet-18).
- Averaged over ImageNet-[A-E], H-PD with DSA/DM achieves 43.2 and 39.1 vs GLaD 41.3 and 37.4 at comparable time (H-PD 73 min vs GLaD 69 min for DSA).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace GLaD with H-PD | GLaD with SRe2L · Tiny-ImageNet and ImageNet-1K, IPC-1 and IPC-10 | accuracy: 3.1±0.3 -> 4.5±1.0; 15.7±0.2 -> 18.3±0.5; 1.2±0.1 -> 2.6±0.2; 21.9±0.8 -> 23.5±0.4 | Helped |
| replace Minimax or D4M with H-PD | dataset distillation on ImageNet subsets at IPC-1 · ImNette, ImWoof, ImNet-Birds, ImNet-Fruits, ImNet-Cats | accuracy: 22.8±0.5 -> 45.4±1.1; 17.8±0.3 -> 28.3±0.2; 23.2±0.7 -> 39.7±0.8; 17.6±0.6 -> 25.6±0.7; 23.4±0.7 -> 29.6±1.0 | Helped |
| add average latent initialization (ablation) | GLaD-TESLA · ImageNet-[A, B, C, D, E] | accuracy: 51.9±1.3 -> 53.5±0.7 | Helped |
| add average latent initialization (ablation) | GLaD-TESLA · ImNet-Fruits | accuracy: 23.1±0.4 -> 22.7±1.2↓ | Hurt |
| add average latent initialization (ablation) | GLaD-DSA · ImNet-B | accuracy: 49.2±1.1 -> 48.9±0.8↓ | Hurt |
| add average latent initialization (ablation) | GLaD-DSA · ImNet-C | accuracy: 42.0±0.6 -> 40.6±0.7↓ | Hurt |
| add hierarchical feature layers (ablation) | GLaD with TESLA · ImageNet subsets | accuracy: 53.5±0.7 -> 56.2±0.7 | Helped |
| add hierarchical feature layers (ablation) | GLaD with DSA · ImageNet subsets | accuracy: 48.9±0.8 -> 50.1±1.1 | Helped |
| add hierarchical feature layers (ablation) | GLaD with DM · ImageNet subsets | accuracy: 43.2±1.6 -> 44.2±2.1 | Helped |
| add class-relevant feature distance (ablation) | H-PD with TESLA · ImageNet subsets | accuracy: 56.2±0.7 -> 57.4±0.3 | Helped |
| add class-relevant feature distance (ablation) | H-PD with DSA · ImageNet subsets | accuracy: 50.1±1.1 -> 50.7±0.9 | Helped |
| add class-relevant feature distance (ablation) | H-PD with DM · ImageNet subsets | accuracy: 44.2±2.1 -> 44.7±1.3 | Helped |
Surprising findings
- Loss value was a poor substitute for real performance when selecting latents, sometimes performing worse than no search; the class-relevant feature distance correlated more strongly with accuracy.
- Average-noise latent initialization decreased performance for DSA on some subsets, attributed to DSA's inclination toward noise and edge samples.
Limitations
- Gains on same-architecture evaluation are not uniform; some configurations show small or negative changes (e.g., ablation initialization hurting DSA).
- Method relies on a pretrained StyleGAN-XL generator and class-relevant CAM distance, limiting applicability beyond GAN-based parameterization.
- Some comparisons use results copied from original papers and mismatched settings, reducing direct comparability.
Experimental setup
Datasets: ImageNet-1K CIFAR-10 Tiny-ImageNet ImageNette ImageWoof
Benchmarks: ImageNet-Subset (ImNet-A to ImNet-E, ImNette, ImWoof, ImNet-Birds, ImNet-Fruits, ImNet-Cats) CIFAR-10 Tiny-ImageNet ImageNet-1K
Metrics: test accuracy (%) time complexity (min)
Baselines: GLaD ITGAN FreD Minimax D4M SRe2L TESLA DSA DM
Builds on: GLaD MTT DSA DM TESLA
Closest papers by meaning
- Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximation (ICLR 2025 · similarity 0.88)
- Data Distillation Can Be Like Vodka: Distilling More Times For Better Quality (ICLR 2024 · similarity 0.87)
- Large Scale Dataset Distillation with Domain Shift (ICML 2024 · similarity 0.87)
- Hybrid Data-Free Knowledge Distillation (AAAI 2025 · similarity 0.87)
- Going Beyond Feature Similarity: Effective Dataset distillation based on Class-aware Conditional Mutual Information (ICLR 2025 · similarity 0.87)
- Unlocking Dataset Distillation with Diffusion Models (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.