Research map / Architectures and efficiency / RNNs and normalization
Dataset Distillation with Neural Characteristic Function: A Minmax Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reformulates dataset distillation via distribution matching as a minmax problem, using a learned neural characteristic function discrepancy (NCFD) that aligns phase and amplitude of features, achieving SOTA accuracy with much lower GPU memory.
Problem
Existing distribution-matching dataset distillation relies on MSE or MMD distances that fail to capture full distributional differences between real and synthetic data, yielding unreliable discrepancy measures.
Approach
Replace fixed MSE/MMD metrics with a characteristic-function discrepancy whose frequency arguments are sampled from a learned scale-mixture-of-normals distribution; a sampling network maximizes this discrepancy while synthetic data minimizes it (minmax), with the metric decomposed into amplitude and phase terms.
Key results
- NCFM reaches 71.8% on CIFAR-10 at 10 IPC and 78.3% at 50 IPC with a 3-layer ConvNet, outperforming DATM (66.8%, 76.1%).
- On ImageSquawk at 10 IPC, NCFM improves accuracy by 22.9% over the MTT baseline.
- NCFM uses about 1.9 GB peak GPU memory on CIFAR-100 at 50 IPC and runs about 20x faster than TESLA, where DATM and MTT run out of memory.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use NCFM instead of the strongest listed baseline | dataset distillation · CIFAR-10, 1 IPC | accuracy (%): 48.3±1.0 -> 50.5±0.3 | Helped |
| Use NCFM instead of the strongest listed baseline | dataset distillation · CIFAR-100, 1 IPC | accuracy (%): 28.7±0.1 -> 34.4±0.5 | Helped |
| Use NCFM instead of the strongest listed baseline | dataset distillation · CIFAR-100, 50 IPC | accuracy (%): 57.0±0.1 -> 55.1±0.2 | Hurt |
| Use NCFM instead of the strongest listed baseline | dataset distillation · Tiny ImageNet, 10 IPC | accuracy (%): 31.1±0.3 -> 27.4±0.6 | Hurt |
| Use NCFM instead of the strongest listed baseline | dataset distillation · Tiny ImageNet, 50 IPC | accuracy (%): 39.7±0.3 -> 29.8±0.5 | Hurt |
| Use NCFM instead of the strongest listed baseline | dataset distillation · ImageNette, 10 IPC | accuracy (%): 63.2±0.7 -> 80.2±1.0 | Helped |
| Use NCFM instead of MTT | dataset distillation · ImageSquawk, 10 IPC | accuracy (%): 52.3±1.0 -> 75.2±0.9 | Helped |
| Add sampling network ψ (ablation) | NCFM · CIFAR-10, 50 IPC | accuracy (%): 74.2 -> 77.4 | Helped |
| Add sampling network ψ (ablation) | NCFM · CIFAR-100, 10 IPC | accuracy (%): 45.9 -> 48.7 | Helped |
| Add sampling network ψ (ablation) | NCFM · Tiny ImageNet, 10 IPC | accuracy (%): 14.2 -> 24.3 | Helped |
| Add sampling network ψ (ablation) | NCFM · ImageMeow, 10 IPC | accuracy (%): 51.6 -> 54.4 | Helped |
| Balance amplitude and phase components rather than letting either dominate (ablation) | NCFM · CIFAR-10, 10 IPC | accuracy (%): null | Helped |
Surprising findings
- Adding the learned sampling network improves accuracy only modestly on some settings (e.g., +3.2% on CIFAR-10 at 50 IPC) but is reported as essential to the minmax paradigm.
- Amplitude-dominated or phase-dominated loss weightings both reduce accuracy, showing a balance is needed.
Limitations
- Results on high-resolution ImageNet subsets are reported mainly at 1 and 10 IPC, with less coverage at higher IPC.
- The additional data-curation steps (soft labels, fine-tuning) are optional and not used in the main pure-minmax results, so comparisons with soft-label methods mix settings.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny ImageNet ImageNette ImageWoof ImageFruit ImageMeow ImageSquawk ImageYellow
Benchmarks: CIFAR-10 CIFAR-100 Tiny ImageNet ImageNet subsets DD-Ranking
Metrics: test accuracy training speed (s/iter) peak GPU memory (GB) DD-Ranking Score
Baselines: DC DSA DCC DSAC FrePo MTT ATT FTD TESLA DATM CAFE DM
Builds on: distribution matching maximum mean discrepancy IDM DANCE
Closest papers by meaning
- Hyperbolic Dataset Distillation (NeurIPS 2025 · similarity 0.87)
- Large Scale Dataset Distillation with Domain Shift (ICML 2024 · similarity 0.87)
- Going Beyond Feature Similarity: Effective Dataset distillation based on Class-aware Conditional Mutual Information (ICLR 2025 · similarity 0.86)
- OPTICAL: Leveraging Optimal Transport for Contribution Allocation in Dataset Distillation (CVPR 2025 · similarity 0.86)
- D^4: Dataset Distillation via Disentangled Diffusion Model (CVPR 2024 · similarity 0.86)
- Efficient Dataset Distillation for Pre-Trained Self-Supervised Models via Statistical Flow Matching (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.