Research map / Label-efficient and robust learning / Recognition and long-tail
M3D: Dataset Condensation by Minimizing Maximum Mean Discrepancy
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes M3D, a distribution-matching dataset condensation method that minimizes the kernel-based maximum mean discrepancy between real and synthetic feature distributions to align all moment orders, matching or beating optimization-oriented SOTA at much lower cost.
Problem
Dataset condensation: existing distribution-matching methods only align first-order (mean) feature statistics, leaving them less informative than bi-level optimization methods, while optimization methods are too expensive for large datasets like ImageNet subsets.
Approach
Embed real and synthetic feature distributions in an RKHS via a Gaussian kernel and minimize the empirical MMD^2 loss, which implicitly aligns all orders of moments; combined with factor/up-sampling and multiple iterations per randomly initialized model.
Key results
- On CIFAR-10 with IPC=10, M3D reaches 63.5% test accuracy vs. 48.9% for DM and 58.6% for IDM.
- On ImageNet-10 with IPC=20, M3D achieves 76.8% in 3.1h, surpassing IDC (76.6%, 92.8h).
- Adding the M3D loss to DM and IDM improves CIFAR-10 IPC=10 accuracy by 5.8 and 1.9 points respectively.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use M3D instead of the strongest reported baseline | dataset condensation with a 3-layer ConvNet · SVHN, IPC=10 | test accuracy: 79.2 -> 83.3 | Helped |
| use M3D instead of the strongest reported baseline | dataset condensation with a 3-layer ConvNet · CIFAR-10, IPC=10 | test accuracy: 58.6 -> 63.5 | Helped |
| use M3D instead of the strongest reported baseline | dataset condensation with a 3-layer ConvNet · CIFAR-100, IPC=10 | test accuracy: 45.1 -> 42.4 | Hurt |
| use M3D instead of IDC | ImageNet-10 condensation · ImageNet-10, IPC=10 | accuracy: 72.8 -> 73.4 | Helped |
| add 2nd-order moment regularization to DM (ablation) | DM on CIFAR-10, IPC=10 · CIFAR-10, IPC=10 | test accuracy: 48.9 -> 52.1 | Helped |
| add 2nd- and 3rd-order moment regularization to DM (ablation) | DM on CIFAR-10, IPC=10 · CIFAR-10, IPC=10 | test accuracy: 48.9 -> 53.9 | Helped |
| replace DM loss with M3D loss (ablation) | DM on CIFAR-10, IPC=10 · CIFAR-10, IPC=10 | test accuracy: 48.9 -> 54.7 | Helped |
| replace DM loss with M3D loss (ablation) | DM on CIFAR-10, IPC=50 · CIFAR-10, IPC=50 | test accuracy: 63.0 -> 66.0 | Helped |
| replace IDM loss with M3D loss (ablation) | IDM on CIFAR-10, IPC=10 · CIFAR-10, IPC=10 | test accuracy: 58.6 -> 60.5 | Helped |
| replace IDM loss with M3D loss (ablation) | IDM on CIFAR-10, IPC=50 · CIFAR-10, IPC=50 | test accuracy: 67.5 -> 69.6 | Helped |
| increase iterations per model (ablation) | M3D · CIFAR-10, IPC=10 | test accuracy | Mixed |
| change the kernel function among Gaussian, linear, and polynomial (ablation) | M3D · CIFAR-10, IPC=10 | test accuracy | No clear effect |
Surprising findings
- The choice of kernel (Gaussian, linear, polynomial) has minimal impact on condensed-set performance.
Limitations
- Results on ImageNet subsets are reported only for ResNetAP-10 and for a few subsets; scaling to full ImageNet is not shown.
- Most of the claimed gains depend on combining with the factor technique and multiple iterations per model, which add hyperparameters.
- Some benchmark entries are reproduced by the authors or missing, and M3D does not win every setting (e.g., CIFAR-100 IPC=10 and MNIST IPC=10/50 against DSA or IDM).
Experimental setup
Datasets: MNIST Fashion-MNIST SVHN CIFAR-10 CIFAR-100 ImageNet-10 ImageNet-100
Benchmarks: MNIST Fashion-MNIST SVHN CIFAR-10 CIFAR-100 ImageNet-10 ImageNet-100
Metrics: test accuracy condensation time (h)
Baselines: Random Herding K-Center DC DSA IDC CAFE CAFE+DSA DM IDM
Builds on: Distribution Matching (DM) IDC Reproducing Kernel Hilbert Space DSA
Closest papers by meaning
- Elucidating the Design Space of Dataset Condensation (NeurIPS 2024 · similarity 0.88)
- Generalized Large-Scale Data Condensation via Various Backbone and Statistical Matching (CVPR 2024 · similarity 0.85)
- Dataset Distillation for Pre-Trained Self-Supervised Vision Models (NeurIPS 2025 · similarity 0.85)
- Real-Fake: Effective Training Data Synthesis Through Distribution Matching (ICLR 2024 · similarity 0.85)
- DELT: A Simple Diversity-driven EarlyLate Training for Dataset Distillation (CVPR 2025 · similarity 0.85)
- Unlocking Dataset Distillation with Diffusion Models (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.