Research map / Label-efficient and robust learning / Knowledge distillation
Going Beyond Feature Similarity: Effective Dataset distillation based on Class-aware Conditional Mutual Information
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adds a class-aware conditional mutual information (CMI) regularizer, computed in the feature space of pre-trained networks, to existing dataset distillation losses to reduce synthetic-data complexity and improve accuracy and efficiency.
Problem
Dataset distillation methods that match feature or gradient statistics often produce synthetic datasets that are too complex for networks to learn from, hurting performance and training efficiency.
Approach
Measure class-aware complexity of the synthetic set via empirical conditional mutual information (KL divergence of per-sample softmax-normalized features from the class centroid) computed on pre-trained networks, and add it as a plug-and-play regularizer to any DD loss.
Key results
- CMI added to DSA improves CIFAR-10 IPC=50 accuracy by 5.5% (60.6 to 66.1).
- IDC+CMI reaches 88.5% on SVHN IPC=10 (vs 87.5 for IDC) and 70.0% on CIFAR-10 IPC=10 (vs 67.5).
- SRe2L+CMI improves ImageNet-1K IPC=10 from 21.3% to 24.2%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the CMI constraint to IDC | IDC · SVHN, IPC=10 and 50 | Top-1 accuracy: (1.0↑); (2.1↑) | Helped |
| Add the CMI constraint to IDC | IDC · CIFAR10, IPC=10 and 50 | Top-1 accuracy: (2.5↑); (2.1↑) | Helped |
| Add the CMI constraint to IDC | IDC · CIFAR100, IPC=10 and 50 | Top-1 accuracy: (1.5↑) | Mixed |
| Add the CMI constraint to SRe²L | SRe²L · ImageNet-1K, IPC=10, 50, and 100 | Top-1 accuracy: (2.9↑); (2.3↑); (1.8↑) | Helped |
| Use a pre-trained surrogate rather than no pre-training (ablation) | CMI-regularized IDM · CIFAR10, IPC=10, ConvNet evaluation | Top-1 accuracy: 67.3$\pm$0.2 -> 68.7$\pm$0.3 | Helped |
| Compute CMI in feature space rather than probability space (ablation) | CMI-regularized IDM · CIFAR10, IPC=10, ConvNet evaluation | Top-1 accuracy: 68.1$\pm$0.2 -> 68.7$\pm$0.3 | Helped |
| Compute CMI in feature space rather than probability space (ablation) | CMI-regularized IDM · CIFAR10, IPC=10, ResNet18 evaluation | Top-1 accuracy: 67.8$\pm$0.3 -> 70.0$\pm$0.3 | Helped |
| Increase λ from 500 to 5000 (ablation) | CMI-regularized IDM · CIFAR10, IPC=10 | Top-1 accuracy: 1.5% | Mixed |
| Use different pre-trained proxy architectures for CMI (ablation) | DM and DSA · CIFAR10, IPC=10 | Top-1 accuracy | Mixed |
| Add the CMI constraint to IDC | IDC · Tiny-ImageNet, IPC=10 and 50 | Top-1 accuracy: (1.2↑); (1.1↑) | Helped |
| Add CMI to IDM | IDM · CIFAR10, IPC=1 | Top-1 accuracy: (1.9↑) | Helped |
Surprising findings
- CMI is zero when IPC=1, so the regularizer cannot be applied in the one-image-per-class setting.
- Visually, the CMI constraint causes almost imperceptible changes to the distilled images despite large accuracy gains.
Limitations
- Requires multiple pre-trained proxy teacher networks, adding memory and compute overhead.
- Proxy architecture choice is heuristic; the CMI value is zero at IPC=1 so the method is not applicable there.
- Hyperparameter lambda must be tuned per base DD loss because CMI scale is small relative to the distillation loss.
Experimental setup
Datasets: SVHN CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNet-1K
Benchmarks: SVHN CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNet-1K
Metrics: top-1 accuracy training time per iteration CMI value
Baselines: MTT MIM4DD SeqMatch DM IID-DM IDM IID-IDM DSA IDC DREAM PDD SRe2L
Builds on: distribution matching IDC IDM DSA MTT IID SRe2L
Closest papers by meaning
- Enhancing Dataset Distillation via Non-Critical Region Refinement (CVPR 2025 · similarity 0.89)
- Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-training of Deep Networks (ICLR 2025 · similarity 0.88)
- Exploiting Inter-sample and Inter-feature Relations in Dataset Distillation (CVPR 2024 · similarity 0.88)
- Data Distillation Can Be Like Vodka: Distilling More Times For Better Quality (ICLR 2024 · similarity 0.88)
- Rethinking Dataset Distillation for Classification: Do Distilled Sets Outperform Coresets? (NeurIPS 2026 · similarity 0.87)
- Large Scale Dataset Distillation with Domain Shift (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.