Research map / Label-efficient and robust learning / Recognition and long-tail
Exploiting Inter-sample and Inter-feature Relations in Dataset Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adds a class centralization loss and a local-covariance matching loss as plug-and-play constraints on top of distribution matching dataset distillation (DM and IDM), improving CIFAR10/100, SVHN and TinyImageNet accuracy and cross-architecture robustness.
Problem
Distribution matching-based dataset distillation produces synthetic datasets whose same-class features are dispersed, and it matches only mean features, ignoring inter-feature covariance structure, which is poorly estimated when samples per class are fewer than feature dimensions.
Approach
Adds a class centralization loss that pulls synthetic same-class features toward their class center, computed with a pre-trained ResNet18, and a covariance matching loss that aligns class-wise covariance matrices of local feature maps (reshaped to d x hw) between real and synthetic data, as plug-ins to DM and IDM.
Key results
- DM+Ours reaches 55.1% on CIFAR10 at IPC=10 vs 48.5% for DM baseline (+6.6%).
- IDM+Ours reaches 69.0% on CIFAR10 at IPC=50 and 45.7% on CIFAR100 at IPC=10.
- Cross-architecture IDM+Ours on CIFAR10 IPC=50 stays between 67.3% and 69.0% across ConvNet, AlexNet, VGG11, ResNet18 (drop under 1.7%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add class centralization to DM (ablation) | DM · CIFAR10 | accuracy: 48.5±0.2 -> 52.2±0.3 | Helped |
| Add covariance matching to DM (ablation) | DM · CIFAR10 | accuracy: 48.5±0.2 -> 51.6±0.1 | Helped |
| Add both class centralization and covariance matching to DM (ablation) | DM · CIFAR10 | accuracy: 48.5±0.2 -> 55.1±0.1 | Helped |
| Add class centralization to DM (ablation) | DM · CIFAR100 | accuracy: 29.7±0.1 -> 31.1±0.2 | Helped |
| Add covariance matching to DM (ablation) | DM · CIFAR100 | accuracy: 29.7±0.1 -> 31.2±0.1 | Helped |
| Add both class centralization and covariance matching to DM (ablation) | DM · CIFAR100 | accuracy: 29.7±0.1 -> 32.2±0.3 | Helped |
| Increase the class-centralization threshold β (ablation) | DM with class centralization, CIFAR10 IPC=10 · CIFAR10 | accuracy: 52.2±0.1 -> 48.9±0.1 | Hurt |
| Vary λCC from 0.01 to 0.09 (ablation) | DM with proposed constraints, CIFAR10 · CIFAR10 | accuracy: 1.47% | Mixed |
| Vary λCM from 0.005 to 0.015 (ablation) | DM with proposed constraints, CIFAR10 · CIFAR10 | accuracy: 1.1% | Mixed |
| Use the proposed constraints with DM instead of the DM baseline | DM · CIFAR10 IPC=10 | accuracy: 48.5±0.6 -> 55.1±0.1 | Helped |
| Use the proposed constraints with IDM instead of the IDM baseline | IDM · CIFAR10 IPC=10 | accuracy: 57.3±0.3 -> 59.9±0.2 | Helped |
| Use the proposed constraints with IDM instead of the IDM baseline | IDM · CIFAR100 IPC=10 | accuracy: 44.7±0.1 -> 45.7±0.4 | Helped |
Surprising findings
- The class centralization constraint uses a pre-trained ResNet18 as feature extractor, distinct from the ConvNet used in the distillation objective, and this helps cross-architecture generalization.
- The method converges in under about 2,000-3,000 iterations, versus around 20,000 for prior methods.
Limitations
- DM+Ours cannot produce IPC=1 results because the covariance estimate requires more than one sample per class.
- Sensitivity to the loss weights lambda_CC and lambda_CM and to the centralization threshold beta is moderate (about 1.5% and 1.1% variation).
- Experiments are limited to small image datasets up to 64x64 resolution.
Experimental setup
Datasets: SVHN CIFAR10 CIFAR100 TinyImageNet
Benchmarks: SVHN CIFAR10 CIFAR100 TinyImageNet
Metrics: Top-1 accuracy
Baselines: Random Herding K-Center Forgetting DD LD DC DSA KIP CAFE DCC DataDAM
Builds on: DM IDM DSA
Closest papers by meaning
- Breaking Class Barriers: Efficient Dataset Distillation via Inter-Class Feature Compensator (ICLR 2025 · similarity 0.89)
- Going Beyond Feature Similarity: Effective Dataset distillation based on Class-aware Conditional Mutual Information (ICLR 2025 · similarity 0.88)
- Optimizing Distributional Geometry Alignment with Optimal Transport for Generative Dataset Distillation (NeurIPS 2025 · similarity 0.87)
- Dataset Distillation for Pre-Trained Self-Supervised Vision Models (NeurIPS 2025 · similarity 0.87)
- Enhancing Dataset Distillation via Non-Critical Region Refinement (CVPR 2025 · similarity 0.86)
- Efficient Dataset Distillation for Pre-Trained Self-Supervised Models via Statistical Flow Matching (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.