Research map / Label-efficient and robust learning / Recognition and long-tail
Enhancing Dataset Distillation via Non-Critical Region Refinement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NRR-DD, a dataset distillation method that keeps CAM-identified critical regions of synthetic images fixed while refining non-critical regions with class-general information, plus a distance-based relabeling (DBR) that stores two scalars per instance instead of soft labels.
Problem
Dataset distillation methods tend to capture either class-general patterns or instance-specific fine-grained details but not both, and soft-label knowledge transfer for large-scale datasets requires prohibitive storage.
Approach
Select low-confidence, high-CAM patches to form instance-specific initial images, then update only non-critical pixels (weighted by a CAM-derived mask) using CE and BatchNorm losses; replace stored soft labels with two cross-entropy distances per mixed sample (DBR) used to train the student.
Key results
- On ImageNet-1k with ResNet-18 at IPC 10, NRR-DD reaches 60.2% at IPC 50 per the text; the paper reports 4% over RDED and 14% over SRe2L on ImageNet1k.
- On CIFAR-10 with ResNet-18 at IPC 10, NRR-DD achieves 72.2% vs. 37.1% for RDED (per Table 1).
- DBR stores only two distances per instance, giving a 500x storage reduction versus soft labels on ImageNet-1k (200 IPC) while recovering up to 80% of the soft-label gain.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Critical-based Initial Data Discovery (CIDD) rather than RDED data generation (ablation) | NRR-DD · CIFAR10, IPC 10, ConvNet | accuracy: 50.2 ± 0.3 -> 51.3 ± 0.4 | Helped |
| Add Non-Critical Region Refinement to CIDD (ablation) | NRR-DD · CIFAR10, IPC 10, ConvNet | accuracy: 51.3 ± 0.4 -> 66.7 ± 0.4 | Helped |
| Add Non-Critical Region Refinement to CIDD (ablation) | NRR-DD · CIFAR100, IPC 10, ConvNet | accuracy: 50.2 ± 0.3 -> 55.7 ± 0.2 | Helped |
| Add Non-Critical Region Refinement to CIDD (ablation) | NRR-DD · ImageNet1k, IPC 10, ConvNet | accuracy: 21.2 ± 0.3 -> 25.6 ± 0.2 | Helped |
| Add Label Refinement to DBR (ablation) | NRR-DD · ImageNette, IPC 10, ConvNet | accuracy: 52.2 ± 0.4 -> 55.1 ± 0.4 | Helped |
| Add Label Refinement to DBR (ablation) | NRR-DD · ImageNet1k, IPC 10, ConvNet | accuracy: 16.1 ± 0.3 -> 19.2 ± 0.5 | Helped |
| Use NRR-DD instead of RDED | dataset distillation · CIFAR10, IPC 10, ResNet18 | accuracy: 37.1 ± 0.3 -> 72.2 ± 0.4 | Helped |
| Use NRR-DD instead of RDED | dataset distillation · CIFAR100, IPC 10, ResNet18 | accuracy: 42.6 ± 0.2 -> 62.7 ± 0.2 | Helped |
| Use NRR-DD instead of RDED | dataset distillation · Tiny-ImageNet, IPC 10, ConvNet | accuracy: 39.6 ± 0.1 -> 44.3 ± 0.2 | Helped |
| Replace soft labels with DBR (ablation) | NRR-DD · ImageNet1k, IPC 10, ConvNet | accuracy: 25.6 ± 0.2 -> 16.1 ± 0.3 | Hurt |
Surprising findings
- With IPC 1, the DBR method (no soft labels) outperforms RDED with soft labels in all cases, per the paper.
Limitations
- Relies on quality of CAM for identifying critical and non-critical regions, which is sensitive to the pretrained model's training.
- DBR may not fully balance memory efficiency and preservation of fine-grained details for complex tasks.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNette ImageNet-1k
Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNette ImageNet-1k
Metrics: top-1 accuracy storage requirement recover rate
Baselines: MTT IDM TESLA SRe2L RDED
Builds on: RDED SRe2L MTT CutMix Mixup
Closest papers by meaning
- Breaking Class Barriers: Efficient Dataset Distillation via Inter-Class Feature Compensator (ICLR 2025 · similarity 0.91)
- Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximation (ICLR 2025 · similarity 0.89)
- Rethinking Dataset Distillation for Classification: Do Distilled Sets Outperform Coresets? (NeurIPS 2026 · similarity 0.89)
- Going Beyond Feature Similarity: Effective Dataset distillation based on Class-aware Conditional Mutual Information (ICLR 2025 · similarity 0.89)
- Large Scale Dataset Distillation with Domain Shift (ICML 2024 · similarity 0.88)
- Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-training of Deep Networks (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.