Research map / Label-efficient and robust learning / Knowledge distillation
Curriculum Coarse-to-Fine Selection for High-IPC Dataset Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CCFS, a curriculum coarse-to-fine selection method that adds real images to a distilled set over several phases, choosing each phase's real data based on the current synthetic set, for high-IPC dataset distillation.
Problem
Dataset distillation loses effectiveness as images per class (IPC) grows; combination-based methods like SelMatch use one-shot, independent selection of real data, causing incompatibility between distilled and real images.
Approach
Treat real-data selection as a curriculum: starting from the distilled set, repeatedly train a filter on the current synthetic set, keep only misclassified real samples (coarse), then choose the easiest of them by difficulty score (fine), so each added batch is dependent on the current synthetic set.
Key results
- CCFS reaches 92.5% on CIFAR-10 (10% ratio), 71.5% on CIFAR-100 (10%), and 60.2% on Tiny-ImageNet (20%), beating prior SOTA by 6.6, 5.8 and 3.4 points.
- On Tiny-ImageNet at 20% compression, CCFS is 0.3% below full-dataset training (60.5%).
- Ablation on CIFAR-100 (IPC=50) shows coarse misclassified filtering plus fine simplest-first selection gives 71.5%, above other coarse/fine combinations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply curriculum coarse-to-fine selection with CDA initialization | CDA-based dataset distillation · CIFAR-10, IPC=500 (10%) | top-1 test accuracy: 85.9±0.2 -> 92.5±0.2 | Helped |
| Apply curriculum coarse-to-fine selection with CDA initialization | CDA-based dataset distillation · CIFAR-100, IPC=50 (10%) | top-1 test accuracy: 65.7±0.2 -> 71.5±0.3 | Helped |
| Apply curriculum coarse-to-fine selection with CDA initialization | CDA-based dataset distillation · Tiny-ImageNet, IPC=100 (20%) | top-1 test accuracy: 56.8±0.2 -> 60.2±0.2 | Helped |
| Select simplest misclassified samples rather than correctly classified samples (ablation) | CCFS · CIFAR-100, IPC=50 | accuracy: 66.8 -> 71.5 | Helped |
| Select simplest rather than hardest misclassified samples (ablation) | CCFS · CIFAR-100, IPC=50 | accuracy: 65.0 -> 71.5 | Helped |
| Select simplest rather than random misclassified samples (ablation) | CCFS · CIFAR-100, IPC=50 | accuracy: 70.1 -> 71.5 | Helped |
| Replace Forgetting difficulty score with logits (ablation) | CCFS · CIFAR-10, CIFAR-100, and Tiny-ImageNet, 10% compression | accuracy: 91.8, 68.7, 52.5 vs 92.5, 71.5, 55.8 | Hurt |
| Replace Forgetting difficulty score with C-score (ablation) | CCFS · CIFAR-10 and CIFAR-100, 10% compression | accuracy: 92.2, 71.0 vs 92.5, 71.5 | Hurt |
| Use 1 curriculum instead of 3 (ablation) | CCFS · CIFAR-10, CIFAR-100, and Tiny-ImageNet, 10% compression | accuracy: 91.6, 67.9, 54.4 vs 92.5, 71.5, 55.8 | Hurt |
| Use 2 curricula instead of 3 (ablation) | CCFS · CIFAR-10, CIFAR-100, and Tiny-ImageNet, 10% compression | accuracy: 91.8, 70.4, 55.3 vs 92.5, 71.5, 55.8 | Hurt |
| Use 4 curricula instead of 3 (ablation) | CCFS · CIFAR-10, CIFAR-100, and Tiny-ImageNet, 10% compression | accuracy: 92.4, 71.6, 55.7 vs 92.5, 71.5, 55.8 | Mixed |
Surprising findings
- Reversing SelMatch's order (distill first, then select real data) performs better than SelMatch's one-shot independent selection.
Limitations
- Gains flatten beyond 3 curricula, so the number of phases needs tuning.
- Relies on precomputed difficulty scores (Forgetting) and a filter model trained per phase, adding cost.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet
Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet
Metrics: top-1 test accuracy
Baselines: SelMatch DATM CDA CUDD MTT SRe2L DSA DM Glister Forgetting Oracle window
Builds on: SelMatch MTT CDA Forgetting score
Closest papers by meaning
- Rethinking Dataset Distillation for Classification: Do Distilled Sets Outperform Coresets? (NeurIPS 2026 · similarity 0.88)
- SelMatch: Effectively Scaling Up Dataset Distillation via Selection-Based Initialization and Partial Updates by Trajectory Matching (ICML 2024 · similarity 0.87)
- Emphasizing Discriminative Features for Dataset Distillation in Complex Scenarios (CVPR 2025 · similarity 0.86)
- Large Scale Dataset Distillation with Domain Shift (ICML 2024 · similarity 0.86)
- DD-Ranking: Rethinking the Evaluation of Dataset Distillation (NeurIPS 2026 · similarity 0.85)
- Foundation Model Insights and a Multi-Model Approach for Superior Fine-Grained One-shot Subset Selection (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.