Research map / Label-efficient and robust learning / Knowledge distillation
SelMatch: Effectively Scaling Up Dataset Distillation via Selection-Based Initialization and Partial Updates by Trajectory Matching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SelMatch, which combines a sliding-window selection of difficulty-tuned real samples as initialization with partial trajectory-matching updates, to scale dataset distillation to larger IPC on CIFAR-10/100 and Tiny ImageNet.
Problem
Trajectory-matching dataset distillation methods lose effectiveness and can underperform random selection as images per class (IPC) grows, because they under-cover hard samples.
Approach
Initialize the synthetic set with a window of real samples at an IPC-appropriate difficulty level chosen by a sliding-window search over difficulty scores, then update only a fraction of the set via trajectory matching while freezing the rest, and apply DSA only to the updated portion.
Key results
- On CIFAR-100 at IPC=50 (10% subset), SelMatch reaches 54.5% vs. 51.0% for DATM (the next best), a 3.5-point gain.
- SelMatch achieves the best accuracy in all reported settings, e.g., 82.8% on CIFAR-10 at IPC=250 (5%) vs. 80.7% for MTT.
- Selection-based initialization alone raises MTT on CIFAR-100 IPC=100 from 58.7% to 61.0%; partial updates and combined augmentation add further gains to 62.4%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SelMatch instead of the strongest reported baseline at each subset ratio | CIFAR-10 · CIFAR-10, subset ratios 5%–30% | test accuracy: 80.7±0.4 -> 82.8±0.2; 85.2±0.1 -> 85.9±0.2; 89.9±0.5 -> 90.4±0.2; 90.6±0.3 -> 91.3±0.2 | Helped |
| Use SelMatch instead of the strongest reported baseline at each subset ratio | CIFAR-100 · CIFAR-100, subset ratios 5%–30% | test accuracy: 49.9±0.7 -> 50.9±0.3; 51.3±0.4 -> 54.5±0.6; 61.5±0.3 -> 62.4±0.6; 64.7±0.5 -> 67.4±0.2 | Helped |
| Use SelMatch instead of oracle-window selection | Tiny ImageNet · Tiny ImageNet, subset ratios 10%–20% | test accuracy: 42.5±0.3 -> 44.7±0.2; 49.2±0.3 -> 50.4±0.2 | Helped |
| Add selection-based initialization to MTT (ablation) | MTT · CIFAR-100, IPC 100 | accuracy: 58.7 -> 61.0 | Helped |
| Add partial updates after selection-based initialization (ablation) | SelMatch without combined augmentation · CIFAR-100, IPC 100 | accuracy: 61.0 -> 61.5 | Helped |
| Add combined augmentation (ablation) | SelMatch with selection initialization and partial updates · CIFAR-100, IPC 100 | accuracy: 61.5 -> 62.4 | Helped |
| Distill selected and updated samples separately, then merge them (ablation) | MTT with oracle-window initialization · CIFAR-100, IPC 50 | accuracy: 51.3 -> 53.1 | Helped |
| Use joint partial updates instead of post-distillation merging (ablation) | Merge · CIFAR-100, IPC 50 | accuracy: 53.1 -> 54.5 | Helped |
| Replace C-score with Forgetting score (ablation) | SelMatch · CIFAR-100, IPC 50 and 100 | accuracy: 54.5 -> 54.2; 62.4 -> 62.9 | No clear effect |
| Replace C-score with EL2N (ablation) | SelMatch · CIFAR-100, IPC 50 and 100 | accuracy: 54.5 -> 52.9; 62.4 -> 61.6 | Hurt |
| Add selection-based initialization to DSA (ablation) | DSA · CIFAR-100, IPC 25 and 50 | accuracy: 38.3 -> 43.4; 43.6 -> 49.8 | Helped |
| Add partial updates after selection-based initialization (ablation) | DSA with Select Init · CIFAR-100, IPC 25 and 50 | accuracy: 43.4 -> 45.0; 49.8 -> 50.8 | Helped |
Surprising findings
- Oracle window selection (no distillation) outperforms all distillation baselines except SelMatch in every tested ratio on CIFAR-10 and Tiny ImageNet.
- Optimal distillation portion alpha decreases as IPC increases, i.e., updating fewer samples works better at larger IPC.
Limitations
- Requires pre-computed per-sample difficulty scores (C-score or Forgetting), which are computationally expensive to obtain.
- Hyperparameters alpha and beta must be tuned per dataset and IPC, adding computational overhead.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny ImageNet
Benchmarks: CIFAR-10 CIFAR-100 Tiny ImageNet
Metrics: test accuracy
Baselines: Random Forgetting Glister Oracle window DSA DM MTT FTD DATM
Builds on: Matching Training Trajectories (MTT) DATM C-score Differentiable Siamese Augmentation
Closest papers by meaning
- Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matching (ICLR 2024 · similarity 0.89)
- DELT: A Simple Diversity-driven EarlyLate Training for Dataset Distillation (CVPR 2025 · similarity 0.88)
- Rethinking Dataset Distillation for Classification: Do Distilled Sets Outperform Coresets? (NeurIPS 2026 · similarity 0.88)
- Curriculum Coarse-to-Fine Selection for High-IPC Dataset Distillation (CVPR 2025 · similarity 0.87)
- Emphasizing Discriminative Features for Dataset Distillation in Complex Scenarios (CVPR 2025 · similarity 0.87)
- Efficient Dataset Distillation for Pre-Trained Self-Supervised Models via Statistical Flow Matching (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.