Research map / Label-efficient and robust learning / Knowledge distillation
Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DATM, a trajectory-matching dataset distillation method that aligns the difficulty of matched expert trajectory segments with synthetic set size, plus learnable soft labels and sequential generation, reaching lossless distillation on CIFAR-10/100 and Tiny ImageNet with ConvNet.
Problem
Dataset distillation methods become ineffective as the synthetic set size (IPC) grows, often performing worse than random selection, so no method has matched full-dataset performance (lossless distillation).
Approach
Early expert trajectories induce easy patterns that suit small synthetic sets, while late trajectories induce hard patterns that suit large ones; DATM restricts the matching range to a difficulty-aligned window that expands over distillation, and learns soft labels initialized from correctly classified samples with sequential generation for stability.
Key results
- DATM distills CIFAR-10 to IPC=500 (10x reduction, ratio 1/5 at IPC=1000 per paper) with 85.5% accuracy at IPC=1000, matching full-dataset 84.8% with ConvNet, i.e. lossless.
- On CIFAR-100 at IPC=100, DATM reaches 57.5% vs full-dataset 56.2%, and on Tiny ImageNet at IPC=50 it reaches 39.7% vs full-dataset 37.6%.
- Previous TM-based methods such as MTT and FTD fall below random selection at high IPC on CIFAR-10 (e.g., FTD 73.2% vs DATM 76.1% at IPC=50).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DATM instead of the strongest same-architecture baseline | ConvNet dataset distillation · CIFAR-100, IPC=50 | accuracy: 48.5±0.3 -> 55.0±0.2 | Helped |
| Add difficulty alignment (ablation) | DATM with soft labels · CIFAR-100, IPC=50 | accuracy: 52.96 -> 55.03 | Helped |
| Add soft labels (ablation) | DATM with difficulty alignment · CIFAR-100, IPC=50 | accuracy: 50.79 -> 55.03 | Helped |
| Add label learning (ablation) | DATM without sequential generation · CIFAR-10, IPC=50 | accuracy: 72.8 -> 75.0 | Helped |
| Add sequential generation (ablation) | DATM without label learning · CIFAR-10, IPC=50 | accuracy: 72.8 -> 75.6 | Helped |
| Add sequential generation to label learning (ablation) | DATM · CIFAR-10, IPC=50 | accuracy: 75.0 -> 76.1 | Helped |
| Match early rather than late expert trajectories | Trajectory matching with low IPC · CIFAR-10, low IPC | accuracy | Helped |
| Match late rather than early expert trajectories | Trajectory matching with high IPC · CIFAR-10, high IPC | accuracy | Helped |
| Assign soft labels to FTD distilled data (ablation) | FTD · CIFAR-10, IPC=1 and IPC=50 | accuracy: 46.0 -> 45.5; 73.2 -> 72.8 | Mixed |
| Assign soft labels to FTD distilled data (ablation) | FTD · CIFAR-100, IPC=10 and IPC=50 | accuracy: 42.5 -> 44.5; 48.5 -> 51.2 | Helped |
| Initialize synthetic data from correctly classified real samples (ablation) | DATM · CIFAR-100, IPC=10 | optimization convergence speed | Helped |
| Optimize soft labels during distillation instead of keeping them fixed (ablation) | DATM · Dataset distillation | matching loss | Helped |
Surprising findings
- Matching early trajectories performs worse than random selection at high IPC, since synthetic images collapse toward mean patterns.
- Distillation cost does not grow linearly with IPC; IPC=1000 costs only about 1.05x IPC=50 because late-trajectory matching converges faster.
Limitations
- Distilled data is lossless only for the distillation backbone (ConvNet); cross-architecture performance still drops.
- TM-based methods are costly and hard to scale to large datasets.
- Heavy GPU memory reliance for large synthetic steps.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny ImageNet
Benchmarks: CIFAR-10 CIFAR-100 Tiny ImageNet
Metrics: test accuracy
Baselines: Random DC DM DSA CAFE KIP FRePo RCIG MTT TESLA FTD Glister
Builds on: MTT FTD dataset pruning difficulty findings
Closest papers by meaning
- Embarrassingly Simple Dataset Distillation (ICLR 2024 · similarity 0.89)
- Towards Stable and Storage-efficient Dataset Distillation: Matching Convexified Trajectory (CVPR 2025 · similarity 0.89)
- SelMatch: Effectively Scaling Up Dataset Distillation via Selection-Based Initialization and Partial Updates by Trajectory Matching (ICML 2024 · similarity 0.89)
- Path-Guided Flow Matching for Dataset Distillation (NeurIPS 2026 · similarity 0.89)
- Data Distillation Can Be Like Vodka: Distilling More Times For Better Quality (ICLR 2024 · similarity 0.87)
- Distilling Long-tailed Datasets (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.