Research map / Optimization / Optimizers
Towards Stable and Storage-efficient Dataset Distillation: Matching Convexified Trajectory
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Matching Convexified Trajectory (MCT), which replaces MTT's SGD expert trajectory with a convex-combination (piecewise-linear) trajectory that is cheap to store and supports continuous sampling for dataset distillation.
Problem
Matching Training Trajectories (MTT) dataset distillation suffers from unstable SGD expert trajectories, slow convergence of distillation, and high storage cost of storing all expert checkpoints.
Approach
Reformulates MTT's objective as learning guidance vectors along the expert path, then replaces the oscillating SGD expert trajectory with a convex combination of start and end expert models (with two interpolated points forming a polyline), which only needs a few stored models and per-step constants, and enables continuous real-valued sampling of timesteps during distillation.
Key results
- MCT reaches 48.5% on CIFAR-10 ipc=1 versus 46.3% for MTT, and 72.3% at ipc=50 versus 71.6% for MTT.
- Convexified trajectory requires about 8% of the storage of the MTT expert trajectory.
- MCT reaches peak validation accuracy 200-1,200 iterations earlier than MTT and remains stable after convergence.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace the MTT expert trajectory with MCT's convexified trajectory and continuous sampling | MTT dataset distillation · CIFAR-10 | accuracy: 46.3±0.8 -> 48.5±0.2; 65.3±0.7 -> 66.0±0.3; 71.6±0.2 -> 72.3±0.3 | Helped |
| replace the MTT expert trajectory with MCT's convexified trajectory and continuous sampling | MTT dataset distillation · CIFAR-100 | accuracy: 24.3±0.3 -> 24.5±0.5; 40.1±0.4 -> 42.5±0.5; 47.7±0.2 -> 46.8±0.2 | Mixed |
| replace the MTT expert trajectory with MCT's convexified trajectory and continuous sampling | MTT dataset distillation · Tiny-ImageNet | accuracy: 8.8±0.3 -> 9.6±0.5; 23.2±0.2 -> 22.6±0.8; 28.0±0.3 -> 27.6±0.4 | Mixed |
| add continuous sampling (ablation) | MCT · CIFAR-10, ipc=10 | accuracy: 54.8±0.2 -> 56.2±0.3; 60.6±0.2 -> 61.3±0.5; 61.5±0.3 -> 61.8±0.6; 62.3±0.3 -> 62.8±0.3; 62.1±0.4 -> 62.8±0.2 | Helped |
| vary the expert updating step M (ablation) | MCT · CIFAR-10, ipc=1, 10, and 50 | accuracy: 46.7; 47.1; 48.5; 48.0; 45.6; 62.3; 62.6; 65.0; 66.0; 65.2; 70.0; 71.4; 71.8; 72.3; 71.8 | Mixed |
Surprising findings
- On CIFAR-100 at ipc=10 and ipc=50 and on Tiny-ImageNet at ipc=10 and 50, MCT underperforms MTT (e.g., 42.5 vs 40.1 is better, but 46.8 vs 47.7 and 22.6 vs 23.2 are worse), so gains are not uniform.
Limitations
- Gains over MTT are modest and not consistent across all settings (some Tiny-ImageNet and CIFAR-100 settings are worse).
- Evaluated only on small image datasets with ConvNet surrogate; scalability to LLMs is argued but not tested.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet
Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet
Metrics: validation accuracy convergence iterations storage requirement
Baselines: DC DSA CAFE DM PP MTT Random
Builds on: Matching Training Trajectories (MTT) Dataset Condensation (DC) Differentiable Siamese Augmentation (DSA)
Closest papers by meaning
- Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matching (ICLR 2024 · similarity 0.89)
- Distilling Long-tailed Datasets (CVPR 2025 · similarity 0.86)
- DELT: A Simple Diversity-driven EarlyLate Training for Dataset Distillation (CVPR 2025 · similarity 0.86)
- Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-training of Deep Networks (ICLR 2025 · similarity 0.85)
- Embarrassingly Simple Dataset Distillation (ICLR 2024 · similarity 0.85)
- SelMatch: Effectively Scaling Up Dataset Distillation via Selection-Based Initialization and Partial Updates by Trajectory Matching (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.