Research map / Label-efficient and robust learning / Recognition and long-tail
Distilling Long-tailed Datasets
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Distribution-agnostic Matching and Expert Decoupling to make trajectory-matching dataset distillation work on long-tailed datasets, reaching large gains over prior DD methods on CIFAR-10/100-LT, Tiny-ImageNet-LT and ImageNet-LT.
Problem
Existing dataset distillation methods degrade badly when the target dataset is long-tailed: biased expert trajectories and poor tail-class experts lead to imbalanced, low-quality distilled sets.
Approach
Train the student with a balanced-softmax-style long-tailed loss so it mimics imbalanced expert trajectories without distorting the distilled data, and decouple expert training into a representation expert and a classifier expert, matching backbone and classifier jointly and using the classifier expert for soft-label initialization.
Key results
- On CIFAR-10-LT with imbalance factor 200 and IPC=50, the method reaches 62.3% versus 40.1% for DATM and 49.9% for random selection, nearly matching the full dataset (62.1%).
- On CIFAR-10-LT with imbalance factor 10 and IPC=50, the method reaches 70.5% accuracy versus 66.7% for the best baseline DATM.
- Ablation on CIFAR-10-LT (ω=200, IPC=50) shows the base method at 43.8%, rising to 58.1% with DAM and 62.6% with Expert Decoupling added.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Distribution-agnostic Matching (ablation) | Trajectory-matching dataset distillation · CIFAR-10-LT (ω=200, IPC=50) | accuracy: 14.3% | Helped |
| Add Expert Decoupling to Distribution-agnostic Matching (ablation) | LTDD · CIFAR-10-LT (ω=200, IPC=50) | accuracy: 58.1% to 62.6% | Helped |
| Match the student backbone to the representation expert and the student classifier to the classification expert (ablation) | Expert Decoupling · CIFAR-10-LT | accuracy | Helped |
| Include the representation expert's classifier layer in matching (ablation) | Expert Decoupling · CIFAR-10-LT | accuracy | Hurt |
| Match the entire classification expert rather than only its classifier layer (ablation) | Expert Decoupling · CIFAR-10-LT | accuracy | Hurt |
| Use sequential matching of representation then classification experts (ablation) | Expert Decoupling · CIFAR-10-LT with low imbalance factor | accuracy | Hurt |
| Match only the classification expert (ablation) | Expert Decoupling · CIFAR-10-LT with low imbalance factor | accuracy | Hurt |
| Oversample tail classes before distillation (ablation) | Long-tailed dataset distillation · CIFAR-10-LT with high imbalance factor | accuracy | Hurt |
| Increase ϑcls from 0.2 to 0.4 while keeping ϑrep=1 (ablation) | LTDD · CIFAR-10-LT (ω=10, IPC=50) | accuracy: 70.5 to 68.9 | Hurt |
| Increase ϑcls from 0.2 to 0.6 while keeping ϑrep=1 (ablation) | LTDD · CIFAR-10-LT (ω=10, IPC=50) | accuracy: 70.5 to 68.0 | Hurt |
| Increase ϑrep from 0.2 to 0.4 while keeping ϑcls=1 (ablation) | LTDD · CIFAR-10-LT (ω=200, IPC=50) | accuracy: 62.4 to 61.7 | Hurt |
| Increase ϑrep from 0.2 to 0.6 while keeping ϑcls=1 (ablation) | LTDD · CIFAR-10-LT (ω=200, IPC=50) | accuracy: 62.4 to 61.3 | Hurt |
Surprising findings
- Several existing DD methods (e.g., DATM, MTT, DREAM) perform worse than random selection on long-tailed data at high imbalance factors.
- Distilling a long-tailed dataset gives up to 28.6% lower accuracy than distilling a balanced subset with the same total number of samples.
Limitations
- Experiments are limited to image classification on four long-tailed benchmarks.
- Hyperparameters (weights of representation and classifier matching) depend on the imbalance factor and must be tuned.
- Tail classes with fewer samples than IPC require heuristic handling (oversampling or noise).
Experimental setup
Datasets: CIFAR-10-LT CIFAR-100-LT Tiny-ImageNet-LT ImageNet-LT
Benchmarks: CIFAR-10-LT CIFAR-100-LT Tiny-ImageNet-LT ImageNet-LT
Metrics: top-1 test accuracy
Baselines: Random K-Center Greedy Graph-Cut DC MTT DREAM IDM Minimax DATM TESLA G-VBSM SRe2L
Builds on: Matching Training Trajectories (MTT) DATM Decoupling representation and classifier Balanced Softmax
Closest papers by meaning
- Rectifying Soft-Label Entangled Bias in Long-Tailed Dataset Distillation (NeurIPS 2025 · similarity 0.90)
- Large Scale Dataset Distillation with Domain Shift (ICML 2024 · similarity 0.89)
- Rethinking Dataset Distillation for Classification: Do Distilled Sets Outperform Coresets? (NeurIPS 2026 · similarity 0.89)
- Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximation (ICLR 2025 · similarity 0.88)
- Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-training of Deep Networks (ICLR 2025 · similarity 0.88)
- Embarrassingly Simple Dataset Distillation (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.