Research map / Label-efficient and robust learning / Recognition and long-tail
Rectifying Soft-Label Entangled Bias in Long-Tailed Dataset Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ADSA, a post-hoc, hyperparameter-free soft-label calibration module that removes two sources of imbalance bias (labeling model and distilled images) to improve long-tailed dataset distillation.
Problem
Dataset distillation methods degrade on long-tailed data; soft labels produced by distillation models are biased toward head classes, hurting tail-class accuracy of models trained on distilled data.
Approach
Derive an imbalance-aware generalization bound showing soft-label mismatch matters, identify two bias sources (labeling model and distilled images), and calibrate soft labels with logit adjustment whose strength tau is chosen to minimize class-wise average-confidence variance over the distilled images used as a hold-out set.
Key results
- On ImageNet-1k-LT with EDC at IPC=50, ADSA raises overall accuracy to 41.4% (from 38.6%) and tail accuracy to 24.2%.
- On CIFAR-10-LT with GVBSM at IPC=10, IF=10, ADSA improves accuracy from 49.5% to 54.4%.
- With SRe2L on ImageNet-1k-LT at IPC=10, tail accuracy rises from 7.6% to 14.2%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ADSA to EDC | EDC · ImageNet-LT, IPC=50 | Overall accuracy: 38.6 -> 41.4 | Helped |
| Add ADSA to SRe2L | SRe2L · ImageNet-LT, IPC=10 | Tail accuracy: 7.6 -> 14.2 | Helped |
| Add ADSA to MTT | MTT · CIFAR-10-LT, IPC=10 | Accuracy: 33.4 -> 40.4 | Helped |
| Add ADSA to DREAM | DREAM · CIFAR-10-LT, IPC=10 | Accuracy: 56.0 -> 59.9 | Helped |
| Add ADSA to EDC | EDC · CIFAR-10-LT, IF=10, IPC=50 | Top-1 accuracy: 77.8±0.1 -> 80.8±0.2 | Helped |
| Add ADSA to EDC | EDC · CIFAR-100-LT, IF=10, IPC=50 | Top-1 accuracy: 57.0±0.8 -> 58.0±0.2 | Helped |
| Generate and reuse ADSA-calibrated soft labels with EP=300 | EDC on ImageNet-LT, IPC=50 · ImageNet-LT, IPC=50, EP=300 | Accuracy: +3.2 | Helped |
| Calibrate labels from an imbalanced labeling model (ablation) | SRe2L · CIFAR-10-LT, IF=10, IPC=10 | Accuracy: 34.3 -> 41.4 | Helped |
| Calibrate soft labels for biased distilled images (ablation) | SRe2L · CIFAR-10-LT, IF=10, IPC=10 | Accuracy: 34.7 -> 40.2 | Helped |
| Use ADSA instead of uncalibrated SRe2L soft labels | SRe2L · CIFAR-100-LT, IF=10, IPC=1 | Accuracy: 17.3 -> 17.3 | Hurt |
| Use ADSA rather than SRe2L | SRe2L · CIFAR-100-LT, IF=10, IPC=1, best epoch | Accuracy: 9.1 -> 7.6 | Hurt |
Surprising findings
- Imbalanced distilled images alone bias soft labels, so a balanced labeling model still yields biased labels.
- Adding soft labels to MTT and DREAM sometimes lowers accuracy relative to the base method, while ADSA-calibrated labels improve it.
Limitations
- Assumes shared class-conditional distribution between training and test sets, which may fail under domain shift.
- Requires access to long-tail class prior statistics, so not directly suited to online or streaming learning.
- No error bars reported; results from fixed seeds.
Experimental setup
Datasets: CIFAR-10-LT CIFAR-100-LT ImageNet-1k-LT
Benchmarks: CIFAR-10-LT CIFAR-100-LT ImageNet-1k-LT
Metrics: top-1 accuracy head/mid/tail accuracy
Baselines: LTDD SRe2L GVBSM EDC MTT DREAM
Builds on: SRe2L EDC GVBSM LTDD D3S logit adjustment
Closest papers by meaning
- Distilling Long-tailed Datasets (CVPR 2025 · similarity 0.90)
- A Label is Worth A Thousand Images in Dataset Distillation (NeurIPS 2024 · similarity 0.89)
- GIFT: Unlocking Full Potential of Labels in Distilled Dataset at Near-zero Cost (ICLR 2025 · similarity 0.88)
- Optimizing Distributional Geometry Alignment with Optimal Transport for Generative Dataset Distillation (NeurIPS 2025 · similarity 0.87)
- Are Large-scale Soft Labels Necessary for Large-scale Dataset Distillation? (NeurIPS 2024 · similarity 0.87)
- Long-tailed Recognition with Model Rebalancing (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.