Research map / Label-efficient and robust learning / Knowledge distillation
Heavy Labels Out! Dataset Distillation with Label Space Lightening
Ruonan Yu, Songhua Liu, Zigeng Chen, Jingwen Ye, Xinchao Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add multi-weak-teacher guidance to linear-probe CLIP projector (ablation) | HeLlO projector · ImageNet-1K IPC 10 | accuracy: 28.2 -> 30.1 | Improved |
| add LoRA-like knowledge transfer (ablation) | HeLlO projector · ImageNet-1K IPC 10 | accuracy: 30.1 -> 43.5 | Improved |
| add text-embedding-based initialization (ablation) | HeLlO projector · ImageNet-1K IPC 10 | accuracy: 43.5 -> 43.6 | No clear effect |
| add image update with projector (ablation) | HeLlO · ImageNet-1K IPC 10 | accuracy: 43.6 -> 43.7 | Improved |
| increase LoRA learnable parameters from 0.3M/0.4M to 0.6M/0.8M (ablation) | HeLlO · ImageNet-100 IPC 10 | accuracy: 45.4 -> 48.9 | Improved |
| increase LoRA learnable parameters from 0.6M/0.8M to 1.2M/1.6M (ablation) | HeLlO · ImageNet-100 IPC 10 | accuracy: 48.9 -> 49.8 | No clear effect |
| use later-stage teachers (51-91) instead of early (1-41) (ablation) | HeLlO · ImageNet-100 IPC 10 | accuracy: 48.9 -> 41.5 | Hurt |
| use later-stage teachers (51-91) instead of mid (11-51) (ablation) | HeLlO · ImageNet-1K IPC 10 | accuracy: 43.7 -> 39.7 | Hurt |
| replace RDED soft labels with HeLlO projector at IPC 10 | HeLlO vs RDED · ImageNet-100 | accuracy: 36.0 -> 48.9 | Improved |
| replace RDED with HeLlO at IPC 50 | HeLlO vs RDED · ImageNet-1K | accuracy: 56.5 -> 52.2 | Hurt |
| replace RDED with HeLlO on Swin-V2-Tiny | HeLlO vs RDED · ImageNet-1K IPC 10 | accuracy: 17.8 -> 29.5 | Improved |
Closest papers by meaning
- Are Large-scale Soft Labels Necessary for Large-scale Dataset Distillation? (NeurIPS 2024 · similarity 0.91)
- GIFT: Unlocking Full Potential of Labels in Distilled Dataset at Near-zero Cost (ICLR 2025 · similarity 0.91)
- A Label is Worth A Thousand Images in Dataset Distillation (NeurIPS 2024 · similarity 0.90)
- Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximation (ICLR 2025 · similarity 0.88)
- Large Scale Dataset Distillation with Domain Shift (ICML 2024 · similarity 0.87)
- Dataset Distillation for Pre-Trained Self-Supervised Vision Models (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
