Research map / Label-efficient and robust learning / Knowledge distillation
Beyond Soft Label: Dataset Distillation via Orthogonal Gradient Matching
Deyu Bo, Xinchao Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| match orthogonalized gradients (OGM) instead of BN statistics | ImageNet-1K distillation with hard labels · ImageNet-1K IPC=10 hard labels | accuracy (%): 1.5 -> 11.8 | Improved |
| set singular values to one (orthogonalize gradients) before matching (ablation) | OGM · ImageNet-1K IPC=10 hard labels (Muon eval) | accuracy (%): 14.8 -> 16.7 | Improved |
| orthogonalize gradients (ablation) | OGM · ImageNet-1K IPC=10 soft labels (Muon eval) | accuracy (%): 45.4 -> 47.0 | Improved |
| initialize synthetic images with RDED instead of raw real images (ablation) | OGM · ImageNet-1K IPC=10 hard labels | accuracy (%): 5.3 -> 10.2 | Improved |
| use patch-level instead of global augmentation (ablation) | OGM · ImageNet-1K IPC=10 hard labels | accuracy (%): 10.2 -> 11.8 | Improved |
| use Muon instead of AdamW in evaluation training (ablation) | OGM · ImageNet-1K IPC=10 soft labels | accuracy (%): 39.4 -> 47.0 | Improved |
| use Muon instead of SGD for training on IPC=10 subset | ResNet-18 on ImageNet-1K · ImageNet-1K IPC=10 subset | accuracy (%): +7.1% accuracy | Improved |
| use Muon instead of SGD on full ImageNet-1K | ResNet-18 · full ImageNet-1K | accuracy (%): +0.8% accuracy | Improved |
| evaluate with Muon instead of Adam for all DD methods | RDED, SRe2L, CDA, G-VBSM, LPLD, DWA, OGM · ImageNet-1K IPC=10 hard labels | accuracy (%): SRe2L 1.5 -> 5.5; RDED 11.5 -> 16.4 | Improved |
| evaluate with SGD instead of Adam on soft labels | DD methods · ImageNet-1K IPC=10 soft labels | accuracy (%): Random 35.8 -> 33.6; DWA 37.9 -> 34.4 | Hurt |
| BN-matching synthesis (SRe2L) instead of random real subset with hard labels | ResNet-18 · ImageNet-1K IPC=10 hard labels | accuracy (%): 4.6 -> 1.5 | Hurt |
| use OGM instead of RDED on a ConvNext-Tiny evaluator with hard labels | cross-architecture · ImageNet-1K IPC=10 hard labels | accuracy (%): 14.3 -> 12.5 | Hurt |
Closest papers by meaning
- Optimizing Distributional Geometry Alignment with Optimal Transport for Generative Dataset Distillation (NeurIPS 2025 · similarity 0.88)
- DELT: A Simple Diversity-driven EarlyLate Training for Dataset Distillation (CVPR 2025 · similarity 0.88)
- Dataset Distillation for Pre-Trained Self-Supervised Vision Models (NeurIPS 2025 · similarity 0.88)
- Efficient Dataset Distillation for Pre-Trained Self-Supervised Models via Statistical Flow Matching (NeurIPS 2026 · similarity 0.87)
- Are Large-scale Soft Labels Necessary for Large-scale Dataset Distillation? (NeurIPS 2024 · similarity 0.87)
- Emphasizing Discriminative Features for Dataset Distillation in Complex Scenarios (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
