AI Paper Atlas

Research map / Label-efficient and robust learning / Knowledge distillation

Beyond Soft Label: Dataset Distillation via Orthogonal Gradient Matching

Deyu Bo, Xinchao Wang

CVPR 2026 · Label-efficient and robust learning › Knowledge distillation · approach: Patient consistent distillation

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
match orthogonalized gradients (OGM) instead of BN statisticsImageNet-1K distillation with hard labels · ImageNet-1K IPC=10 hard labelsaccuracy (%): 1.5 -> 11.8Improved
set singular values to one (orthogonalize gradients) before matching (ablation)OGM · ImageNet-1K IPC=10 hard labels (Muon eval)accuracy (%): 14.8 -> 16.7Improved
orthogonalize gradients (ablation)OGM · ImageNet-1K IPC=10 soft labels (Muon eval)accuracy (%): 45.4 -> 47.0Improved
initialize synthetic images with RDED instead of raw real images (ablation)OGM · ImageNet-1K IPC=10 hard labelsaccuracy (%): 5.3 -> 10.2Improved
use patch-level instead of global augmentation (ablation)OGM · ImageNet-1K IPC=10 hard labelsaccuracy (%): 10.2 -> 11.8Improved
use Muon instead of AdamW in evaluation training (ablation)OGM · ImageNet-1K IPC=10 soft labelsaccuracy (%): 39.4 -> 47.0Improved
use Muon instead of SGD for training on IPC=10 subsetResNet-18 on ImageNet-1K · ImageNet-1K IPC=10 subsetaccuracy (%): +7.1% accuracyImproved
use Muon instead of SGD on full ImageNet-1KResNet-18 · full ImageNet-1Kaccuracy (%): +0.8% accuracyImproved
evaluate with Muon instead of Adam for all DD methodsRDED, SRe2L, CDA, G-VBSM, LPLD, DWA, OGM · ImageNet-1K IPC=10 hard labelsaccuracy (%): SRe2L 1.5 -> 5.5; RDED 11.5 -> 16.4Improved
evaluate with SGD instead of Adam on soft labelsDD methods · ImageNet-1K IPC=10 soft labelsaccuracy (%): Random 35.8 -> 33.6; DWA 37.9 -> 34.4Hurt
BN-matching synthesis (SRe2L) instead of random real subset with hard labelsResNet-18 · ImageNet-1K IPC=10 hard labelsaccuracy (%): 4.6 -> 1.5Hurt
use OGM instead of RDED on a ConvNext-Tiny evaluator with hard labelscross-architecture · ImageNet-1K IPC=10 hard labelsaccuracy (%): 14.3 -> 12.5Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy