Research map / Label-efficient and robust learning / Knowledge distillation
Structural Assessment for Understanding and Guiding Dataset Distillation in Discrete Token Space
Yue Cao, Jianyang Gu, Vyacheslav Kungurtsev, Yu Hu, Jozsef Hamari, Zheng Liu, Mohsen Zardadi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| cluster in discrete token space (no PCA, no anchors) (ablation) | TGDD · ImageWoof IPC 20 | accuracy: 43.6 -> 44.3 | Improved |
| add PCA on token histograms (ablation) | TGDD · ImageWoof IPC 50 | accuracy: 57.4 -> 58.9 | Improved |
| add structural-score anchor selection (ablation) | TGDD · ImageWoof IPC 50 | accuracy: 58.9 -> 60.3 | Improved |
| replace score-based anchor selection with random sampling (ablation) | TGDD · ImageWoof IPC 20 | accuracy: 46.3 -> 43.1 | Hurt |
| use all three structural-score terms instead of JSD only (ablation) | TGDD · ImageWoof IPC 10 | accuracy: 38.2 -> 41.2 | Improved |
| use discrete tokenizer guidance instead of DINOv2 (ablation) | TGDD · ImageWoof IPC 50 (DiT) | accuracy: 58.1 -> 60.3 | Improved |
| use VQ-VAE instead of VQGAN tokenizer on LDM (ablation) | TGDD · ImageWoof (LDM) | accuracy: 55.8 -> 55.1 | Hurt |
| weight low-resolution scales more ([3,1,0.5]) vs uniform (ablation) | TGDD · ImageWoof | accuracy: 58.8 -> 60.3 | Improved |
| increase anchors from 10 to 20 at IPC 50 (ablation) | TGDD · ImageNette IPC 50 | accuracy: 81.3 -> 79.9 | Hurt |
| TGDD vs MGD3 on ImageNet-1k IPC 50 | TGDD · ImageNet-1k | accuracy: 60.2 -> 60.3 | No clear effect |
| TGDD vs MGD3 on ImageWoof IPC 50 | TGDD · ImageWoof ResNet-18 | accuracy: 58.3 -> 61.5 | Improved |
| lower token HHI (more balanced token use) in distilled set | TGDD · ImageWoof surrogates | accuracy: w_HHI=-10.99 | Improved |
Closest papers by meaning
- Data Mixture Inference Attack: BPE Tokenizers Reveal Training Data Compositions (NeurIPS 2024 · similarity 0.87)
- DD-Ranking: Rethinking the Evaluation of Dataset Distillation (NeurIPS 2026 · similarity 0.86)
- Rethinking Dataset Distillation for Classification: Do Distilled Sets Outperform Coresets? (NeurIPS 2026 · similarity 0.86)
- Embarrassingly Simple Dataset Distillation (ICLR 2024 · similarity 0.86)
- Large Scale Dataset Distillation with Domain Shift (ICML 2024 · similarity 0.86)
- What is Dataset Distillation Learning? (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
