Research map / Label-efficient and robust learning / Knowledge distillation
Breaking Class Barriers: Efficient Dataset Distillation via Inter-Class Feature Compensator
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes INFER, which replaces class-specific synthesis with a shared Universal Feature Compensator added to natural images from all classes, plus MixUp with static soft labels, for compact dataset distillation.
Problem
Class-specific one-hot dataset distillation wastes budget on duplicated intra-class features and ignores inter-class features, creating class barriers and poor decision boundaries.
Approach
Each synthetic instance is a natural image from every class plus a shared learned compensator vector (optimized with CE and BN-statistics loss across several architectures), so one compensator yields multiple synthetic samples; MixUp is then applied using only static soft labels averaged from M networks.
Key results
- On ImageNet-1k at IPC 50 with ResNet-18, INFER reaches 51.8% vs SRe2L* 17.3% under matched 4.04% compression ratio (the 34.5% gap claimed in the abstract).
- INFER+Dyn reaches 68.9% Top-1 on CIFAR-100 IPC 50 with ResNet-18, surpassing SRe2L by 19.4%.
- INFER with ResNet-50 on ImageNet-1k reaches 53.3% at 4.04% compression ratio, and soft-label storage is reduced by 99.3% relative to dynamic labels.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace class-specific synthesis with INFER and UFCs | dataset distillation · ImageNet-1k, ipc=50, matched CR=4.04% | Top-1 accuracy: 17.3 -> 51.8 | Helped |
| use dynamic rather than static soft labels (ablation) | INFER · CIFAR-100, ipc=50, ResNet18 | Top-1 accuracy: 62.8 ± 0.4 -> 68.9 ± 0.1 | Helped |
| use dynamic rather than static soft labels (ablation) | INFER · Tiny-ImageNet, ipc=50, ResNet18 | Top-1 accuracy: 52.9 ± 0.1 -> 54.6 ± 0.4 | Helped |
| use dynamic rather than static soft labels (ablation) | INFER · ImageNet-1k, ipc=50, CR=22.67%, ResNet-50 | Top-1 accuracy: 55.6 ± 0.3 -> 63.4 ± 0.3 | Helped |
| add MobileNetv2, EfficientNetB0, and ShuffleNetV2 to UFC generation (ablation) | INFER · CIFAR-100, ipc=10 | Top-1 accuracy: + 4.3 | Helped |
| add MobileNetv2, EfficientNetB0, and ShuffleNetV2 to UFC generation (ablation) | INFER · CIFAR-100, ipc=50 | Top-1 accuracy: + 3.1 | Helped |
| add MobileNetv2, EfficientNetB0, and ShuffleNetV2 to dynamic soft-label generation (ablation) | INFER+Dyn · CIFAR-100, ipc=10 | Top-1 accuracy: + 7.0 | Helped |
| add MobileNetv2, EfficientNetB0, and ShuffleNetV2 to dynamic soft-label generation (ablation) | INFER+Dyn · CIFAR-100, ipc=50 | Top-1 accuracy: - 0.1 | No clear effect |
| add MobileNetv2, EfficientNetB0, and ShuffleNetV2 to static soft-label generation (ablation) | INFER · CIFAR-100, ipc=10 | Top-1 accuracy: + 7.6 | Helped |
| add MobileNetv2, EfficientNetB0, and ShuffleNetV2 to static soft-label generation (ablation) | INFER · CIFAR-100, ipc=50 | Top-1 accuracy: + 5.0 | Helped |
| add UFCs to the distilled dataset (ablation) | INFER · CIFAR-100, ipc=10, static labels | Top-1 accuracy: 25.6% | Helped |
Surprising findings
- Static soft labels from INFER (one epoch) are competitive with or better than SRe2L-style dynamic soft labels across several settings, e.g. INFER beats SRe2L+dyn on CIFAR-100 at IPC 50 (62.8% vs 49.5% for SRe2L).
- On CIFAR-10 at IPC 10, INFER (34.0%) underperforms several baselines such as RFAD and SeqMatch, and INFER+Dyn on CIFAR-10 IPC 10 trails SRe2L-style static results in some settings.
Limitations
- Compression-ratio comparison relies on the authors' static-label variants of baselines, which may not reflect the original methods' settings.
- Performance on CIFAR-10 at IPC 10 is lower than several prior methods, and dynamic labels help complex architectures more than static labels.
- Experiments use a limited set of architectures for compensator generation (M=3 or 4) and the hyperparameters follow SRe2L with reduced validation epochs.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNet-1k
Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNet-1k
Metrics: Top-1 accuracy compression ratio
Baselines: Random DC DSA KIP RFAD MTT SeqMatch G-VBSM SRe2L
Builds on: SRe2L MixUp DSA MTT
Closest papers by meaning
- Enhancing Dataset Distillation via Non-Critical Region Refinement (CVPR 2025 · similarity 0.91)
- Exploiting Inter-sample and Inter-feature Relations in Dataset Distillation (CVPR 2024 · similarity 0.89)
- Rethinking Dataset Distillation for Classification: Do Distilled Sets Outperform Coresets? (NeurIPS 2026 · similarity 0.89)
- A Label is Worth A Thousand Images in Dataset Distillation (NeurIPS 2024 · similarity 0.89)
- GIFT: Unlocking Full Potential of Labels in Distilled Dataset at Near-zero Cost (ICLR 2025 · similarity 0.88)
- Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximation (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.