Research map / Label-efficient and robust learning / Knowledge distillation
Distilling Dataset into Neural Field
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DDiF, which parameterizes each distilled instance as a sine-activated neural field queried on a coordinate grid, improving dataset distillation under a fixed storage budget across image, video, audio, and 3D voxel data.
Problem
Dataset distillation is limited by inefficient parameterization of synthetic instances under a storage budget, especially for high-resolution and high-dimensional data (video, 3D, audio).
Approach
Store each synthetic instance as a SIREN-style neural field that maps coordinates to values, so storage is independent of data resolution; decode by querying coordinates, enabling cross-resolution decoding and a larger feasible space than frequency-domain (FreD) parameterization under the same budget.
Key results
- On ImageNet-Subset 128x128 under IPC=1 (TM), DDiF reaches 72.0% on Nette vs 66.8% for FreD, with up to 4.1 points over the second-best method.
- On 3D voxel ModelNet under IPC=1 with DM, DDiF gets 88.4% vs 87.3% for FreD; on audio with 10 spectrograms per class it gets 90.5% vs 82.9% for IDC.
- Cross-architecture results show DDiF beats the second-best method by 10.4 to 23.1 points on ImageNet-Subset across AlexNet, VGG11, ResNet18, and ViT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use neural-field parameterization (DDiF) instead of Vanilla+RTP | ImageNet-Subset distilled dataset · ImageNette 128×128, IPC=1 | accuracy (%): 69.6 -> 72.0 | Helped |
| Use neural-field parameterization (DDiF) instead of Vanilla+RTP | ImageNet-Subset distilled dataset · ImageWoof 128×128, IPC=1 | accuracy (%): 38.8 -> 42.9 | Helped |
| Use DDiF instead of the strongest listed baseline | ImageNet-Subset distilled dataset · ImageNet-Subset 256×256, IPC=1, ImageSquawk | accuracy (%): 47.1 -> 67.0 | Helped |
| Use DDiF instead of IDC | Mini Speech Commands distilled dataset · 10 per class | accuracy (%): 82.9 -> 90.5 | Helped |
| Use DDiF instead of FreD | 3D voxel distillation with DM · ShapeNet, IPC=1 | accuracy (%): 90.6 -> 93.1 | Helped |
| Use DDiF instead of SPEED | CIFAR-100 distilled dataset · IPC=1 | accuracy (%): 40.4 -> 42.1 | Helped |
| Increase the neural-field layer count (ablation) | DDiF · ImageNet-Subset dataset | accuracy | Mixed |
| Increase neural-field width (ablation) | DDiF · ImageNet-Subset dataset | accuracy | Mixed |
| Replace the neural-field structure with FFN (ablation) | DDiF · ImageNet-Subset dataset | accuracy | Helped |
| Resize distilled data using interpolation instead of decoding at the target resolution | Previous parameterization methods · Cross-resolution evaluation | accuracy degradation | Hurt |
| Use DDiF rather than Vanilla at fixed DIPC=1 with a smaller budget | ImageNet-Subset 128×128, Nette · One decoded instance per class | accuracy (%): 51.4 -> 49.1 | Hurt |
| Use DDiF rather than Vanilla at fixed DIPC=51 | ImageNet-Subset 128×128, Squawk · 51 decoded instances per class | accuracy (%): 69.0 -> 67.0 | Hurt |
Surprising findings
- On ImageNet-Subset 256x256 with a fixed number of decoded instances, DDiF sometimes outperforms the input-sized Vanilla baseline despite using a much smaller budget.
- On CIFAR-10 and CIFAR-100 (low-dimensional data), gains over prior parameterizations are smaller, with some baselines (e.g., NSD) matching or beating DDiF on some settings.
Limitations
- Gains are smaller on low-dimensional datasets such as CIFAR-10/100, where some parameterized baselines are competitive.
- Resolution changes at test time are mainly evaluated with fixed network architectures and interpolation baselines; theoretical comparison is only made against FreD, not IDC.
Experimental setup
Datasets: ImageNet-Subset CIFAR-10 CIFAR-100 MiniUCF Mini Speech Commands ModelNet-10 ShapeNet
Benchmarks: ImageNet-Subset 128x128 ImageNet-Subset 256x256 CIFAR-10 CIFAR-100 MiniUCF Mini Speech Commands ModelNet ShapeNet
Metrics: test accuracy
Baselines: Vanilla (TM) IDC FreD HaBa SPEED Vanilla+RTP LatentDD NSD GLaD H-GLaD LD3M FRePo
Builds on: trajectory matching distribution matching FreD SIREN
Closest papers by meaning
- Large Scale Dataset Distillation with Domain Shift (ICML 2024 · similarity 0.88)
- Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximation (ICLR 2025 · similarity 0.87)
- Data Distillation Can Be Like Vodka: Distilling More Times For Better Quality (ICLR 2024 · similarity 0.87)
- Taming Diffusion for Dataset Distillation with High Representativeness (ICML 2025 · similarity 0.86)
- Rethinking Dataset Distillation for Classification: Do Distilled Sets Outperform Coresets? (NeurIPS 2026 · similarity 0.85)
- Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-training of Deep Networks (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.