Research map / Architectures and efficiency / RNNs and normalization
Generalized Large-Scale Data Condensation via Various Backbone and Statistical Matching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes G-VBSM, a dataset condensation method that extends SRe2L's one-sided "local-match-global" matching to multiple backbones, layers, and statistics, plus data densification, to scale distilled datasets to full ImageNet-1k.
Problem
Dataset condensation on full-resolution ImageNet-1k is computationally feasible via SRe2L, but its matching is limited to one backbone layer (BatchNorm), one model, and channel statistics, which limits distilled dataset generalization.
Approach
"Generalized matching": sufficient and varied local-match-global matching across backbones (randomly sampled per iteration with soft-label ensembling), layers (Convolution patch/channel statistics recorded in one pass plus BatchNorm), and statistics, combined with a data densification loss that encourages per-class linear independence and an EMA-based SDS-style statistic matching.
Key results
- G-VBSM reaches 31.4% Top-1 on full ImageNet-1k at IPC 10 with ResNet18, 10.1 points above SRe2L (21.3%).
- G-VBSM achieves 38.7% on CIFAR-100 (IPC 10) and 47.6% on Tiny-ImageNet (IPC 50, ResNet18), surpassing prior SOTA by 3.9 and 6.5 points.
- On ImageNet-1k with IPC 10, G-VBSM improves over SRe2L by 7.0% for ResNet50 and 7.3% for ResNet101; with IPC 50 it reaches 51.8% on ResNet18 vs 46.8% for SRe2L.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DD, GBM, and GSM with generalized matching (ablation) | SRe2L distilled data · ImageNet-1k, IPC 10 | Top-1 accuracy: 21.3% -> 31.4% | Helped |
| Add MSE+0.1×GT in the evaluation loss (ablation) | SRe2L · ImageNet-1k, IPC 10 | Top-1 accuracy: +0.9% | Helped |
| Add data densification (DD) (ablation) | SRe2L · ImageNet-1k, IPC 10 | Top-1 accuracy: 21.3% -> 22.4% | Helped |
| Add generalized backbone matching (GBM) (ablation) | DD · ImageNet-1k, IPC 10 | Top-1 accuracy: 22.4% -> 25.1% | Helped |
| Add GSM with βdr=1.0 (ablation) | DD + GBM + MSE+GT · ImageNet-1k, IPC 10 | Top-1 accuracy: 25.1% -> 27.8% | Helped |
| Set GSM drop probability βdr from 1.0 to 0.0 (ablation) | DD + GBM + GSM + MSE+GT · ImageNet-1k, IPC 10 | Top-1 accuracy: 27.8% -> 31.4% | Helped |
| Use EMA matching with α above 0 (ablation) | G-VBSM BatchNorm matching · ImageNet-1k, IPC 10 | Top-1 accuracy: 30.4% -> 31.4% | Helped |
| Increase EMA factor α from 0.4 to 0.8 (ablation) | G-VBSM · ImageNet-1k, IPC 10 | Top-1 accuracy: 31.4% -> 31.9% | Helped |
| Use logit normalization during soft-label generation (ablation) | G-VBSM · ImageNet-1k, IPC 10; ResNet18 evaluation | Top-1 accuracy: 31.0% -> 31.4% | Helped |
| Use logit normalization during soft-label generation (ablation) | G-VBSM · ImageNet-1k, IPC 10; ResNet50 evaluation | Top-1 accuracy: 35.4% -> 34.5% | Hurt |
| Ensemble more candidate backbones for soft-label generation (ablation) | G-VBSM · ImageNet-1k, IPC 10; ResNet18 evaluation | Top-1 accuracy: 25.7% -> 31.4% | Helped |
| Use the four-model G-VBSM distilled dataset instead of SRe2L | ImageNet-1k, IPC 50 · DeiT-Tiny, ResNet18, MobileNetV2, and Swin-Tiny | Top-1 accuracy: 15.41% -> 29.43%; 46.79% -> 51.82%; 36.59% -> 48.66%; 39.23% -> 57.40% | Helped |
Surprising findings
- Reordering the data loop for data densification degraded ResNet50 accuracy by 2.6% on ImageNet-1k, which motivated the EMA-based SDS-style matching.
- Logit normalization in soft label generation helps ResNet18 evaluation but hurts ResNet50 and ResNet101, so it is used only for some backbones.
Limitations
- Efficiency gains over SRe2L are reduced by GSM's extra cost, which needs random dropping of matching terms.
- Logit normalization and loss choices are not universally effective across evaluation models, requiring per-model settings.
- Evaluated only on classification datasets; detection and segmentation are left to future work.
Experimental setup
Datasets: ImageNet-1k Tiny-ImageNet CIFAR-10 CIFAR-100
Benchmarks: ImageNet-1k Tiny-ImageNet CIFAR-10 CIFAR-100
Metrics: Top-1 validation accuracy
Baselines: SRe2L TESLA DataDAM MTT DM CAFE KIP DC FTD
Builds on: SRe2L DeepInversion MTT score distillation sampling FKD
Closest papers by meaning
- DELT: A Simple Diversity-driven EarlyLate Training for Dataset Distillation (CVPR 2025 · similarity 0.86)
- M3D: Dataset Condensation by Minimizing Maximum Mean Discrepancy (AAAI 2024 · similarity 0.85)
- FADRM: Fast and Accurate Data Residual Matching for Dataset Distillation (NeurIPS 2025 · similarity 0.84)
- Optimizing Distributional Geometry Alignment with Optimal Transport for Generative Dataset Distillation (NeurIPS 2025 · similarity 0.84)
- GIM: Learning Generalizable Image Matcher From Internet Videos (ICLR 2024 · similarity 0.84)
- GSDD: Generative Space Dataset Distillation for Image Super-resolution (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.