Research map / Label-efficient and robust learning / Self-supervised and contrastive

Efficient Dataset Distillation for Pre-Trained Self-Supervised Models via Statistical Flow Matching

NeurIPS 2026 · Label-efficient and robust learning › Self-supervised and contrastive · approach: Self-supervision in supervised pipelines · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Statistical Flow Matching (SFM), which replaces batch-level linear gradient matching with a fixed global class-center "flow" computed once from the original data, enabling single-augmentation dataset distillation on frozen self-supervised backbones.

Problem

Linear gradient matching for dataset distillation with pre-trained self-supervised models requires loading many real images per step and multiple differentiable augmentations, causing high compute and memory cost and instability.

Approach

Shows the linear gradient reduces to a local "flow" between target and non-target class centers, then replaces it with a precomputed global statistical flow from the full original dataset and matches synthetic flow to it via cosine loss, plus a classifier-inheritance evaluation plug-in.

Key results

• On ImageNet-100 with IPC=1, SFM (single augmentation) averages 87.7% vs LGM 83.5%; SFM* with multiple augmentations reaches 88.5% vs LGM* 87.2%.
• On ImageNet-1k with CLIP, SFM achieves 67.1% vs LGM 55.5% and LGM* 63.0%.
• Compared to LGM with ten augmentations, SFM with one augmentation reduces GPU memory 10x (16 GB vs 165 GB) and distillation time 4x (18 min vs 80 min), per the paper's claims.

Surprising findings

• Randomly sampling the linear classifier W during distillation has no significant effect; fixed or analytic W perform similarly, suggesting the random-init constraint is redundant.
• CLIP and MoCo-v3 form an anomalous distill/eval pair whose cross-model generalization falls below real-image baselines.

Limitations

• Memory still grows with the number of augmentations because all classes are processed simultaneously; the authors propose class-block processing as future work.
• Scaling to ImageNet-1k with multiple augmentations still causes high memory demands and OOM for LGM on two 90GB GPUs.
• Cross-model generalization is weaker for CLIP and MoCo-v3 pairs, which the authors attribute to feature-space inconsistencies.

Experimental setup

Datasets: ImageNet-1k ImageNet-100 ImageFruits ImageNette ImageWoof Stanford Dogs CUB-200-2011 ArtBench

Benchmarks: ImageNet-100 ImageNet-1k Stanford Dogs CUB-200-2011 ImageFruits

Metrics: top-1 accuracy GPU memory distillation time

Baselines: LGM LGM* MGD^3 Random Centroids Neighbors

Builds on: Linear Gradient Matching MGD^3 hypothesis transfer learning DINO

Closest papers by meaning

• Dataset Distillation for Pre-Trained Self-Supervised Vision Models (NeurIPS 2025 · similarity 0.93)
• Path-Guided Flow Matching for Dataset Distillation (NeurIPS 2026 · similarity 0.90)
• DELT: A Simple Diversity-driven EarlyLate Training for Dataset Distillation (CVPR 2025 · similarity 0.89)
• Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximation (ICLR 2025 · similarity 0.89)
• D^4: Dataset Distillation via Disentangled Diffusion Model (CVPR 2024 · similarity 0.89)
• Self-Corrected Flow Distillation for Consistent One-Step and Few-Step Image Generation (AAAI 2025 · similarity 0.89)