Research map / Label-efficient and robust learning / Knowledge distillation

Closed-Form Linear-Probe Dataset Distillation for Pre-trained Vision Models

NeurIPS 2026 · Label-efficient and robust learning › Knowledge distillation · approach: Few-shot and metric distillation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CLP-DD, a closed-form (sample-space kernel ridge) bilevel dataset distillation method for frozen pre-trained encoders with a temperature-scaled class-anchor cross-entropy outer loss, matching LGM with DSA at far lower cost on ImageNet-1K.

Problem

Dataset distillation for frozen pre-trained vision backbones with linear probing; existing trajectory-based methods (LGM) are costly and closed-form NTK methods (KIP, FRePo) target from-scratch training.

Approach

Use the closed-form ridge linear-probe solution on synthetic features (sample-space kernel form, N x N solve) as the inner problem, and optimize synthetic images with a temperature-scaled softmax cross-entropy on real features using the induced classifier columns as class anchors, instead of MSE to one-hot targets.

Key results

• On ImageNet-100 at IPC=1 with DINOv2, CLP-DD reaches 89.2% vs 82.6% for DSA-free LGM and 91.4% for LGM w/ DSA.
• On ImageNet-1K at IPC=1, CLP-DD averages 68.3% vs 67.8% for LGM w/ DSA, matching or surpassing it on three of four backbones.
• On the DINOv2 ImageNet-100 efficiency test, CLP-DD takes 4.07s and 2,361MB versus 57.49s and 19,320MB for LGM w/ DSA.

Surprising findings

• Pairing the closed-form solver with a standard MSE outer loss substantially underperforms trajectory-based methods (ablation average 82.1% vs 85.0% with class-anchor loss on ImageNet-100).
• CLP-DD distilled embeddings lie near class-cluster margins rather than centroids, unlike the Centroid real-image baseline.

Limitations

• Sample-space solver is suited to low IPC; larger synthetic sets may need primal, block-wise or approximate solvers.
• Distilled images are backbone-specific and do not transfer across backbones.
• CLP-DD trails LGM w/ DSA on ImageNet-100 (85.0 vs 87.0) and on Spawrious; code not released at submission.

Experimental setup

Datasets: ImageNet-100 ImageNet-1K Spawrious WaterBirds ImageNet-Fruits

Benchmarks: ImageNet-100 ImageNet-1K Spawrious WaterBirds

Metrics: linear-probe test accuracy wall-clock time peak GPU memory

Baselines: Random Centroid Neighbor LGM LGM w/ DSA KIP-style MSE (CLP-DD w/o CA)

Builds on: Linear Gradient Matching KIP FRePo Differentiable Siamese Augmentation

Closest papers by meaning

• Dataset Distillation for Pre-Trained Self-Supervised Vision Models (NeurIPS 2025 · similarity 0.90)
• Efficient Dataset Distillation for Pre-Trained Self-Supervised Models via Statistical Flow Matching (NeurIPS 2026 · similarity 0.86)
• Unlocking Dataset Distillation with Diffusion Models (NeurIPS 2025 · similarity 0.86)
• Breaking Class Barriers: Efficient Dataset Distillation via Inter-Class Feature Compensator (ICLR 2025 · similarity 0.86)
• Emphasizing Discriminative Features for Dataset Distillation in Complex Scenarios (CVPR 2025 · similarity 0.85)
• D^4: Dataset Distillation via Disentangled Diffusion Model (CVPR 2024 · similarity 0.85)