Research map / Label-efficient and robust learning / Recognition and long-tail
Distilled Datamodel with Reverse Gradient Matching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Distilled Datamodel (DDM), which distills training-data influence into a tiny synthetic set via reverse gradient matching so that leave-one-cluster-out retraining can be done quickly for training data attribution.
Problem
Leave-one-out retraining to measure how training data affects a model is computationally prohibitive, since each altered dataset requires a new training run; existing influence approximations are limited to small perturbations or are unreliable in non-convex settings.
Approach
Distill the effect of each training cluster into one synthetic image per cluster by matching the reverse of the gradient trajectory (the unlearning gradient at step t against the synset gradient at step tau-t), so that fine-tuning the target model on the synset approximates retraining without that cluster; the synset is then perturbed to build an attribution matrix for any evaluation objective.
Key results
- On Dist1 influence, DDM reaches 10.8 (MNIST), 6.8 (CIFAR-10) and 5.3 (CIFAR-100) versus 3.3, 2.7 and 2.3 for random selection and 10.0/5.6/2.6 for Koh et al.
- Reverse gradient matching preserves final accuracy: on CIFAR-100 fine-tuning from the synset yields 56.1 (matching normal training) while standard gradient matching yields 23.5.
- On TinyImageNet, DDM reaches 37.5 accuracy at theta_tau, versus 7.8 for DDM-match.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DDM to locate influential training data rather than Scaleup | training-data influence analysis · MNIST | Dist1: 10.4 -> 10.8 | Helped |
| use DDM to locate influential training data rather than the strongest listed baseline | training-data influence analysis · CIFAR-10 | Dist1: 6.5 -> 6.8 | Helped |
| use DDM to locate influential training data rather than Scaleup | training-data influence analysis · CIFAR-100 | Dist1: 3.9 -> 5.3 | Helped |
| remove the clustering strategy and cluster data by sequence number (ablation) | DDM · MNIST | Dist1: 10.8 -> 9.1 | Hurt |
| remove the clustering strategy and cluster data by sequence number (ablation) | DDM · CIFAR-10 | Dist1: 6.8 -> 5.9 | Hurt |
| remove the clustering strategy and cluster data by sequence number (ablation) | DDM · CIFAR-100 | Dist1: 5.3 -> 3.5 | Hurt |
| replace reverse gradient matching with gradient matching (ablation) | DDM · MNIST | Dist1: 10.8 -> 10.8 | No clear effect |
| replace reverse gradient matching with gradient matching (ablation) | DDM · CIFAR-10 | Dist1: 6.8 -> 5.8 | Hurt |
| replace reverse gradient matching with gradient matching (ablation) | DDM · CIFAR-100 | Dist1: 5.3 -> 4.8 | Hurt |
| replace reverse gradient matching with gradient matching (ablation) | DDM · CIFAR-10 | accuracy: 40.2 -> 85.0 | Hurt |
| replace reverse gradient matching with gradient matching (ablation) | DDM · CIFAR-100 | accuracy: 23.5 -> 56.1 | Hurt |
| replace reverse gradient matching with gradient matching (ablation) | DDM · TinyImageNet | accuracy: 7.8 -> 37.5 | Hurt |
Surprising findings
- Deleting 10% of MNIST training data via the method slightly improved final test performance (95.7 to 95.9 at the 10% row in the detection table, per the paper's interpretation).
Limitations
- Evaluation limited to small image classification benchmarks (MNIST, CIFAR, TinyImageNet) with small CNN/ViT architectures.
- Attribution is at cluster level (K-means clusters), so single data points are not resolved.
- Speedup relative to retraining is asserted but not quantified with timing results in the main text.
Experimental setup
Datasets: MNIST CIFAR-10 CIFAR-100 TinyImageNet
Benchmarks: MNIST CIFAR-10 CIFAR-100 TinyImageNet
Metrics: Dist1 (L2 distance between unlearn-network and target-networ Dist2 (cross-entropy) Dist3 (inverse L2 distance) test accuracy after fine-tuning on synset (Acc. theta_0 and
Baselines: Koh et al. influence functions FASTIF Scaleup Random Select DDM-match (standard gradient matching)
Builds on: Datamodels dataset condensation with gradient matching influence functions (Koh and Liang) FASTIF
Closest papers by meaning
- When to retrain a machine learning model (ICML 2025 · similarity 0.87)
- Embarrassingly Simple Dataset Distillation (ICLR 2024 · similarity 0.86)
- Dataset Distillation for Pre-Trained Self-Supervised Vision Models (NeurIPS 2025 · similarity 0.85)
- Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximation (ICLR 2025 · similarity 0.85)
- Machine Unlearning under Overparameterization (NeurIPS 2025 · similarity 0.85)
- Data Shapley in One Training Run (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.