Research map / Learning theory and algorithms / Kernels and regression theory
Sketchy Moment Matching: Toward Fast and Provable Data Selection for Finetuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Sketchy Moment Matching (SkMM), a two-stage data selection method for finetuning that sketches model gradients to find a low-dimensional subspace and then selects data by moment matching in that subspace, with fast-rate generalization guarantees.
Problem
Data selection for finetuning in high-dimensional parameter spaces (r > n) where variance minimization alone does not characterize generalization, requiring control of bias from low-rank approximation as well.
Approach
Combine gradient sketching (JL transform of per-model Jacobians) to explore a low-dimensional subspace that controls bias, with a relaxed moment-matching objective (commuting-matrix heuristic, quadratic with linear constraints, projected gradient descent) that reduces variance within that subspace, yielding fast O(ī/n) generalization independent of parameter dimension.
Key results
- On UTKFace with CLIP ViT-B/32 frozen features, SkMM reaches MAE 3.54 at n=2000 and 3.25 at n=3000, and roughly halves MAE versus uniform sampling at n=100 (5.57 vs 10.55).
- On StanfordCars linear probing, SkMM-LP achieves the best accuracy at every coreset size, e.g., 76.34% vs 75.40% for uniform at n=4000.
- In the synthetic GMM experiment, SkMM achieves the lowest empirical risk at small n (3.54e-3 at n=48 vs 1.14e-1 for uniform), matching the full-data risk 2.95e-3 at n=1600.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SkMM data selection instead of Adaptive | Synthetic high-dimensional linear probing · GMM (n=48) | empirical risk: (3.81 $\pm$ 0.65)e-3 -> (3.54 $\pm$ 0.51)e{-3} | Helped |
| Use SkMM data selection instead of Uniform Sampling | CLIP ViT-B/32 linear probing · UTKFace (n=200) | MAE: $ 8.94 \pm 3.48$ -> $ 4.70 \pm 0.05$ | Helped |
| Use SkMM-LP data selection instead of Entropy | CLIP ViT-B/32 linear probing · StanfordCars (n=2000) | accuracy (%): 67.95 $\pm$ 0.11 -> 68.27 $\pm$ 0.03 | Helped |
| Use SkMM-LP data selection instead of Forgetting | CLIP ViT-B/32 linear probing · StanfordCars (n=2000) | F1 (%): 64.85 $\pm$ 0.13 -> 65.29 $\pm$ 0.03 | Helped |
| Use a smaller sketching dimension m (ablation) | SkMM classification data selection · StanfordCars, low data regime | finetuning performance | Helped |
| Increase the slackness lower bound c_S (ablation) | SkMM classification data selection · StanfordCars, low data regime | finetuning performance | Helped |
Surprising findings
- Uniform sampling is a surprisingly strong baseline for low-dimensional linear probing and at large coreset sizes relative to N.
- Herding performs very poorly in the synthetic experiment at small coreset sizes (empirical risk around 7.4e+2).
Limitations
- The commuting-matrix assumption used to relax the moment-matching objective does not hold in general (heuristic).
- Finetuning experiments on StanfordCars use LP on CLIP and FT on an ImageNet-pretrained ResNet18; full FT results are deferred to the appendix.
- Theory relies on kernel/linearized regime and low intrinsic dimension assumptions.
Experimental setup
Datasets: UTKFace StanfordCars Synthetic Gaussian mixture model
Benchmarks: UTKFace StanfordCars Synthetic GMM regression
Metrics: empirical risk mean absolute error accuracy F1 score
Baselines: Uniform sampling Herding K-center greedy Adaptive sampling Leverage score sampling Contextual Diversity Glister GraNd Forgetting DeepFool Entropy Margin
Builds on: V-optimal experimental design coreset selection for linear regression Johnson-Lindenstrauss transform leverage score sampling
Closest papers by meaning
- SelMatch: Effectively Scaling Up Dataset Distillation via Selection-Based Initialization and Partial Updates by Trajectory Matching (ICML 2024 · similarity 0.84)
- On the Closed-Form of Flow Matching: Generalization Does Not Arise from Target Stochasticity (NeurIPS 2025 · similarity 0.84)
- Boundary Matters: A Bi-Level Active Finetuning Method (NeurIPS 2024 · similarity 0.84)
- Slower Generalization, Faster Memorization: A Sweet Spot in Algorithmic Learning (NeurIPS 2026 · similarity 0.84)
- PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularization (NeurIPS 2024 · similarity 0.84)
- Asymptotically Free Sketched Ridge Ensembles: Risks, Cross-Validation, and Tuning (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.