Research map / Optimization / Optimizers
Stable Coresets via Posterior Sampling: Aligning Induced and Full Loss Landscapes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes posterior-sampled (Gaussian weight perturbation in batch-norm layers) gradient-matching coreset selection that smooths the induced loss landscape, with theory on Hessian/Newton-step alignment and an improved SGD convergence rate; evaluated on noisy-label vision and NLP benchmarks against Craig, Glister, Crest, and Random.
Problem
Gradient-based coreset selection produces subsets whose induced loss landscape misaligns with the full dataset's, especially under label noise or small budgets, causing poor generalization; prior Hessian-based fixes are costly and unstable.
Approach
Select coresets by matching gradients averaged over Gaussian weight perturbations (posterior samples) of the current model, applied only to batch-normalization layers, which smooths the selection objective and aligns curvature without explicit Hessians.
Key results
- On SNLI with 50% label corruption, the method reaches 0.6062 test accuracy versus 0.5316 for Random and 0.5104 for Crest (per the large-scale table).
- On CIFAR-10 at 90% corruption, the method's accuracy is 0.0953 vs 0.0950 Random and 0.0938 Crest, and at 60% corruption it reaches 0.6704 vs 0.6422 Random.
- Sampling only BN layers gives 0.7318 accuracy at 50% corruption on CIFAR-10, versus 0.7288 for sampling all parameters and 0.7232 for FC-only sampling.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use posterior-smoothed coreset selection instead of Random sampling | Stable Coresets · SNLI, corruption 0.0–0.5 | accuracy: 0.9046±0.0020 -> 0.9132±0.0013; 0.8324±0.0054 -> 0.8664±0.0012; 0.7529±0.0031 -> 0.7841±0.0021; 0.5316±0.0024 -> 0.6062± | Helped |
| use posterior-smoothed coreset selection instead of the strongest listed baseline | Stable Coresets · TinyImageNet, corruption 0.0–0.5 | accuracy: 0.5609±0.0040 -> 0.5732±0.0011; 0.5176±0.0057 -> 0.5526±0.0041; 0.4760±0.0061 -> 0.4832±0.0043; 0.2857±0.0137 -> 0.3644± | Helped |
| use posterior-smoothed coreset selection instead of Crest | Stable Coresets · ImageNet-1k, corruption 0.0 | accuracy: 0.7136±0.0015 -> 0.7091±0.0004 | Hurt |
| use posterior-smoothed coreset selection instead of Random sampling | Stable Coresets · ImageNet-1k, corruption 0.1–0.5 | accuracy: 0.6905±0.0016 -> 0.6977±0.0016; 0.6514±0.0001 -> 0.6837±0.0007; 0.5939±0.0017 -> 0.6388±0.0008 | Helped |
| restrict posterior sampling to batch-normalization layers rather than perturbing all parameters (ablation) | Stable Coresets, ResNet-20 · CIFAR-10, corruption 0.0–0.5 | accuracy: 0.8654 ± 0.0094 -> 0.8757 ± 0.0029; 0.8479 ± 0.0081 -> 0.8544 ± 0.0034; 0.8079 ± 0.0084 -> 0.8120 ± 0.0058; 0.7288 ± 0.0 | Helped |
| restrict posterior sampling to batch-normalization layers rather than the fully connected layer (ablation) | Stable Coresets, ResNet-20 · CIFAR-10, corruption 0.0–0.5 | accuracy: 0.8705 ± 0.0019 -> 0.8757 ± 0.0029; 0.8525 ± 0.0033 -> 0.8544 ± 0.0034; 0.8099 ± 0.0043 -> 0.8120 ± 0.0058; 0.7232 ± 0.0 | Helped |
| use spherical-Gaussian posterior instead of Hessian-Gaussian posterior (ablation) | Stable Coresets · CIFAR-10 and CIFAR-100, corruption 0.0–0.5 | accuracy: CIFAR-10: 0.8721 ± 0.0035 -> 0.8757 ± 0.0029; 0.8523 ± 0.0022 -> 0.8544 ± 0.0034; 0.8041 ± 0.0240 -> 0.8120 ± 0.0058; 0. | Helped |
| use spherical-Gaussian posterior instead of Ensemble posterior (ablation) | Stable Coresets · CIFAR-10 and CIFAR-100, corruption 0.0–0.5 | accuracy: CIFAR-10: 0.8738 ± 0.0010 -> 0.8757 ± 0.0029; 0.8506 ± 0.0010 -> 0.8544 ± 0.0034; 0.8036 ± 0.0010 -> 0.8120 ± 0.0058; 0. | Helped |
| use spherical-Gaussian posterior instead of Hessian-Gaussian posterior (ablation) | Stable Coresets · CIFAR-10 training time | training time: 5962.6s -> 2073.5s | Helped |
| use spherical-Gaussian posterior instead of Ensemble posterior (ablation) | Stable Coresets · CIFAR-10 training time | training time: 3454.2s (parallel) -> 2073.5s | Helped |
Surprising findings
- Naive random sampling is a strong baseline, and Crest's second-order method can underperform it under heavy label noise (e.g., 0.3374 on CIFAR-10 at 60% corruption vs 0.6422 for Random).
- Sampling in all weights or final-layer weights performs worse than sampling only batch-norm layers.
Limitations
- Generalization is not guaranteed for other data types such as sound or video.
- Theoretical analysis relies on strong assumptions (Lipschitz/smoothness, bounded gradients) and the BN-sampling choice is empirically motivated without full theory.
- Code is not released at submission time; authors promise release upon acceptance.
Experimental setup
Datasets: MNIST EMNIST CIFAR-10 CIFAR-100 TinyImageNet ImageNet-1k SNLI
Benchmarks: MNIST EMNIST CIFAR-10 CIFAR-100 TinyImageNet ImageNet-1k SNLI
Metrics: test accuracy time-to-accuracy memory footprint gradient matching error
Baselines: Random Craig Glister Crest Hessian-Gaussian posterior Ensemble posterior
Builds on: Craig Crest Random weight perturbation Stochastic gradient descent
Closest papers by meaning
- Refined Coreset Selection: Towards Minimal Coreset Size under Model Performance Constraints (ICML 2024 · similarity 0.90)
- Mind the Boundary: Coreset Selection via Reconstructing the Decision Boundary (ICML 2024 · similarity 0.88)
- Contributing Dimension Structure of Deep Feature for Coreset Selection (AAAI 2024 · similarity 0.87)
- The Impact of Coreset Selection on Spurious Correlations and Group Robustness (NeurIPS 2025 · similarity 0.87)
- Efficient Core-set Selection for Deep Learning Through Squared Loss Minimization (ICML 2025 · similarity 0.86)
- $\mathbb{D}^2$ Pruning: Message Passing for Balancing Diversity & Difficulty in Data Pruning (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.