Research map / Probabilistic and scientific modeling / Variational and generative sampling
Laplace Sample Information: Data Informativeness Through a Bayesian Lens
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Laplace Sample Information (LSI), a per-sample informativeness measure computed as the KL divergence between Laplace-approximated parameter posteriors with and without a sample, made cheap via a probe model on frozen features.
Problem
Measuring how much information an individual training sample contributes to a neural network's parameters in a way that is information-theoretically grounded, applicable across architectures and learning settings, and computationally tractable.
Approach
Construct a quasi-Bayesian learner post hoc via a Laplace approximation to the MLE weights and compute the closed-form KL divergence between the Gaussian posteriors trained with and without a sample (leave-one-out), using a small probe on frozen features to make LOO retraining feasible; the resulting ordering transfers to larger models.
Key results
- LSI computed on a probe correlates with LSI on a full CNN with Spearman R of 0.93 (and 0.98 for last-layer parameters).
- Models trained on the intermediate and high LSI subsets of CIFAR-10 achieve the best test accuracy and smallest train/test gap, while low-LSI subsets are fit well but generalize worse.
- LSI on a probe weakly correlates with TRAK (Spearman 0.15) on Imagenette, and LSI is strongly correlated with PSMI and weakly with SUI on a CIFAR-10 subset.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the probe model to compute LSI instead of computing it on all CNN parameters | LSI on CNN · CIFAR-10 | Spearman's R: 0.93 | No clear effect |
| Use diagonal or K-FAC Hessian approximation instead of the full Hessian (ablation) | LSI · CIFAR-10 | Spearman's R: R>0.9 | No clear effect |
| Use human mislabels rather than correct labels | LSI · CIFAR-10N | LSI | Mixed |
| Flip labels on 10% of samples | LSI · CIFAR-10 and pediatric pneumonia | LSI | Helped |
| Train on the intermediate-LSI subset rather than the low- or high-LSI subset | CIFAR-10 models trained on one-third of the data · CIFAR-10 | test accuracy | Helped |
| Train on the low-LSI subset rather than the intermediate-LSI subset | CIFAR-10 models trained on one-third of the data · CIFAR-10 | training accuracy | Helped |
| Increase additive Gaussian noise on one class's feature embeddings | LSI class-wise distributions · Pediatric pneumonia | LSI | Helped |
| Use a smaller dataset | LSI · CIFAR-10 subset versus CIFAR-10 | LSI: around one order of magnitude | Helped |
| Increase the DP-SGD clipping norm (ablation) | LSI · CIFAR-10 | LSI | Hurt |
| Decrease the DP-SGD clipping threshold (ablation) | LSI · CIFAR-10 | LSI variance across samples | Hurt |
| Add zero-centered Gaussian noise to DP-SGD gradients (ablation) | LSI · CIFAR-10 | LSI distribution | No clear effect |
| Compute LSI using Riemann Laplace approximation with kernel density estimation (ablation) | LSI · Leave-one-out retraining | LSI fidelity | Hurt |
Surprising findings
- Samples with high LSI often turn out to be mislabeled or out-of-distribution, and some mislabeled samples have higher LSI than the most informative correctly labeled samples.
- With a smaller dataset the LSI values are about one order of magnitude larger, since individual samples carry more information.
Limitations
- Laplace approximation assumes a converged model and quadratic loss around the minimum, though LSI ordering is consistent early in training.
- Requires a Hessian approximation (diagonal or K-FAC) for scalability; full Hessian is intractable.
- Computing LSI relies on leave-one-out retraining, which is costly, so a probe model is used rather than the full target model.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Imagenette Imagewoof pediatric pneumonia IMDb COCO CIFAR-10N
Benchmarks: CIFAR-10 CIFAR-100 Imagenette Imagewoof pediatric pneumonia IMDb COCO
Metrics: Spearman rank correlation train accuracy test accuracy
Baselines: TRAK PSMI SUI
Builds on: Laplace approximation sample information (algorithmic stability) TRAK influence functions
Closest papers by meaning
- FSP-Laplace: Function-Space Priors for the Laplace Approximation in Bayesian Deep Learning (NeurIPS 2024 · similarity 0.85)
- Connecting the Dots: Is Mode-Connectedness the Key to Feasible Sample-Based Inference in Bayesian Neural Networks? (ICML 2024 · similarity 0.82)
- Sampling Binary Data by Denoising through Score Functions (ICML 2025 · similarity 0.82)
- Stabilizing Sample Similarity in Representation via Mitigating Random Consistency (ICML 2025 · similarity 0.82)
- Outlier Gradient Analysis: Efficiently Identifying Detrimental Training Samples for Deep Learning Models (ICML 2025 · similarity 0.82)
- Structural-Entropy-Based Sample Selection for Efficient and Effective Learning (ICLR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.