Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
A Bayesian Approach to Data Point Selection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BADS, a Bayesian data point selection method that infers the joint posterior of a main network and instance-wise weights (via a weight network) using SGLD, avoiding bi-level optimisation.
Problem
Data point selection (reweighting or filtering uncurated training data using a small in-domain meta set) is dominated by bi-level optimisation methods that are memory- and compute-heavy and lack convergence guarantees with minibatches.
Approach
Treat DPS as posterior inference in a Bayesian model where the backbone is conditioned on weighted train data and the meta data is the likelihood; sample backbone and instance weights jointly with stochastic gradient Langevin dynamics, giving an SGD-like update with minibatch convergence guarantees.
Key results
- On CIFAR-10 with 80% label noise, BADS exceeds BLO and non-DPS approaches by about 15% and 20% in accuracy.
- On WebNLG, BADS achieves a 2 BLEU advantage over the second-best system (Duplicate_Meta) and more than 5 BLEU over the rest.
- BADS uses about 1.5x the GPU memory of the base training, versus about 2.5x for BLO, and matches base training time (61.03 s vs 61.2 s per 100 steps).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BADS instead of the strongest listed non-BADS baseline | OpenLLaMA 3B instruction fine-tuning · MMLU | accuracy: 26.59 vs 25.62 | Helped |
| Use BADS instead of AskLLM-O | OpenLLaMA 3B instruction fine-tuning · ARC-challenge | accuracy: 34.39 vs 35.15 | Hurt |
| Use BADS instead of ClassAct | OpenLLaMA 3B instruction fine-tuning · ARC-easy | accuracy: 67.00 vs 67.34 | Hurt |
| Use BADS instead of AskLLM-O | OpenLLaMA 3B instruction fine-tuning · HellaSwag | accuracy: 52.91 vs 53.71 | Hurt |
| Replace the BADS weight network with individual scalar weights (ablation) | BADS on CIFAR-10 denoising · CIFAR-10 | accuracy | Helped |
| Reduce the weight-prior impact constant σ (ablation) | BADS across the three proof-of-concept scenarios · MNIST, CIFAR, WebNLG | performance | No clear effect |
| Increase sparsity level β in the WebNLG and MNIST settings (ablation) | BADS · WebNLG and MNIST | performance | Hurt |
| Decrease sparsity level β in the CIFAR denoising setting (ablation) | BADS · CIFAR | performance | Mixed |
| Use σ = 0.5 rather than a sufficiently low σ (ablation) | BADS on CIFAR-10 with 50% noisy data · CIFAR-10 | clean-example weighting | Hurt |
| Use σ = 0.05 rather than a sufficiently low σ (ablation) | BADS on CIFAR-10 with 50% noisy data · CIFAR-10 | clean-example weighting | No clear effect |
| Set σ to ≤ 5 × 10^{-5} (ablation) | BADS on CIFAR-10 with 50% noisy data · CIFAR-10 | clean-example weighting | Helped |
| Decrease β from 0.8 to 0.05 (ablation) | BADS trained on WebNLG City and Artist domains · WebNLG | Artist-domain weighting | Helped |
Surprising findings
- Most individual weights tend to collapse to zero under plain sparsity priors, so the paper needed a prior that enforces a target non-zero selection level.
- Unsupervised AskLLM-O and ClassAct-style methods did not consistently beat standard mixing on LLM instruction tuning, while BADS did on most tasks.
Limitations
- Many hyperparameters (sparsity level beta, impact constants rho, sigma, learning rates) need careful tuning.
- GPU memory footprint is higher than non-DPS methods because the weight network or all weights are kept in memory.
- Theoretical convergence guarantee relies on adjusted assumptions from prior SGLD analysis and the paper has no formal proof of the posterior approximation beyond that.
Experimental setup
Datasets: MNIST CIFAR-10 WebNLG 2020 FLAN V2 CoT Dolly Open Assistant 1 MMLU ARC-challenge ARC-easy HellaSwag
Benchmarks: MNIST CIFAR-10 WebNLG 2020 MMLU ARC-challenge ARC-easy HellaSwag
Metrics: accuracy BLEU GPU memory (MB) training time (s)
Baselines: Mixing Meta_Only Random_Select Duplicate_Meta BLO CDS ClassAct AskLLM-O
Builds on: bi-level optimisation (learning to reweight) stochastic gradient Langevin dynamics contrastive data selection ClassAct
Closest papers by meaning
- A Study of Bayesian Neural Network Surrogates for Bayesian Optimization (ICLR 2024 · similarity 0.86)
- Scalable Bayesian Learning with posteriors (ICLR 2025 · similarity 0.86)
- Microcanonical Langevin Ensembles: Advancing the Sampling of Bayesian Neural Networks (ICLR 2025 · similarity 0.85)
- Parameter Expanded Stochastic Gradient Markov Chain Monte Carlo (ICLR 2025 · similarity 0.85)
- Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AI (ICML 2024 · similarity 0.85)
- A Symmetry-Aware Exploration of Bayesian Neural Network Posteriors (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.