Research map / Learning theory and algorithms / Matrix and tensor methods
An Iterative Min-Min Optimization Method for Sparse Bayesian Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an iterative Min-Min optimization (CCCP-based, with auxiliary latent variables) that analytically updates SBL hyperparameters for prior and noise level, proven globally convergent to a local minimum or saddle point of the marginal likelihood.
Problem
Classical sparse Bayesian learning (SBL) algorithms for maximizing the marginal likelihood lack global convergence guarantees, and jointly estimating prior and noise hyperparameters is hard because the CCCP subproblem has no closed-form solution.
Approach
Decompose the negative log marginal likelihood into concave and convex parts, apply CCCP, and introduce a latent variable theta to re-express the data term so that both the prior hyperparameters gamma and noise level lambda have closed-form updates; the method is equivalent to iteratively reweighted l2 minimization with automatically updated regularization and weights.
Key results
- On Gaussian-matrix sparse recovery at 35 dB SNR with ±1 spikes, the proposed method achieves 100% success versus below 50% for MacKay SBL, EM SBL, IR SBL and VI SBL.
- On the Lorenz system discovery task, IR SBL and VI SBL fail at all noise levels, while the proposed method is more robust to noise than MacKay SBL and EM SBL.
- Theory: the sequence of (gamma, lambda) monotonically converges to a local minimum or saddle point of the marginal likelihood for any starting point.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the iterative Min-Min SBL method instead of MacKay SBL | Sparse signal recovery with a Gaussian-distributed matrix · ±1 spike signal, 35 dB SNR | successful recovery rate: 48% -> 100% | Helped |
| Use the iterative Min-Min SBL method instead of MacKay SBL | Sparse signal recovery with a Gaussian-distributed matrix · Gaussian signal, 35 dB SNR | successful recovery rate: 48% -> 94% | Helped |
| Use the iterative Min-Min SBL method instead of MacKay SBL | Sparse signal recovery with a Gaussian-distributed matrix · Uniformly distributed signal, 35 dB SNR | successful recovery rate: 82% -> 90% | Helped |
| Use the iterative Min-Min SBL method instead of MacKay SBL | Sparse signal recovery with a low-rank matrix · ±1 spike signal, 35 dB SNR | successful recovery rate: 33% -> 93% | Helped |
| Use the iterative Min-Min SBL method instead of MacKay SBL | Sparse signal recovery with a low-rank matrix · Gaussian signal, 35 dB SNR | successful recovery rate: 30% -> 80% | Helped |
| Use the iterative Min-Min SBL method instead of MacKay SBL | Sparse signal recovery with a low-rank matrix · Uniformly distributed signal, 35 dB SNR | successful recovery rate: 58% -> 89% | Helped |
| Use the iterative Min-Min SBL method instead of IR SBL | Sparse kernel regression · Red Wine Quality, exponential kernel | MRE: 0.1006 -> 0.0951 | Helped |
| Use the iterative Min-Min SBL method instead of IR SBL | Sparse kernel regression · Red Wine Quality, Matérn-3/2 kernel | MRE: 0.1006 -> 0.0957 | Helped |
| Use the iterative Min-Min SBL method instead of MacKay SBL | Sparse kernel regression · Red Wine Quality, linear kernel | MRE: 0.0952 -> 0.0970 | Hurt |
| Use the iterative Min-Min SBL method instead of IR SBL | Sparse kernel regression · Red Wine Quality, Gaussian kernel | MRE: 0.1004 -> 0.0980 | Helped |
Surprising findings
- MacKay SBL and EM SBL achieve 0% success at most SNR levels on Gaussian dictionaries, indicating large gaps in recovery robustness.
Limitations
- High computational complexity from matrix inversions scaling as n^3, limiting large-scale problems.
- Global convergence is only to a local minimum or saddle point, not a global optimum.
- With additional constraints on w, global convergence of the resulting algorithm is not guaranteed.
Experimental setup
Datasets: Red Wine Quality (UCI) Synthetic sparse signals (random Gaussian and low-rank dictionaries) Lorenz system trajectories (simulated with ode45)
Benchmarks: Synthetic sparse signal recovery (success probability) Lorenz system discovery Red Wine Quality kernel regression
Metrics: probability of successful support recovery probability of discovering Lorenz system Sparsity MRE combined distance d
Baselines: MacKay SBL EM SBL IR SBL VI SBL AMP
Builds on: Sparse Bayesian learning (Tipping 2001) concave-convex procedure (Yuille & Rangarajan 2001) iteratively reweighted l2 minimization (Wipf & Nagarajan 2010) Global Convergence Theorem (Luenberger & Ye)
Closest papers by meaning
- Training Bayesian Neural Networks with Sparse Subspace Variational Inference (ICLR 2024 · similarity 0.87)
- Posterior Contraction for Sparse Neural Networks in Besov Spaces with Intrinsic Dimensionality (NeurIPS 2025 · similarity 0.87)
- Block Sparse Bayesian Learning: A Diversified Scheme (NeurIPS 2024 · similarity 0.87)
- Sparse Bayesian Generative Modeling for Compressive Sensing (NeurIPS 2024 · similarity 0.86)
- Sparse Variational Student-t Processes (AAAI 2024 · similarity 0.85)
- Bayesian Optimisation with Unknown Hyperparameters: Regret Bounds Logarithmically Closer to Optimal (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.