Research map / Probabilistic and scientific modeling / Variational and generative sampling
Supervised Score-Based Modeling by Gradient Boosting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SSM, a supervised score-based model that recasts gradient boosting as noise-free Langevin refinement of denoising score matching, with step-switching and refinement-rate rules to cut inference time.
Problem
Score-based generative models used for supervised regression and classification are stochastic and slow at inference, which hurts prediction accuracy and robustness.
Approach
Views the GBM iterative update as a noise-free Langevin equation over a conditional denoising score network trained across geometric noise levels; inference uses error-based end-signals to switch noise levels early, with theoretical bounds on switch time and refinement rate.
Key results
- On UCI regression, SSM (L2) achieves the lowest RMSE on 6 of 10 datasets listed in the main table (paper text claims 8 of 10 in one place), e.g., Concrete 3.20 vs CARD 4.77 and Naval 0.01 vs 0.02.
- On CIFAR-10, SSM (L2) reaches 90.99% accuracy versus CARD's 90.93%, with inference time 15.43 s vs 50.30 s.
- On CIFAR-100, SSM (L1) reaches 71.51% accuracy versus CARD's 71.42%, with inference time 32.30 s vs 90.58 s.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SSM instead of CARD | SSM · CIFAR-10 | accuracy: 90.93 $\pm$ 0.02 -> 90.91 $\pm$ 0.00 (L1); 90.93 $\pm$ 0.02 -> 90.99 $\pm$ 0.00 (L2) | Mixed |
| Use SSM instead of CARD | SSM · CIFAR-100 | accuracy: 71.42 $\pm$ 0.01 -> 71.51 $\pm$ 0.00 (L1); 71.42 $\pm$ 0.01 -> 71.38 $\pm$ 0.00 (L2) | Mixed |
| Use SSM error-refinement sampling instead of CARD sampling | SSM · CIFAR-10 and CIFAR-100 | inference time: 50.30 $\pm$ 0.30 -> 19.63 $\pm$ 1.21 and 15.43 $\pm$ 1.12; 90.58 $\pm$ 0.12 -> 32.30 $\pm$ 0.60 and 27.16 $\pm$ 0.19 | Helped |
| Use SSM (L2) instead of the strongest listed baseline | SSM · Boston, Concrete, Kin8nm, Naval, Wine, Yacht | RMSE: 2.23 $\pm$ 0.04 -> 2.04 $\pm$ 0.14; 4.62 $\pm$ 0.13 -> 3.20 $\pm$ 1.00; 6.32 $\pm$ 0.18 -> 4.66 $\pm$ 0.30; 0.02 $\pm$ 0 | Helped |
| Use SSM (L2) instead of the strongest listed baseline | SSM · Energy, Power, Protein | RMSE: 0.23 $\pm$ 0.01 -> 0.31 $\pm$ 0.02; 3.22 $\pm$ 0.03 -> 3.67 $\pm$ 0.33; 3.53 $\pm$ 0.01 -> 3.95 $\pm$ 0.43 | Hurt |
| Use SSM (L1) instead of the strongest listed baseline | SSM · UCI regression tasks | RMSE | Hurt |
| Add fast sampling (end-signal switching) to SSM (ablation) | SSM toy regression · Linear, Quadratic, Log-Log Linear, Log-Log Cubic, Sinusoidal | inference time | Helped |
| Add fast sampling (end-signal switching) to SSM (ablation) | SSM toy regression · Linear, Quadratic, Log-Log Linear, Log-Log Cubic, Sinusoidal | RMSE | Mixed |
| Remove the Gaussian noise term from Langevin dynamics (ablation) | SSM toy regression · Linear, Quadratic, Log-Log Linear, Log-Log Cubic, Sinusoidal | RMSE | Mixed |
| Use loss weight $\lambda(\sigma_i)=\sigma_i^2$ instead of $\lambda(\sigma_i)=\sigma_i$ (ablation) | SSM toy regression · Linear, Quadratic, Log-Log Linear, Log-Log Cubic, Sinusoidal | RMSE: 0.0667 vs 0.0684; 0.1197 vs 0.0950; 0.0654 vs 0.0794; 3.2213 vs 1.9919; 0.0094 vs 0.0033 | Mixed |
| Use Langevin dynamics with noise instead of noise-free dynamics (ablation) | SSM toy regression · Linear, Quadratic, Log-Log Linear, Log-Log Cubic, Sinusoidal | inference time: 0.0354 vs 0.1404; 0.2743 vs 1.0957; 0.4151 vs 1.1151; 0.1311 vs 0.5741; 0.1766 vs 1.0863 | Hurt |
Surprising findings
- On several UCI datasets (e.g., Power, Protein, Year), SSM has higher RMSE than the GBM baselines or CARD, so gains are not uniform.
- The paper notes that the number of denoising steps for later noise levels often hits the 30-step upper bound, so the theoretical switch-time bound is not always consistent with actual computation.
Limitations
- Theoretical switch-time and error bounds rely on approximating network error by training loss, which the authors say is inaccurate and conservative.
- Actual denoising steps often reach the upper bound, so the bound is not always consistent with real computation.
- Performance is mixed on UCI: several datasets show SSM worse than baselines such as XGBoost or CARD.
Experimental setup
Datasets: UCI regression datasets (Boston, Concrete, Energy, Kin8nm, Naval, Power, Protein, Wine, Yacht, Year) 5 synthetic toy regression examples CIFAR-10 CIFAR-100
Benchmarks: UCI regression CIFAR-10 CIFAR-100 toy regression examples
Metrics: RMSE accuracy inference time
Baselines: NGBoost CARD DBT CatBoost XGBoost GCDS MC Dropout Deep Ensembles
Builds on: CARD NCSN gradient boosting machine denoising score matching annealed Langevin dynamics
Closest papers by meaning
- Neural Network-Based Score Estimation in Diffusion Models: Optimization and Generalization (ICLR 2024 · similarity 0.88)
- Approximation and Generalization Abilities of Score-based Neural Network Generative Models for Sub-Gaussian Distributions (NeurIPS 2025 · similarity 0.87)
- Algorithm- and Data-Dependent Generalization Bounds for Diffusion Models (NeurIPS 2025 · similarity 0.86)
- One-Step Diffusion Distillation through Score Implicit Matching (NeurIPS 2024 · similarity 0.85)
- A Non-Asymptotic Convergent Analysis for Scored-Based Graph Generative Model via a System of Stochastic Differential Equations (ICML 2025 · similarity 0.85)
- Don't be so Negative! Score-based Generative Modeling with Oracle-assisted Guidance (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.