Research map / Learning theory and algorithms / Prediction and decision losses
NRGBoost: Energy-Based Generative Boosted Trees
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NRGBoost, an energy-based generative boosting method for tabular data that fits tree weak learners via a second-order approximation to log-likelihood, with amortized Gibbs sampling via rejection-based sample reuse.
Problem
Generative modeling of tabular data with explicit (unnormalized) density estimation, where tree ensembles are dominant for discriminative tasks but lack generative counterparts supporting arbitrary inference and sampling.
Approach
Boosting in function space over energy functions: each round maximizes a local second-order (Newton-style) approximation of the expected log-likelihood using piecewise-constant tree weak learners, with leaf values exp-like in the P/Q ratio, and reuses previous Gibbs samples via rejection sampling to amortize MCMC cost.
Key results
- NRGBoost achieves R^2 0.547 on Abalone and 0.850 on California Housing, close to XGBoost (0.552, 0.849), and outperforms other generative tree baselines in single-variable inference.
- In ML efficiency sampling evaluation, NRGBoost reaches 0.895 accuracy-type score on Covertype vs TabDDPM's 0.818 and is the only tabular model producing passable MNIST samples.
- In the discriminator measure, NRGBoost obtains AUC 0.625 on Abalone and 0.559 on Adult (lower is better), beating other generative methods on most datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use NRGBoost instead of generative baselines for single-variable inference | Generative density models · Abalone, California Housing, Protein, Adult, MiniBooNE, MNIST, Covertype | discriminative performance | Helped |
| Use NRGBoost instead of XGBoost for single-variable inference | Tabular inference · California Housing | R^2: 0.849 -> 0.850 | Helped |
| Use NRGBoost instead of XGBoost for single-variable inference | Tabular inference · Abalone, Protein, Adult, MiniBooNE, MNIST, Covertype | R^2, AUC, or accuracy | Hurt |
| Train XGBoost on NRGBoost-generated samples instead of real data | XGBoost synthetic-data ML efficiency · Abalone, California Housing, Protein, Adult, MiniBooNE, MNIST, Covertype | R^2, AUC, or accuracy | Mixed |
| Use NRGBoost instead of other sampling methods | Synthetic tabular data · Abalone, California Housing, Adult, MNIST, Covertype | real-vs-generated discriminator AUC | Helped |
| Use NRGBoost instead of other sampling methods | Synthetic tabular data · Protein, MiniBooNE | real-vs-generated discriminator AUC | Mixed |
Surprising findings
- Although NRGBoost is trained with MCMC-based Gibbs sampling, its discriminative inference performance nearly matches XGBoost on smaller datasets.
- DET-based bagged forests (DEF) require very deep trees and underperform NRGBoost despite needing no sampling for training.
Limitations
- Sampling is approximate via MCMC with autocorrelated chains, making independent sampling inefficient.
- Training requires repeated sampling during boosting, though amortized sampling mitigates the cost.
- Theoretical likelihood-increase guarantee assumes exact model probabilities, while practice uses MCMC estimates.
Experimental setup
Datasets: Abalone California Housing Protein Adult MiniBooNE Covertype MNIST (downsampled 2x)
Benchmarks: Abalone California Housing Protein Adult MiniBooNE Covertype MNIST (downsampled)
Metrics: R^2 AUC accuracy ML efficiency discriminator AUC Wasserstein distance
Baselines: XGBoost NGBoost RFDE ARF DEF (ISE) DEF (KL) TVAE TabDDPM Forest-Flow
Builds on: XGBoost NGBoost Density Estimation Trees TabDDPM Forest-Flow
Closest papers by meaning
- Generative Forests (NeurIPS 2024 · similarity 0.89)
- TabEBM: A Tabular Data Augmentation Method with Distinct Class-Specific Energy-Based Models (NeurIPS 2024 · similarity 0.85)
- TabSDS: a Lightweight, Fully Non-Parametric, and Model Free Approach for Generating Synthetic Tabular Data (ICML 2025 · similarity 0.85)
- Revisiting Agnostic Boosting (NeurIPS 2025 · similarity 0.85)
- Better by default: Strong pre-tuned MLPs and boosted trees on tabular data (NeurIPS 2024 · similarity 0.85)
- Gradient Regularized Newton Boosting Trees with Global Convergence (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.