Research map / Learning theory and algorithms / Kernels and regression theory

Random Feature Representation Boosting

ICML 2025 · Learning theory and algorithms › Kernels and regression theory · approach: High-dimensional asymptotic generalization · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RFRBoost, which builds deep residual random feature networks by greedy layer-wise functional-gradient boosting with random features, giving closed-form solutions for MSE and a quadratically constrained least squares fit for general losses.

Problem

Deep residual random feature neural networks are hard to build because naively stacking random layers degrades performance, and existing deep RFNN methods are limited to Fourier activations.

Approach

Each residual block is a random-feature map A f_t whose linear map A is fit to the negative functional gradient of the loss with respect to the representation. For MSE this has closed-form solutions (sandwiched least squares); for general losses the fit is a norm-constrained least squares, keeping the convex-optimization benefits of RFNNs.

Key results

• Regression: Gradient RFRBoost mean RMSE 0.408 vs RFNN 0.434, E2E MLP ResNet 0.412 (19.3 s fit) and XGBoost 0.394; RFRBoost trains in about 1.7 s.
• Classification: RFRBoost mean accuracy 0.853, tied with XGBoost (0.853), above E2E MLP ResNet (0.851) and RFNN (0.845).
• Concentric circles: RFRBoost reaches 0.997 accuracy vs 0.887 for RFNN and 0.732 for E2E MLP ResNet.

Measured effects reported in the paper

Surprising findings

• Gradient-greedy RFRBoost outperforms exact-greedy on regression, contrary to prior SGD-based gradient representation boosting results.
• XGBoost achieves slightly lower RMSE than RFRBoost on regression, though RFRBoost ranks higher on average rank.

Limitations

• On large datasets (~100k to ~500k samples) E2E networks eventually outperform RFRBoost, and XGBoost does so on some datasets.
• Experiments restricted to tabular data with up to 5000 observations per dataset in the main benchmark.
• Theoretical bound relies on weak learning assumptions and sample splitting with unclear practical tightness.

Experimental setup

Datasets: OpenML regression benchmark suite OpenML classification benchmark suite concentric circles point cloud four larger-scale datasets (~100k and ~500k samples)

Benchmarks: OpenML regression suite (91 datasets total across regression and classification) OpenML classification suite concentric circles point cloud separation

Metrics: RMSE accuracy average relative rank (critical difference diagram) fit time

Baselines: XGBoost E2E MLP ResNet RFNN Ridge regression Logistic regression

Builds on: gradient representation boosting gradient boosting SWIM random features ResFGB-FW GrowNet

Closest papers by meaning

• Functional Gradient Boosting based on Residual Network Perception (ICML 2018 · similarity 0.85)
• Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs (ICLR 2021 · similarity 0.85)
• More is Better: when Infinite Overparameterization is Optimal and Overfitting is Obligatory (ICLR 2024 · similarity 0.84)
• Feature Learning in $L_2$-regularized DNNs: Attraction/Repulsion and Sparsity (NeurIPS 2022 · similarity 0.83)
• Random Sparse Lifts: Construction, Analysis and Convergence of finite sparse networks (ICLR 2024 · similarity 0.83)
• Asymptotics of Learning with Deep Structured (Random) Features (ICML 2024 · similarity 0.82)