Research map / Learning theory and algorithms / Prediction and decision losses

Autoencoding Random Forests

NeurIPS 2025 · Learning theory and algorithms › Prediction and decision losses · approach: Approximate nearest neighbor indexing · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes an autoencoder for random forests that embeds data via diffusion maps over the RF kernel and decodes latent vectors back to input space using constrained optimization, split relabeling, or k-NN regression.

Problem

Random forests produce implicit kernels but lack principled nonlinear embeddings and any method to decode latent vectors back to the input feature space.

Approach

Treat the normalized RF kernel as a doubly stochastic PSD kernel, use diffusion maps for spectral embedding with Nystrom out-of-sample projection, and invert the embedding with leaf-assignment ILP, split relabeling in embedding space, or k-NN regression over synthetic inputs from leaf intersections.

Key results

• RFAE (k-NN decoder) achieves the best average rank of 1.80 across 20 tabular datasets, best in 12 of 20 datasets versus TVAE, TTVAE, AE, VAE.
• Theory: RF kernel matrix is doubly stochastic and the RF RKHS sequence is asymptotically universal and characteristic.
• On scRNA-seq, RFAE denoising helps align manifolds of two mouse brain studies under PCA and tSNE.

Measured effects reported in the paper

Surprising findings

• The exact ILP leaf-assignment decoder's uniqueness guarantee cannot be shown in general; a minimal binary counterexample exists.
• eForest cannot be used as a baseline since it is fixed at 2x feature dimension and cannot compress.

Limitations

• Decoding computational cost is nontrivial, and ILP is NP-complete and infeasible for large problems.
• k-NN results depend on the choice of k.
• RFAE is not trained end-to-end and is a post-processing step on a pretrained forest; it is not optimized for image data.

Experimental setup

Datasets: abalone adult banknote bc car churn credit diabetes dry_bean forestfires hd king

Benchmarks: 20 tabular datasets reconstruction distortion benchmark MNIST reconstruction scRNA-seq batch alignment

Metrics: distortion (1 - R^2 for continuous plus classification error average rank

Baselines: TVAE TTVAE AE VAE eForest

Builds on: Scornet kernel diffusion maps eForest adaptive random forests (ARF) Moore-Aronszajn theorem

Closest papers by meaning

• Random Forest Autoencoders for Guided Representation Learning (NeurIPS 2025 · similarity 0.86)
• DynFrs: An Efficient Framework for Machine Unlearning in Random Forest (ICLR 2025 · similarity 0.84)
• Minimax Rates and Spectral Distillation for Tree Ensembles (NeurIPS 2026 · similarity 0.84)
• Hybrid Autoencoders for Tabular Data: Leveraging Model-Based Augmentation in Low-Label Settings (NeurIPS 2025 · similarity 0.83)
• Understanding and Enhancing Mask-Based Pretraining towards Universal Representations (NeurIPS 2025 · similarity 0.83)
• Sparse Autoencoders, Again? (ICML 2025 · similarity 0.83)