Research map / Learning theory and algorithms / Matrix and tensor methods
The tree autoencoder model, with application to hierarchical data visualization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the PCA tree, a tree autoencoder with sparse oblique decision nodes and local PCA leaves, trained by a TAO-based algorithm that monotonically decreases reconstruction error, for hierarchical 2D data visualization.
Problem
Dimensionality reduction for visualization that avoids the distortions of global 2D maps (e.g., t-SNE, UMAP) while providing explicit, fast out-of-sample projection and reconstruction mappings.
Approach
Defines a hard, sparse oblique regression tree as an autoencoder whose encoder routes to a leaf and projects onto that leaf's PCA, and whose decoder is the leaf's local linear reconstruction; learns it by TAO-style alternating optimization of reconstruction error with l1-regularized logistic regression at decision nodes and PCA at leaves.
Key results
- On Fashion MNIST and 20newsgroups, the PCA tree with L=2 per leaf achieves much lower reconstruction error than PCA, which needs many more components to match it (e.g., L=10 or 18 per leaf for Fashion MNIST PCA tree vs. larger PCA L).
- Training on 1M Infinite MNIST samples at depth 4 takes less than an hour, with linear scaling in N, while t-SNE and UMAP show superlinear cost.
- The objective decreases monotonically and typically converges in under 10 iterations; the Fashion MNIST PCA tree reaches an RMSE per pixel of 0.158 at depth 4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a depth-4 PCA tree with 2D PCA in each leaf instead of global PCA | PCA dimensionality reduction · Fashion MNIST | reconstruction error | Helped |
| Use a PCA tree with 2D PCA in each leaf instead of global PCA | PCA dimensionality reduction · 20newsgroups | reconstruction error | Helped |
| Train a PCA tree instead of t-SNE or UMAP | Dimensionality reduction · Infinite MNIST | training time scalability | Helped |
Surprising findings
- Training time is non-monotonic in tree depth: deeper trees can train faster because leaf PCA cost decreases as per-leaf sample size shrinks.
- Always accepting the surrogate logistic-regression update (even if it increases the objective) works as well in practice, though it makes iterates slightly noisy.
Limitations
- Optimization converges only to a local optimum that depends on initialization.
- Needs explicit feature vectors, unlike graph-based methods that use pairwise affinities.
- Like any DR method, reveals only some structure; practitioners should use complementary methods.
Experimental setup
Datasets: MNIST Fashion MNIST Letter Recognition 20newsgroups Amazon Reviews Infinite MNIST
Benchmarks: Fashion MNIST MNIST 20newsgroups Amazon Reviews Letter Recognition Infinite MNIST
Metrics: squared reconstruction error per sample training time per iteration objective function value
Baselines: PCA t-SNE UMAP
Builds on: Tree Alternating Optimization PCA t-SNE LIBLINEAR
Closest papers by meaning
- Principle Component Trees and Their Persistent Homology (AAAI 2024 · similarity 0.87)
- Efficient Sparse PCA via Block-Diagonalization (ICLR 2025 · similarity 0.83)
- SASA: Subspace-Aware Sparse Autoencoders for Effective Mechanistic Interpretability (NeurIPS 2026 · similarity 0.82)
- TreeVI: Reparameterizable Tree-structured Variational Inference for Instance-level Correlation Capturing (NeurIPS 2024 · similarity 0.82)
- Contrastive Multi-view Subspace Clustering via Tensor Transformers Autoencoder (AAAI 2025 · similarity 0.82)
- Autoencoding Random Forests (NeurIPS 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.