Research map / Architectures and efficiency / RNNs and normalization
Amortized Eigendecomposition for Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes amortized eigendecomposition, replacing per-step eigh/svd inside neural network training with QR-reparameterized Stiefel-manifold variables and an eigen loss based on Brockett's cost or convex trace functions.
Problem
Eigendecomposition and SVD inside neural network training are orders of magnitude slower than other operations, becoming the training bottleneck for tasks like nuclear norm regularization, latent PCA, and graph structure learning.
Approach
Relax the exact eigendecomposition into an unconstrained optimization over a QR-reparameterized orthonormal matrix, adding an eigen loss (Brockett's cost or convex trace loss) jointly optimized with the training loss, so only a cheap QR is needed per iteration.
Key results
- On a 512x512 nuclear norm regularization task, the amortized method gives a 105.4x speed-up over svd relative to backbone time per iteration.
- On a 19717-node low-rank GCN, per-iteration time drops from 4.941 s (eigh) to 0.0273 s (amortized), a 615.2x speed-up relative to backbone.
- Latent-space PCA on MNIST achieves about 40% faster training than conventional eigh, with nearly identical reconstruction loss curves.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace svd with amortized eigendecomposition | Nuclear norm regularization · 128 × 128 | speed-up: 4.06× | Helped |
| Replace svd with amortized eigendecomposition | Nuclear norm regularization · 256 × 256 | speed-up: 13.5× | Helped |
| Replace svd with amortized eigendecomposition | Nuclear norm regularization · 512 × 512 | speed-up: 105.4× | Helped |
| Replace eigh with amortized eigendecomposition | Latent-space PCA · 256 × 2 | speed-up: 1.47× | Helped |
| Replace eigh with amortized eigendecomposition | Latent-space PCA · 512 × 2 | speed-up: 1.45× | Helped |
| Replace eigh with amortized eigendecomposition | Latent-space PCA · 1028 × 2 | speed-up: 1.39× | Helped |
| Replace eigh with amortized eigendecomposition | Low-rank GCN · Cora (2708 × 16) | speed-up: 23.4× | Helped |
| Replace eigh with amortized eigendecomposition | Low-rank GCN · Citeseer (3312 × 16) | speed-up: 23.7× | Helped |
| Replace eigh with amortized eigendecomposition | Low-rank GCN · Pubmed (19717 × 16) | speed-up: 615.2× | Helped |
Surprising findings
- Eigh-based eigendecomposition scales steeply with dimension and becomes the dominant cost, far exceeding fully-connected layer cost at large widths.
Limitations
- As a pure numerical eigensolver it offers no speed or precision advantage over conventional methods.
- Only the top-k eigenpairs are targeted with specific loss forms (Brockett or convex trace); the approach relies on tuning eta and M.
- Code not yet released at submission time.
Experimental setup
Datasets: MNIST Celeb-A-HQ Cora Citeseer PubMed
Benchmarks: MNIST Celeb-A-HQ Cora Citeseer PubMed
Metrics: execution time per iteration speed-up ratio MSE of eigenvalues reconstruction loss node classification accuracy
Baselines: eigh-based eigendecomposition svd-based nuclear norm GCN with SVD low-rank graph (Pro-GNN-style)
Builds on: Brockett's cost function von Neumann trace inequality QR-based orthogonal reparameterization GCN
Closest papers by meaning
- Backpropagation-Friendly Eigendecomposition (NeurIPS 2019 · similarity 0.90)
- An Investigation into Neural Net Optimization via Hessian Eigenvalue Density (ICML 2019 · similarity 0.86)
- STNet: Spectral Transformation Network for Solving Operator Eigenvalue Problem (NeurIPS 2025 · similarity 0.83)
- Explicit Eigenvalue Regularization Improves Sharpness-Aware Minimization (NeurIPS 2024 · similarity 0.83)
- The Nuclear Route: Sharp Asymptotics of ERM in Overparameterized Quadratic Networks (NeurIPS 2025 · similarity 0.83)
- Improving Adaptivity via Over-Parameterization in Sequence Models (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.