Research map / Learning theory and algorithms / Matrix and tensor methods
Operator SVD with Neural Networks via Nested Low-Rank Approximation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NeuralSVD, which learns top-L singular functions of a linear operator by minimizing an unconstrained low-rank approximation (LoRA) objective with sequential or joint "nesting" to recover ordered singular functions.
Problem
Computing the leading eigenvalues/eigenfunctions or singular functions of high-dimensional linear operators (e.g., PDE eigenproblems, canonical dependence kernels) where matrix methods scale poorly and existing neural approaches (SpIN, NeuralEF) struggle with orthogonality and ordering.
Approach
Use Schmidt's low-rank approximation objective (unconstrained, unbiased gradients) and break the rotational symmetry with nesting, either sequential (disjoint networks, summing per-level objectives) or joint (weighted sum of nested objectives with masked gradients), so ordered singular functions are learned without explicit orthonormality constraints.
Key results
- On ZS-SBIR (Sketchy Extended, split 1), NeuralSVD-jnt achieves P@100 0.670 and mAP 0.581, above LCALE (0.583/0.476) and IIAE (0.659/0.573); split 2 gives P@100 0.724.
- On the 2D hydrogen atom, both NeuralSVD variants outperform SpIN and NeuralEF by roughly an order of magnitude on subspace distance; LoRA without nesting gives 3.56e-4 vs 2.06e-4 for NeuralSVD-seq.
- Learned function norms estimate the eigenvalues, and the non-positive part of the 2D harmonic oscillator spectrum converges to zero functions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use NeuralSVD joint instead of IIAE | ZS-SBIR retrieval on Sketchy Extended · ZS-SBIR Sketchy Extended, split 1 | P@100: 0.659 -> ${\bf 0.670}_{\pm0.010}$ | Helped |
| Use NeuralSVD joint instead of IIAE | ZS-SBIR retrieval on Sketchy Extended · ZS-SBIR Sketchy Extended, split 1 | mAP: 0.573 -> ${\bf 0.581}_{\pm0.008}$ | Helped |
| Add nesting to the LoRA objective (ablation) | NeuralSVD on the 2D hydrogen atom · 2D hydrogen atom, 16 eigenmodes | subspace distance: 3.56\times 10^{-4}${ ${\pm 6.60\times 10^{-5}}$} -> 2.12\times 10^{-4}${ ${\pm 2.09\times 10^{-5}}$} | Helped |
| Add nesting to the LoRA objective (ablation) | NeuralSVD on the 2D hydrogen atom · 2D hydrogen atom, 16 eigenmodes | subspace distance: 3.56\times 10^{-4}${ ${\pm 6.60\times 10^{-5}}$} -> 2.06\times 10^{-4}${ ${\pm 1.91\times 10^{-5}}$} | Helped |
| Use joint rather than sequential nesting with a shared parameterization (ablation) | NeuralSVD on the 2D hydrogen atom · 2D hydrogen atom | eigenfunction fidelity | Hurt |
| Use sequential rather than joint nesting with a shared parameterization (ablation) | NeuralSVD for ZS-SBIR retrieval · ZS-SBIR Sketchy Extended | P@100: around 0.2 | Hurt |
| Use the most significant modes instead of the least significant modes (ablation) | NeuralSVD retrieval representations · ZS-SBIR Sketchy Extended | retrieval performance | Helped |
| Remove nesting from the LoRA objective (ablation) | NeuralSVD retrieval representations · ZS-SBIR Sketchy Extended | retrieval performance | Mixed |
Surprising findings
- Sequential nesting performed much worse than joint nesting on the ZS-SBIR task (P@100 around 0.2), reversing its advantage on PDE tasks.
- Joint nesting was reported to outperform sequential nesting in the shared-parameter case.
Limitations
- Parametric approach less theoretically understood; no guarantees that networks approximate the operator or that optimization reaches the global optimum.
- Orthogonality is only implicitly promoted, so imperfect orthogonality can affect downstream accuracy.
- Experiments are small-scale with simple MLPs; scalability to larger applications is untested.
Experimental setup
Datasets: Sketchy Extended
Benchmarks: Sketchy Extended ZS-SBIR 2D hydrogen atom TISE 2D harmonic oscillator TISE
Metrics: P@100 mAP eigenfunction subspace distance eigenvalue error
Baselines: SpIN NeuralEF LOBPCG LCALE IIAE LoRA without nesting
Builds on: SpIN NeuralEF EigenGame Matryoshka Representation Learning
Closest papers by meaning
- Revisiting Orbital Minimization Method for Neural Operator Decomposition (NeurIPS 2025 · similarity 0.87)
- Fast Updating Truncated SVD for Representation Learning with Sparse Matrices (ICLR 2024 · similarity 0.85)
- Efficient Parametric SVD of Koopman Operator for Stochastic Dynamical Systems (NeurIPS 2025 · similarity 0.85)
- Learning to Discover Iterative Spectral Algorithms (NeurIPS 2026 · similarity 0.85)
- STNet: Spectral Transformation Network for Solving Operator Eigenvalue Problem (NeurIPS 2025 · similarity 0.84)
- MgNO: Efficient Parameterization of Linear Operators via Multigrid (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.