Research map / Optimization / Implicit bias and generalization
Implicit Regularization for Tubal Tensor Factorizations via Gradient Descent
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that gradient descent on an overparametrized tubal tensor factorization with small random initialization implicitly converges to low tubal-rank solutions, with a spectral (power-method-like) alignment stage followed by a convergence stage.
Problem
Implicit regularization of gradient descent for tensor factorization under the tubal (t-product) rank model, where prior rigorous results were limited to gradient flow or lazy training or required strong initialization.
Approach
Proves for gradient descent (not just gradient flow) that small random Gaussian initialization in an overparametrized tubal factorization X*X^T yields low tubal-rank recovery, by showing early iterates approximate tensor power method iterations (spectral stage) and then a signal/noise decomposition gives convergence.
Key results
- Under RIP(2r+1) with small initialization scale alpha and R>=3r, gradient descent reaches relative reconstruction error bounded by a term scaling as alpha^(21/16) with probability at least 1-Cke^{-c r}.
- Numerical experiments show test error decreases roughly polynomially with initialization scale alpha, consistent with the theory.
- Overparametrization (larger R from 10 to 400) accelerates convergence to the ground truth tensor.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| decrease the initialization scale | tubal tensor gradient descent · synthetic tubal tensor recovery | test error | Helped |
| increase the number of tensor columns R | tubal tensor gradient descent · synthetic tubal tensor recovery | convergence speed | Helped |
| increase the number of tensor columns R | tubal tensor gradient descent · synthetic tubal tensor recovery | test error | Helped |
| use small random initialization | overparameterized tubal tensor factorization with gradient descent · low-tubal-rank tensor recovery | reconstruction error | Helped |
Limitations
- Analysis restricted to the tubal factorization model and symmetric X*X^T measurements with RIP assumption
- Bounds involve large polynomial and exponential dependence on condition number and initialization scale constants
Experimental setup
Datasets: synthetic Gaussian tubal tensors with random Gaussian measurement tensors
Benchmarks: synthetic tubal tensor recovery (n=10, k=4, r=3)
Metrics: relative Frobenius reconstruction error training loss principal angles between tensor-column subspaces
Baselines: spectral tubal initialization
Builds on: implicit regularization of matrix factorization with small initialization t-SVD tubal tensor algebra spectral initialization for low-tubal-rank recovery
Closest papers by meaning
- Implicit Regularization in Tensor Factorization (ICML 2021 · similarity 0.93)
- Implicit Regularization with Polynomial Growth in Deep Tensor Factorization (ICML 2022 · similarity 0.93)
- Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networks (ICML 2022 · similarity 0.91)
- Deep Weight Factorization: Sparse Learning Through the Lens of Artificial Symmetries (ICLR 2025 · similarity 0.88)
- Mask in the Mirror: Implicit Sparsification (ICLR 2025 · similarity 0.87)
- Geometry-aware training of factorized layers in tensor Tucker format (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.