Research map / Optimization / Optimizers
Supervised Matrix Factorization: Local Landscape Analysis and Applications
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Provides a local landscape analysis of supervised matrix factorization (SMF), yielding a BCD algorithm with adaptive step sizes, convergence guarantees, minimal L2 regularization for local strong convexity, a local MLE consistency result, and a GPU-friendly neural BCD implementation.
Problem
Training supervised matrix factorization (SMF) requires solving a non-convex, factor-wise constrained problem with three or four parameter blocks, and its optimization landscape and algorithms' behavior are poorly understood.
Approach
Uses the block structure of the 4x4 Hessian: diagonal blocks give Lipschitz constants for block-wise projected gradient steps with adaptive step sizes (BCD as block majorization-minimization), and the full Hessian via block diagonal dominance gives minimal L2 regularization ensuring local strong convexity and local minimizers near the true factors.
Key results
- BCD with adaptive step sizes achieves an epsilon-stationary point within O(eps^-1 (log eps^-1)^2) iterations (Theorem 4.4).
- On CuMiDa breast cancer microarray data, SMF-W (BCD) reaches 0.924 (0.02) accuracy versus 0.890 for FFNN and 0.866 for SVM.
- SMF-W shows the best overall performance in the Pareto plots of F-score/accuracy versus relative reconstruction error, interpolating between LR and MF-LR as xi varies.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SMF-W with BCD instead of SMF-H with BCD | SMF · Breast cancer microarray | classification score: 0.880 (0.02) -> 0.924 (0.01) | Helped |
| Use SMF-W with BCD instead of SMF-H with BCD | SMF · Pancreatic cancer microarray | classification score: 0.823 (0.06) -> 0.869 (0.02) | Helped |
| Use neural implementation instead of BCD | SMF-W · Pancreatic cancer microarray | classification score: 0.869 (0.02) -> 0.854 (0.04) | Hurt |
| Use neural implementation instead of BCD | SMF-W · Breast cancer microarray | classification score: 0.924 (0.01) -> 0.881 (0.02) | Hurt |
Surprising findings
- SMF-H converges faster for large xi while SMF-W converges faster for small xi, as predicted by the theory.
Limitations
- Results are local: local landscape and local consistency guarantees hold only near stationary points with sufficiently small parameter norms.
- Neural implementation only approximately implements the BCD algorithm.
- Experiments are limited to a few datasets and the analysis assumes bounded activations and compact constraint sets.
Experimental setup
Datasets: MNIST Employment Scam Aegean Dataset CuMiDa pancreatic cancer microarray CuMiDa breast cancer microarray
Benchmarks: MNIST (semi-synthetic) Employment Scam Aegean Dataset CuMiDa pancreatic cancer CuMiDa breast cancer
Metrics: training loss vs. time F-score accuracy relative reconstruction error classification accuracy
Baselines: LPGD (Lee et al. 2023) logistic regression on raw data MF-LR PCA-LR CNN FFNN Naive Bayes SVM Random Forest LDA NMF GSM
Builds on: Lee et al. 2023 low-rank projected gradient descent for SMF block majorization-minimization (Lyu & Li 2023) nonnegative matrix factorization
Closest papers by meaning
- Exponentially Convergent Algorithms for Supervised Matrix Factorization (NeurIPS 2023 · similarity 0.91)
- Online Convex Matrix Factorization with Representative Regions (NeurIPS 2019 · similarity 0.88)
- SMMF: Square-Matricized Momentum Factorization for Memory-Efficient Optimization (AAAI 2025 · similarity 0.86)
- On the Crucial Role of Initialization for Matrix Factorization (ICLR 2025 · similarity 0.85)
- Provable Acceleration of Nesterov's Accelerated Gradient for Asymmetric Matrix Factorization and Linear Neural Networks (NeurIPS 2024 · similarity 0.85)
- Statistically Optimal $K$-means Clustering via Nonnegative Low-rank Semidefinite Programming (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.