Research map / Learning theory and algorithms / Matrix and tensor methods
Nuclear Norm Regularization for Deep Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that the Jacobian nuclear norm of a composed function f = g∘h can be exactly replaced by average squared Frobenius norms of Jg and Jh, then approximates these with a denoising-style, Jacobian-free estimator for scalable deep learning regularization.
Problem
Penalizing the Jacobian nuclear norm of neural networks is intractable in high dimensions because it requires computing large Jacobians and their SVDs.
Approach
For compositions f = g∘h, the nuclear norm of Jf equals the minimum over factorizations of half the squared Frobenius norms of the factors; the paper proves an exact equivalence of the learning problems and estimates the Frobenius terms via a noise-perturbation (denoising-style) finite difference, so no Jacobian or SVD is needed.
Key results
- On ImageNet with sigma=1, the unsupervised regularized denoiser reaches 23.10 dB PSNR versus 23.37 dB for the supervised denoiser and 21.26 dB for BM3D.
- On CBSD68 with sigma=2, the method reaches 19.10 dB PSNR, versus 19.54 dB supervised and 16.77 dB BM3D.
- In the ROF validation, the Jacobian-free regularized objective converges to nearly the same value as the direct nuclear norm penalty, as predicted by the equivalence theorem.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on noisy images with Jacobian nuclear norm regularization instead of BM3D | Unet denoiser · ImageNet, σ=1 | PSNR (dB): 21.26 ± 2.81 -> 23.10 ± 3.12 | Helped |
| Train on noisy images with Jacobian nuclear norm regularization instead of BM3D | Unet denoiser · ImageNet, σ=2 | PSNR (dB): 18.71 ± 2.33 -> 21.05 ± 2.85 | Helped |
| Train on noisy images with Jacobian nuclear norm regularization instead of BM3D | Unet denoiser · CBSD68, σ=1 | PSNR (dB): 19.42 ± 1.88 -> 21.08 ± 2.04 | Helped |
| Train on noisy images with Jacobian nuclear norm regularization instead of BM3D | Unet denoiser · CBSD68, σ=2 | PSNR (dB): 16.77 ± 1.30 -> 19.10 ± 1.80 | Helped |
| Train on noisy images with Jacobian nuclear norm regularization instead of N2N | Unet denoiser · ImageNet, σ=1 | PSNR (dB): 23.12 ± 3.05 -> 23.10 ± 3.12 | No clear effect |
| Train on noisy images with Jacobian nuclear norm regularization instead of N2N | Unet denoiser · ImageNet, σ=2 | PSNR (dB): 21.21 ± 3.02 -> 21.05 ± 2.85 | No clear effect |
| Train on noisy images with Jacobian nuclear norm regularization instead of N2N | Unet denoiser · CBSD68, σ=1 | PSNR (dB): 20.37 ± 1.71 -> 21.08 ± 2.04 | Helped |
| Train on noisy images with Jacobian nuclear norm regularization instead of N2N | Unet denoiser · CBSD68, σ=2 | PSNR (dB): 19.37 ± 1.89 -> 19.10 ± 1.80 | No clear effect |
| Train exclusively on noisy images with Jacobian nuclear norm regularization instead of supervised clean-noisy pairs | Unet denoiser · ImageNet, σ=1 | PSNR (dB): 23.37 ± 3.25 -> 23.10 ± 3.12 | No clear effect |
| Train exclusively on noisy images with Jacobian nuclear norm regularization instead of supervised clean-noisy pairs | Unet denoiser · ImageNet, σ=2 | PSNR (dB): 21.39 ± 2.97 -> 21.05 ± 2.85 | No clear effect |
| Train exclusively on noisy images with Jacobian nuclear norm regularization instead of supervised clean-noisy pairs | Unet denoiser · CBSD68, σ=1 | PSNR (dB): 21.62 ± 2.28 -> 21.08 ± 2.04 | Hurt |
| Train exclusively on noisy images with Jacobian nuclear norm regularization instead of supervised clean-noisy pairs | Unet denoiser · CBSD68, σ=2 | PSNR (dB): 19.54 ± 1.95 -> 19.10 ± 1.80 | Hurt |
Surprising findings
- A denoiser trained only on noisy images, without clean targets, nearly matches a supervised denoiser and performs comparably to Noise2Noise.
Limitations
- Hutchinson-based Frobenius estimator introduces error, giving slightly diffuse boundaries in ROF solutions.
- Experiments are limited to small-scale synthetic problems and a single denoising setting plus an autoencoder demonstration.
- Equivalence is proved for the continuous setting with smooth functions; finite-sample/practical gap is not analyzed in depth.
Experimental setup
Datasets: ImageNet CBSD68 CelebA
Benchmarks: ImageNet validation (100 random crops) CBSD68
Metrics: PSNR absolute error on ROF problem objective values Jacobian singular value decay
Baselines: BM3D Supervised denoiser Noise2Noise beta-VAE Unregularized autoencoder
Builds on: Srebro nuclear norm factorization identity Noise2Noise Hutchinson trace estimator Unet
Closest papers by meaning
- On Dropout and Nuclear Norm Regularization (ICML 2019 · similarity 0.88)
- A Kernel Perspective for Regularizing Deep Neural Networks (ICML 2019 · similarity 0.86)
- How DNNs break the Curse of Dimensionality: Compositionality and Symmetry Learning (ICLR 2025 · similarity 0.86)
- Neglected Hessian component explains mysteries in sharpness regularization (NeurIPS 2024 · similarity 0.84)
- Scalable Optimization in the Modular Norm (NeurIPS 2024 · similarity 0.84)
- Cauchy-Schwarz Regularizers (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.