Research map / Optimization / Optimizers

Matrix-Free Natural Gradient via Stochastic Truncated Metrics

NeurIPS 2026 · Optimization › Optimizers · approach: Natural gradient and Fisher approximations

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's abstract only. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a matrix-free natural gradient optimizer built on an unbiased stochastic estimator of the Fisher information matrix with a truncated metric.

Problem

Natural gradient descent is theoretically grounded but existing implementations either use structural approximations of the curvature matrix or incur large computational and memory costs, making direct application hard at deep-network scale.

Approach

Builds an optimizer on an unbiased stochastic estimator of the Fisher information matrix that employs a truncated metric, avoiding explicit matrix construction (matrix-free) and structural approximations.

Closest papers by meaning

• Exact natural gradient in deep linear networks and its application to the nonlinear case (NeurIPS 2018 · similarity 0.89)
• Fast Convergence of Natural Gradient Descent for Over-Parameterized Neural Networks (NeurIPS 2019 · similarity 0.88)
• Near-optimal Sketchy Natural Gradients for Physics-Informed Neural Networks (ICML 2025 · similarity 0.87)
• A Layer-Wise Natural Gradient Optimizer for Training Deep Neural Networks (NeurIPS 2024 · similarity 0.86)
• Structured Inverse-Free Natural Gradient Descent: Memory-Efficient & Numerically-Stable KFAC (ICML 2024 · similarity 0.86)
• The Global Convergence Time of Stochastic Gradient Descent in Non-Convex Landscapes: Sharp Estimates via Large Deviations (ICML 2025 · similarity 0.86)