Research map / Architectures and efficiency / RNNs and normalization

Revitalizing SVD for Global Covariance Pooling: Halley’s Method to Overcome Over-Flattening

NeurIPS 2025 · Architectures and efficiency › RNNs and normalization · approach: Symmetry and equivariance in loss landscapes

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's abstract only. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a Halley's-method-based approach to SVD-based global covariance pooling that addresses over-flattening of eigenvalues in exact SVD GCP, while retaining stable early training.

Problem

Global Covariance Pooling (GCP) methods trade off across training: Newton-Schulz-based iSQRT-COV stabilizes early learning but is limited later, while exact or near-exact SVD methods suffer from over-flattening of the spectrum during training.

Approach

Revives SVD for GCP by replacing the standard SVD-based matrix square-root gradient with a Halley's-method iteration, which mitigates over-flattening of the covariance spectrum and avoids the gradient instability of exact SVD.

Experimental setup

Baselines: iSQRT-COV

Builds on: iSQRT-COV global covariance pooling

Closest papers by meaning

• Towards Faster Training of Global Covariance Pooling Networks by Iterative Matrix Square Root Normalization (CVPR 2018 · similarity 0.90)
• DropCov: A Simple yet Effective Method for Improving Deep Architectures (NeurIPS 2022 · similarity 0.89)
• What Deep CNNs Benefit From Global Covariance Pooling: An Optimization Perspective (CVPR 2020 · similarity 0.89)
• Global Second-Order Pooling Convolutional Networks (CVPR 2019 · similarity 0.88)
• Temporal-attentive Covariance Pooling Networks for Video Recognition (NeurIPS 2021 · similarity 0.87)
• Improving Visual Prompt Tuning by Gaussian Neighborhood Minimization for Long-Tailed Visual Recognition (NeurIPS 2024 · similarity 0.84)