Research map / Optimization / Implicit bias and generalization

Differentiable Sparsity via $D$-Gating: Simple and Versatile Structured Penalization

NeurIPS 2025 · Optimization › Implicit bias and generalization · approach: Stability-based generalization analysis · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes D-Gating, a differentiable structured overparameterization that splits each group's weights into a primary vector and D-1 scalar gates, so SGD on the gated objective with weight decay is theoretically equivalent to L_{2,2/D} structured penalization.

Problem

Structured sparsity regularization (group lasso and non-convex L_{2,q} penalties) is non-differentiable, so it is incompatible with standard SGD and typically requires proximal optimizers or post-hoc pruning without guarantees.

Approach

Factor each weight group as a primary vector times D-1 scalar gating factors and apply plain L2 (weight decay) to all factors; the local minima of this smooth objective coincide with those of the non-smooth L_{2,2/D} penalized loss, and the gap decays exponentially under gradient flow.

Key results

• Theorem: local minima of the D-gated objective correspond exactly to local minima of the L_{2,2/D}-regularized loss, and the gap between objectives decays as C*exp(-4 lambda t / D) under gradient flow.
• On VGG-16 (CIFAR-10), D=4 gating reaches 18.88x FLOPs speed-up within 5% of max accuracy, versus 6.97x for L2,1 plus magnitude pruning.
• On Wine tree sparsity, near-full RMSE is matched using fewer than 5 trees out of 500 at lambda=3e-3.

Measured effects reported in the paper

Surprising findings

• Plain L2,1 optimization with SGD fails to reach exact sparsity and min/max head norms converge for large lambda in NanoGPT.
• D=2 gating is often inferior to D=3 or D=4 gating in feature selection and filter sparsity experiments.

Limitations

• Analysis is in the gradient flow limit; discrete SGD with large learning rates is not theoretically characterized.
• Integration into large-scale foundation model training pipelines is left open.
• Equivalence is for local minima; global behavior and the role of D beyond ablations are not fully characterized.

Experimental setup

Datasets: MNIST CIFAR-10 CIFAR-100 SVHN TinyShakespeare Wine

Benchmarks: MNIST CIFAR-10 CIFAR-100 SVHN TinyShakespeare Wine

Metrics: test accuracy validation perplexity FLOPs speed-up test RMSE active trees sparsity

Baselines: L2,1 with magnitude pruning network slimming global magnitude pruning LassoNet HSIC SNAMs proximal gradient descent

Builds on: group lasso LassoNet network slimming NanoGPT

Closest papers by meaning

• An Alternating Proximal Splitting Method with Global Convergence for Nonconvex Structured Sparsity Optimization (AAAI 2016 · similarity 0.87)
• SequentialAttention++ for Block Sparsification: Differentiable Pruning Meets Combinatorial Optimization (NeurIPS 2024 · similarity 0.87)
• Dynamic Sparse Training with Structured Sparsity (ICLR 2024 · similarity 0.87)
• Generalized Gradient Norm Clipping & Non-Euclidean $(L_0,L_1)$-Smoothness (NeurIPS 2025 · similarity 0.86)
• Global Minimizers of $\ell^p$-Regularized Objectives Yield the Sparsest ReLU Neural Networks (NeurIPS 2025 · similarity 0.86)
• Dynamic Sparse Training of Diagonally Sparse Networks (ICML 2025 · similarity 0.86)