Research map / Learning theory and algorithms / Kernels and regression theory

A path-norm toolkit for modern networks: consequences, promises and challenges

ICLR 2024 · Learning theory and algorithms › Kernels and regression theory · approach: Generalization theory of deep networks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds a path-norm toolkit (path-lifting, mixed path-norms, contraction lemmas, peeling) for general DAG ReLU networks with biases, skip connections and k-max-pooling, yielding an L1 path-norm generalization bound and showing it is far too large for pretrained ImageNet ResNets.

Problem

Path-norm tools and path-norm generalization bounds were limited to simple models (layered fully-connected, no pooling, skip connections or biases), so their promises could not be tested on modern networks.

Approach

Extends path-lifting and path-norms to general DAG ReLU networks with biases, identity and k-max-pooling neurons, proves mixed path-norm Lipschitz bounds and single-forward-pass computation, and derives a generalization bound depending only on the global L1 path-norm via new contraction lemmas and a peeling argument.

Key results

• The L1 path-norm of pretrained PyTorch ResNet18 is about 1.3e30, so the generalization bound is roughly 30 orders of magnitude too large to be informative.
• Iterative magnitude pruning of ResNet18 decreases the L1 path-norm from about 1e30 to about 1e17 after 19 pruning iterations.
• The generalization bound with the sharpened analysis for ResNet18 to ResNet152 gives constants 4CB/sqrt(n) between 0.06 and 0.23 across depths.

Measured effects reported in the paper

Surprising findings

• Pretrained ResNets have L1 path-norms far too large for path-norm generalization bounds to be informative, despite good test accuracy.
• Products of operator norms can be arbitrarily large while the path-norm is zero, so operator-norm bounds decouple layers in a harmful way.

Limitations

• Bound does not cover attention, or activations other than ReLU, identity and k-max-pooling (e.g., SiLU, Hardswish).
• Bound is worst-case over all inputs and may be K times too loose for k-max-pooling neurons.
• Current numerical bounds are far from informative for dense ResNets.

Experimental setup

Datasets: ImageNet1k

Benchmarks: ImageNet1k

Metrics: L1 path-norm L2 path-norm L4 path-norm generalization bound value top-1 accuracy

Baselines: product of operator norms bounds Golowich et al. L2/L1 path-norm bounds covering-number bounds for Lipschitz functions

Builds on: path-norm generalization bounds (Neyshabur et al.) peeling argument for layered ReLU networks (Golowich et al.) path-lifting (Stock and Gribonval) Ledoux-Talagrand contraction

Closest papers by meaning

• Path Regularization: A Convexity and Sparsity Inducing Regularization for Parallel ReLU Networks (NeurIPS 2023 · similarity 0.89)
• Capacity Control of ReLU Neural Networks by Basis-Path Norm (AAAI 2019 · similarity 0.89)
• A Rescaling-Invariant Lipschitz Bound Based on Path-Metrics for Modern ReLU Network Parameterizations (ICML 2025 · similarity 0.89)
• Norm-based Generalization Bounds for Sparse Neural Networks (NeurIPS 2023 · similarity 0.87)
• Generalization of Scaled Deep ResNets in the Mean-Field Regime (ICLR 2024 · similarity 0.85)
• Generalization Bounds for Rank-sparse Neural Networks (NeurIPS 2025 · similarity 0.85)