Research map / Architectures and efficiency / RNNs and normalization

Why Geometric Continuity Emerges in Deep Neural Networks: Residual Connections and Rotational Symmetry Breaking

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Symmetry and equivariance in loss landscapes · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Identifies residual connections (cross-layer gradient coherence) and symmetry-breaking nonlinearities/normalization (pinning a shared coordinate frame) as the two mechanisms behind geometric continuity of principal singular vectors in weight matrices, with projection-specific patterns in transformers.

Problem

Weight matrices of adjacent layers in deep networks, and LLMs in particular, have aligned principal singular vectors (geometric continuity), but the origin of this property is unexplained.

Approach

Continuity arises from residual connections, which make cumulative gradients cross-layer coherent, combined with symmetry-breaking nonlinearities (element-wise activation, LayerNorm) that prevent per-layer SO(d) rotation drift; rotation-equivariant nonlinearity fails to retain continuity, isolating symmetry breaking rather than nonlinearity per se.

Key results

• Res+ReLU reaches weight v1 continuity 0.959 versus 0.217 for Res+None and 0.225 for rotation-equivariant Res+Radial, with Delta_GW of 0.003 vs 0.641 and 0.633.
• Removing the MLP activation in a small transformer drops Gate v1 continuity from 0.662 to 0.366 while perplexity rises from 41.9 to 43.2.
• In pretrained Llama-3.1-8B, mean adjacent-layer continuity is 0.82 for Q, 0.77 for K, 0.80 for Gate, 0.89 for O (u1), 0.74 for Down (u1), and 0.14 for V.

Surprising findings

• A nonlinear but rotation-equivariant activation (radial tanh-norm) fails to produce weight continuity, matching the linear baseline, so nonlinearity per se is not sufficient.
• Removing residual connections yields high u1 continuity (0.70) for NoRes+ReLU despite low v1 continuity, suggesting a separate forward-signal pathway.

Limitations

• Projection-specific mechanistic claims rest on a single direct ablation (Gate); Q/K/V explanations are indirect.
• Transformer experiments use a small 34M model on WikiText-103 with a single seed; rotation drift not verified at larger scales.
• Toy MLP results rely on MNIST-scale experiments with 16 layers and no bias.

Experimental setup

Datasets: MNIST EMNIST-Letters Fashion-MNIST SVHN CIFAR-10 CIFAR-100 WikiText-103

Benchmarks: MNIST WikiText-103

Metrics: cosine similarity of principal singular vectors (continuity) sigma^2-weighted alignment test accuracy perplexity gradient-weight coherence gap

Baselines: Res+None Res+Radial NoRes+ReLU Res+Tanh

Builds on: deep linear network dynamics intertwiner group analysis of activations residual Jacobian alignment

Closest papers by meaning

• Geometric Inductive Biases of Deep Networks: The Role of Data and Architecture (ICLR 2025 · similarity 0.84)
• Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks (ICML 2024 · similarity 0.83)
• Parameter symmetries determine representational geometry in overparameterized nonlinear networks (NeurIPS 2026 · similarity 0.83)
• The Geometric Wall: Manifold Structure Predicts Layerwise Sparse Autoencoder Scaling Laws (NeurIPS 2026 · similarity 0.83)
• Exploring and Exploiting the Asymmetric Valley of Deep Neural Networks (NeurIPS 2024 · similarity 0.83)
• Task structure and nonlinearity jointly determine learned representational geometry (ICLR 2024 · similarity 0.83)