Research map / Architectures and efficiency / RNNs and normalization
Geometric Inductive Biases of Deep Networks: The Role of Data and Architecture
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the Geometric Invariance Hypothesis, arguing that a network's input-space geometry (average geometry and its evolution at initialization) changes only along architecture-dependent directions, which explains why ResNets fail to generalize on some planar non-linear tasks while MLPs do not.
Problem
Understanding how a neural network's input-space geometry evolves during training and how architecture and data induce geometric inductive biases that affect generalization.
Approach
Defines average geometry (expected outer product of input gradients) and average geometry evolution, shows evolution at initialization is approximately the data covariance projected onto the initial geometry, and conjectures that geometry is invariant in directions weakly correlated with the initial geometry (GIH).
Key results
- ResNet18 fails to generalize on a planar concentric-circle task depending on orientation while MLPs succeed for any orthogonal orientation (qualitative, Figure 1).
- Removing the lowest-correlated directions with the initial geometry causes smaller accuracy drops than random removal, e.g., ResNet18 at 90% removal 54.4% (score-based) vs 52.4% (random).
- Theorem: for a single-hidden-layer wide MLP with ReLU, the initial evolution approaches the data covariance S as D and width grow; linear CNN with pooling gives G S G.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| score-based (lowest GIH geometry score) data removal instead of random removal (ablation) | LeNet · CIFAR-10, 30% removed | test accuracy (%): +0.8 (62.5 vs 61.7) | Helped |
| score-based data removal instead of random removal (ablation) | ResNet18 · CIFAR-10, 30% removed | test accuracy (%): +1.2 (77.0 vs 75.8) | Helped |
| score-based data removal instead of random removal (ablation) | ViT · CIFAR-10, 30% removed | test accuracy (%): +0.9 (58.4 vs 57.5) | Helped |
| score-based data removal instead of random removal (ablation) | LeNet, ResNet18, ViT · CIFAR-10, 10-90% removed | test accuracy (%): score-based removal higher at every removal level in appendix table | Helped |
| orthogonal projection removing features with lowest geometry correlation (GIH directions) instead of random directions (ablation) | LeNet/ResNet18/ViT · CIFAR-2 feature elimination | test accuracy | Helped |
| gradient-flow/GIH-based choice of discriminant direction u (large eigenvalue of G S G) versus small eigenvalue | MLP, LeNet, ResNet18, ViT · CIFAR-10 synthetic linear/quadratic/sinusoidal labels | test accuracy | Helped |
| training on data with flip(G) covariance instead of G covariance | CNN/ResNet-type models · synthetic Gaussian data | average geometry change (geometric velocity) and training accuracy | No clear effect |
| ResNet18 vs MLP architecture (ResNet fails to generalize depending on plane orientation, MLP does not) | ResNet18 vs MLP · synthetic two-concentric-circles problem in high dimension | generalization across plane orientations | Hurt |
| relabeling CIFAR-2 with random labels for the independence-of-labels experiment | MLP, LeNet, ResNet18, ViT · CIFAR-2 random labels | correlation between G^t and G S G | No clear effect |
Surprising findings
- Sample removal based on geometry score gives only modest gains (at most about 2%) over random removal.
- For non-isotropic models, the model learns the non-linear component when the linear component is weakly correlated with the geometry, contrary to the simplicity bias prediction.
Limitations
- Theory is developed under gradient flow and SSE loss while experiments use cross-entropy and SGD.
- GIH is analyzed mainly at initialization; later-training behavior is conjectured and supported empirically.
- Theoretical results are for simplified architectures (single hidden layer, linear CNN with pooling) rather than full ResNets or ViTs.
Experimental setup
Datasets: CIFAR-10 CIFAR-2 (animal vs non-animal subset) synthetic Gaussian data
Benchmarks: CIFAR-10 CIFAR-2 synthetic planar classification
Metrics: test accuracy generalization gap correlation between average geometry matrices geometric velocity
Baselines: MLP LeNet ResNet18 ViT NADs
Builds on: neural anisotropy directions simplicity bias hypothesis neural tangent kernel
Closest papers by meaning
- Empirical Study of the Topology and Geometry of Deep Networks (CVPR 2018 · similarity 0.88)
- Neural Feature Geometry Evolves as Discrete Ricci Flow (ICML 2026 · similarity 0.88)
- The Geometry of Neural Nets' Parameter Spaces Under Reparametrization (NeurIPS 2023 · similarity 0.88)
- Task structure and nonlinearity jointly determine learned representational geometry (ICLR 2024 · similarity 0.87)
- Geometry-Aware Similarity Metrics for Neural Representations on Riemannian and Statistical Manifolds (NeurIPS 2026 · similarity 0.86)
- Estimating Shape Distances on Neural Representations with Limited Samples (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.