Research map / Probabilistic and scientific modeling / Dynamical systems
The Geometry of Projection Heads: Conditioning, Invariance, and Collapse
Faris Chaudhry
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the projection head on top of the backbone, orbit spread | SimCLR ResNet-18 · CIFAR-10 | mean orbit spread (x1e-2): 2.25 -> 0.10 (21.85x collapse) | Improved |
| evaluate head output instead of backbone, linear probing | SimCLR ResNet-18 · CIFAR-10 | linear probing accuracy (%): 52.27 -> 37.55 | Hurt |
| evaluate head output instead of backbone, MLP probing | SimCLR ResNet-18 · CIFAR-10 | MLP probing accuracy (%): 55.46 -> 43.56 | Hurt |
| evaluate head output instead of backbone, nonlinearity gap | SimCLR ResNet-18 · CIFAR-10 | MLP-linear probing gap (%): 3.19 -> 6.01 | Mixed |
| head vs backbone local curvature | SimCLR ResNet-18 · CIFAR-10 | local curvature: 0.125 -> 0.242 (1.93x) | Improved |
| head vs backbone intra-orbit distance | SimCLR ResNet-18 · CIFAR-10 | intra-orbit distance: 0.211 -> 0.044 | Improved |
| head vs backbone class/orbit ratio | SimCLR ResNet-18 · CIFAR-10 | class/orbit distance ratio: 2.04 -> 2.50 | Improved |
| use smooth (Swish) activation instead of ReLU in the head, collapse recovery (ablation) | SimSiam head · CIFAR-10 pseudo-collapsed init | representation variance escape | Improved |
Closest papers by meaning
- The Geometric Wall: Manifold Structure Predicts Layerwise Sparse Autoencoder Scaling Laws (NeurIPS 2026 · similarity 0.86)
- Investigating the Benefits of Projection Head for Representation Learning (ICLR 2024 · similarity 0.85)
- Geometric Inductive Biases of Deep Networks: The Role of Data and Architecture (ICLR 2025 · similarity 0.85)
- The Impact of Geometric Complexity on Neural Collapse in Transfer Learning (NeurIPS 2024 · similarity 0.85)
- No Wrong Turns: The Simple Geometry Of Neural Networks Optimization Paths (ICML 2024 · similarity 0.84)
- Projection Head is Secretly an Information Bottleneck (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
