Research map / Architectures and efficiency / Attention and transformers
Hide & Seek: Transformer Symmetries Obscure Sharpness & Riemannian Geometry Finds It
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes geodesic sharpness, an adaptive sharpness measure defined on the Riemannian quotient manifold that removes parameter symmetries (including the GL(h) symmetry of attention), and shows it correlates with generalization for diagonal nets, ViTs fine-tuned on ImageNet, and BERT on MNLI.
Problem
Sharpness measures fail to predict generalization in transformers because attention layers have rich continuous symmetries (GL(h)) that make parameter-space perturbations ambiguous; existing adaptive sharpness only corrects for simpler rescaling symmetries.
Approach
Define sharpness as the maximal loss increase over a geodesic ball on the symmetry quotient manifold (horizontal lifts, symmetry-compatible metrics), approximated to second order with Christoffel symbols; first-order truncation recovers existing adaptive sharpness measures.
Key results
- On MNLI with 35 fine-tuned BERT models, geodesic sharpness (mixed metric) reaches tau = 0.38 versus 0.06 for adaptive sharpness.
- On ImageNet fine-tuned CLIP ViT-B/32 models, inverse-metric geodesic sharpness reaches tau = -0.71 versus -0.41 for adaptive sharpness.
- On diagonal networks, the mixed-metric geodesic sharpness reaches tau = -0.86 versus -0.68 for adaptive sharpness.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use geodesic sharpness with the inv metric instead of adaptive sharpness | Diagonal networks · Diagonal networks | Kendall rank correlation (τ): -0.68 -> -0.83 | Helped |
| Use geodesic sharpness with the mix metric instead of adaptive sharpness | Diagonal networks · Diagonal networks | Kendall rank correlation (τ): -0.68 -> -0.86 | Helped |
| Use geodesic sharpness with the inv metric instead of adaptive sharpness | CLIP ViT-B/32 · ImageNet | Kendall rank correlation (τ): -0.41 -> -0.71 | Helped |
| Use geodesic sharpness with the mix metric instead of adaptive sharpness | CLIP ViT-B/32 · ImageNet | Kendall rank correlation (τ): -0.41 -> -0.7 | Helped |
| Use geodesic sharpness with the inv metric instead of adaptive sharpness | BERT · MNLI | Kendall rank correlation (τ): 0.06 -> 0.28 | Helped |
| Use geodesic sharpness with the mix metric instead of adaptive sharpness | BERT · MNLI | Kendall rank correlation (τ): 0.06 -> 0.38 | Helped |
| Use the mix metric instead of the inv metric for geodesic sharpness (ablation) | BERT · MNLI | Kendall rank correlation (τ): 0.28 -> 0.38 | Helped |
Surprising findings
- The sign of the correlation between geodesic sharpness and generalization varies across settings, being negative on ImageNet and diagonal networks but positive on MNLI.
- Adaptive sharpness anti-correlates with generalization on diagonal networks, contrary to the common view that flatter models generalize better.
Limitations
- Taking the quotient may be computationally expensive or intractable for other symmetries.
- The sign of the correlation varies across tasks and data, limiting its utility as a predictive tool.
- Geodesics lack analytical solutions for attention layers and must be approximated to second order.
Experimental setup
Datasets: ImageNet-1k MNLI synthetic regression data for diagonal networks
Benchmarks: ImageNet-1k MNLI
Metrics: Kendall rank correlation tau generalization gap
Baselines: adaptive sharpness relative sharpness
Builds on: adaptive sharpness Auto-PGD Kendall rank correlation
Closest papers by meaning
- Task-Induced Riemannian Metrics for Vision Transformer Feature Spaces (NeurIPS 2026 · similarity 0.87)
- Generalized Linear Mode Connectivity for Transformers (NeurIPS 2025 · similarity 0.86)
- Learning Theory of Transformers: Local-to-Global Approximation via Softmax Partition of Unity (NeurIPS 2026 · similarity 0.86)
- Curved Representation Space of Vision Transformers (AAAI 2024 · similarity 0.85)
- On the Role of Attention Masks and LayerNorm in Transformers (NeurIPS 2024 · similarity 0.85)
- Building Transformation Layers for Riemannian Neural Networks (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.