Research map / Label-efficient and robust learning / Recognition and long-tail
Unveiling Transformer Perception by Exploring Input Manifolds
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SiMEC and SiMExp, algorithms that use eigendecomposition of the pullback metric (Jacobian of a Transformer) to walk along equivalence classes of the input space or move across them, with an interpretation step that decodes embeddings back to images or text.
Problem
Understanding how Transformers partition their input space: finding inputs that receive the same class probability distribution as an original input (equivalence classes) and inputs that move to different classes, in a human-interpretable way.
Approach
Treats each Transformer layer as a smooth map between manifolds and uses the pullback of the output-space metric through the Jacobian; the null eigenvectors of this singular metric give directions that stay within an equivalence class (SiMEC), while non-null eigenvectors move across classes (SiMExp). Embeddings are decoded back to images or text via invertible embedding layers or an MLM-head decod
Key results
- SiMEC yields zero or near-zero class ranking changes (e.g., 0.0 on CIFAR10 and MHS at eta=1) while SiMExp produces many more (e.g., 18.70 on CIFAR10 and 3.64 on MHS at eta=10 per 1000 iterations).
- SiMExp explores a hypervolume roughly an order of magnitude (10^1) larger than SiMEC, with Welch t-test p-values below 10^-3.
- Median Wasserstein distance between embedding and decoded-interpretation predictions is 0.0 for SiMEC versus 0.049 for SiMExp.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Explore nonzero-eigenvalue directions instead of zero-eigenvalue directions | SiMExp vs SiMEC · MNIST, CIFAR10, WinoBias, and MHS | class ranking changes per 1000 iterations: 0.03\,(0.18) vs 1.31\,(1.64); 0.0 vs 2.60\,(3.35)**; 0.0 vs 0.13\,(0.33); 0.0 vs 0.75\,(0.89)** | Helped |
| Increase the step multiplier from η = 1 to η = 10 (ablation) | SiMExp · MNIST, CIFAR10, WinoBias, and MHS | class ranking changes per 1000 iterations: 1.31\,(1.64) -> 11.91\,(11.87)**; 2.60\,(3.35)** -> 18.70\,(15.90)**; 0.13\,(0.33) -> 3.25\,(2.41)**; 0.75\,(0.89)** -> | Helped |
| Increase the step multiplier from η = 1 to η = 10 (ablation) | SiMEC · MNIST, CIFAR10, WinoBias, and MHS | class ranking changes per 1000 iterations: 0.03\,(0.18) -> 0.07\,(0.25); 0.0 -> 0.12\,(0.33); 0.0 -> 0.0; 0.0 -> 0.0 | Mixed |
| Explore with SiMExp rather than SiMEC | SiMEC and SiMExp · Image and text datasets | explored hypervolume: 10^1 | Helped |
| Use SiMEC rather than SiMExp for producing interpretation outputs | SiMEC and SiMExp · Interpretation outputs | median Wasserstein distance (p = 1): $0.0$ vs $0.049$ | Helped |
Surprising findings
- A small change in background pixels of a CIFAR10 image flips the ViT prediction from cat to dog while the decoded images still look like cats to humans.
- Around 10.9% of SiMExp interpretation predictions that initially disagree with embedding predictions realign as exploration progresses.
Limitations
- Eigendecomposition scales as O(n^3), which the authors note is a scalability concern for large embedding dimensions (10^3-10^4).
- Numerical approximation and decoder noise cause mismatches between embedding predictions and decoded interpretation predictions, especially for SiMExp.
- The feasible region is an approximate L-infinity bound rather than the exact input space.
Experimental setup
Datasets: MNIST CIFAR10 WinoBias Measuring-Hate-Speech (MHS)
Benchmarks: MNIST CIFAR10 WinoBias Measuring-Hate-Speech (MHS)
Metrics: class probability change class ranking change prediction change explored hypervolume ratio Wasserstein distance between embedding and interpretation pr runtime per iteration
Baselines: gradient-based Gaussian noise approach
Builds on: Riemannian geometry for neural networks (foliation of input space via pullback m Vision Transformer BERT
Closest papers by meaning
- Beyond Components: Singular Vector-Based Interpretability of Transformer Circuits (NeurIPS 2025 · similarity 0.85)
- A unified framework for establishing the universal approximation of transformer-type architectures (NeurIPS 2025 · similarity 0.84)
- Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding (NeurIPS 2025 · similarity 0.84)
- Predictive Geometry of Hidden Trajectories in Transformers (NeurIPS 2026 · similarity 0.84)
- Transformer Layers as Painters (AAAI 2025 · similarity 0.84)
- Hide & Seek: Transformer Symmetries Obscure Sharpness & Riemannian Geometry Finds It (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.