Research map / Architectures and efficiency / RNNs and normalization

The Geometric Wall: Manifold Structure Predicts Layerwise Sparse Autoencoder Scaling Laws

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Sparse dictionary representations · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Fits per-layer no-floor SAE scaling-law surfaces across all 68 layers of Gemma 2 2B and 9B (844 Gemma Scope checkpoints) and regresses the width exponents and fitted floors on four layerwise activation-manifold geometry summaries.

Problem

SAE reconstruction scaling laws were fitted at a single layer of GPT-4; it is unknown whether they vary across layers and what predicts that variation, given that activation manifolds are curved and their intrinsic dimension varies with depth.

Approach

Treats the per-layer SAE width-sparsity scaling exponent and asymptotic floor as functions of layerwise activation-manifold geometry (intrinsic dimension, multi-scale curvature, tangent variation, heterogeneity), via a two-stage fit-then-regress analysis with cross-model transfer, and interprets the floor as a geometry-dependent "wall" from a second-order curvature residual.

Key results

• Multi-scale curvature alone explains R^2 = 0.929 (LOO 0.869) of per-layer width exponent alpha(k=50) across 42 layers of Gemma 2 9B (H1 vs H0 F=523.6).
• Regression coefficients learned on one Gemma 2 model transfer to predict the other model's per-layer width exponent, per the paper's cross-model transfer analysis.
• At the six showcase layers, fitted asymptotic floor B(k=50) ranks consistently with intrinsic dimension and curvature ordering within each model.

Surprising findings

• Width-scaling exponent at depth 5/6 lands within about 0.03 of the GPT-4 reported value of 0.181 despite different architecture (JumpReLU vs TopK).
• Adding further geometric features beyond multi-scale curvature adds little, with AIC/BIC favouring simpler models due to feature collinearity.

Limitations

• Floor identifiable only at six showcase layers per model, so the floor-geometry relation is a ranking rather than a formal regression.
• Per-layer exponents fitted with a no-floor model that underestimates the asymptotic exponent due to floor attenuation.
• Geometry estimated with Euclidean extrinsic estimators rather than the pullback Fisher-Rao metric, which is left as future work.

Experimental setup

Datasets: C4 validation set WikiText-103 Gemma Scope SAE checkpoints

Benchmarks: C4 validation WikiText-103

Metrics: R^2 leave-one-layer-out R^2 AIC BIC F-test NMSE cosine distance permutation p-value

Baselines: H0 geometry-invariant null (single constant exponent) single-layer GPT-4 scaling law exponent (Gao et al.)

Builds on: SAE scaling laws (Gao et al.) JumpReLU SAE Gemma Scope TWO-NN intrinsic dimension

Closest papers by meaning

• Sparse Autoencoders, Again? (ICML 2025 · similarity 0.88)
• Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometry (NeurIPS 2025 · similarity 0.87)
• Task-Induced Riemannian Metrics for Vision Transformer Feature Spaces (NeurIPS 2026 · similarity 0.87)
• Graph Geometry-Preserving Autoencoders (ICML 2024 · similarity 0.87)
• Semantic Optimal Transport for Sparse Autoencoder Feature Matching and Circuit Compression (NeurIPS 2026 · similarity 0.86)
• Dynamics of the Transformer Residual Stream: Coupling Spectral Geometry to Network Topology (NeurIPS 2026 · similarity 0.86)