Research map / Architectures and efficiency / RNNs and normalization
Correlations in the Data Lead to Semantically Rich Feature Geometry Under Superposition
Lucas Prieto, Edward Stevinson, Melih Barsbey, Tolga Birdal, Pedro A. M. Mediano
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| tighten the bottleneck (small latent size m) in a ReLU autoencoder (ablation) | ReLU AE on WikiText-BOWS · WikiText-BOWS V=10k | semantic clustering of word embeddings (UMAP) | Improved |
| add weight decay when training the ReLU AE with a large latent size (ablation) | ReLU AE, m=800 · WikiText-BOWS V=10k | semantic clustering of word embeddings (UMAP): wd 0 -> wd 4 at m=800 | Improved |
| increase latent size m toward the number of features in the cyclic-covariance synthetic data (ablation) | ReLU AE, d=12 synthetic · Synthetic cyclic covariance | weight geometry | Mixed |
| ablate the subspace orthogonal to value-coding features (keep VC only) (ablation) | map model, MAP · MAP | accuracy (%): 97.94 -> 93.16 | No clear effect |
| ablate the subspace orthogonal to value-coding features (keep VC only) (ablation) | modular addition model, Key-freq · Key-freq | accuracy (%): 100.00 -> 93.99 | No clear effect |
| remove the value-coding coordinates (ablation) | map model, MAP · MAP | accuracy (%): 97.94 -> 22.43 | Hurt |
| remove the value-coding coordinates (ablation) | modular addition model, Key-freq · Key-freq | accuracy (%): 100.00 -> 3.11 | Hurt |
| add contextual (correlated) words for the target word 'Beatles' | ReLU AE decoder · WikiText-BOWS validation | per-word R2: 81% of samples better than one-hot | Improved |
| increase latent size so month features become closer to orthogonal (ablation) | ReLU AE, months vs Roman numerals · WikiText-BOWS | structure in weights | Mixed |
Closest papers by meaning
- Semantic Optimal Transport for Sparse Autoencoder Feature Matching and Circuit Compression (NeurIPS 2026 · similarity 0.86)
- Superposition Yields Robust Neural Scaling (NeurIPS 2025 · similarity 0.86)
- Invariant Features in Language Models: Geometric Characterization and Model Attribution (NeurIPS 2026 · similarity 0.85)
- A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders (NeurIPS 2025 · similarity 0.85)
- Dense SAE Latents Are Features, Not Bugs (NeurIPS 2025 · similarity 0.85)
- Rethinking Evaluation of Sparse Autoencoders through the Representation of Polysemous Words (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
