AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

Correlations in the Data Lead to Semantically Rich Feature Geometry Under Superposition

Lucas Prieto, Edward Stevinson, Melih Barsbey, Tolga Birdal, Pedro A. M. Mediano

ICLR 2026 · Architectures and efficiency › RNNs and normalization · approach: Symmetry and equivariance in loss landscapes

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
tighten the bottleneck (small latent size m) in a ReLU autoencoder (ablation)ReLU AE on WikiText-BOWS · WikiText-BOWS V=10ksemantic clustering of word embeddings (UMAP)Improved
add weight decay when training the ReLU AE with a large latent size (ablation)ReLU AE, m=800 · WikiText-BOWS V=10ksemantic clustering of word embeddings (UMAP): wd 0 -> wd 4 at m=800Improved
increase latent size m toward the number of features in the cyclic-covariance synthetic data (ablation)ReLU AE, d=12 synthetic · Synthetic cyclic covarianceweight geometryMixed
ablate the subspace orthogonal to value-coding features (keep VC only) (ablation)map model, MAP · MAPaccuracy (%): 97.94 -> 93.16No clear effect
ablate the subspace orthogonal to value-coding features (keep VC only) (ablation)modular addition model, Key-freq · Key-freqaccuracy (%): 100.00 -> 93.99No clear effect
remove the value-coding coordinates (ablation)map model, MAP · MAPaccuracy (%): 97.94 -> 22.43Hurt
remove the value-coding coordinates (ablation)modular addition model, Key-freq · Key-freqaccuracy (%): 100.00 -> 3.11Hurt
add contextual (correlated) words for the target word 'Beatles'ReLU AE decoder · WikiText-BOWS validationper-word R2: 81% of samples better than one-hotImproved
increase latent size so month features become closer to orthogonal (ablation)ReLU AE, months vs Roman numerals · WikiText-BOWSstructure in weightsMixed

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy