Research map / Architectures and efficiency / RNNs and normalization
Deep sequence models tend to memorize geometrically; it is unclear why
Shahriar Noroozizadeh, Vaishnavh Nagarajan, Elan Rosenfeld, Sanjiv Kumar
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| memorize the graph in weights instead of giving it in context | next-token trained Transformer on path-star · path-star task | first-token accuracy | Improved |
| train only on first-token loss, removing supervision of later path tokens (ablation) | Transformer · in-weights path-star, unseen paths | first-token accuracy | No clear effect |
| add reverse edges to edge-memorization data (ablation) | Transformer · large in-weights path-star graphs | first-token accuracy | Improved |
Closest papers by meaning
- Geometric Factual Recall in Transformers (NeurIPS 2026 · similarity 0.88)
- A Geometric Framework for Understanding Memorization in Generative Models (ICLR 2025 · similarity 0.88)
- Factual recall in linear associative memories: sharp asymptotics and mechanistic insights (NeurIPS 2026 · similarity 0.86)
- Tracing the Representation Geometry of Language Models from Pretraining to Post-training (NeurIPS 2025 · similarity 0.86)
- The Geometry of Forgetting: Temporal Knowledge Drift as an Independent Axis in LLM Representations (NeurIPS 2026 · similarity 0.85)
- The Geometric Inductive Bias of Grokking: Bypassing Phase Transitions via Architectural Topology (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
