Research map / Large language models / Language, safety and interpretability
Large language models transition from integrating across position-yoked, exponential windows to structure-yoked, power-law windows
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| go from layer 1 to layer 11 | GPT-2 · word-swap integration windows | convex combination parameter c (power-law share): 0.108 -> 0.274 | Helped |
| go from layer 3 to layer 12 | GPT-2 · integration windows | power-law parameter a (lower = longer window): 1.022 -> 0.512 | Helped |
| go from layer 2 to layer 6 | GPT-2 · integration windows | exponential parameter b (lower = longer window): 4.386 -> 0.697 | Helped |
| go from layer 1 to layer 12 | GPT-2 · sentence-boundary sequences | structure-yoking metric delta[k]: 0.003 -> 0.074 | Helped |
| use convex exponential plus power-law fit instead of single exponential or power-law (ablation) | window fitting · GPT-2 integration windows | test MSE of fit | Helped |
| untrained GPT-2 instead of trained (ablation) | GPT-2 · integration windows | window shape across layers | Hurt |
| untrained GPT-2 instead of trained (ablation) | GPT-2 · sentence boundaries | structure-yoking metric | Hurt |
| apply analysis to LLaMA (ablation) | LLaMA · integration windows | timescale across layers | Helped |
| apply analysis to roBERTa (ablation) | roBERTa · integration windows | window shape | Mixed |
| change sentence length from 12 to 8 or 36 words (ablation) | GPT-2 · structure yoking | boundary metric | No clear effect |
| use noun phrases instead of sentences (ablation) | GPT-2 · structure yoking | boundary metric | No clear effect |
| use natural 84-word sequences with centered sentence (ablation) | GPT-2 · structure yoking | boundary metric | Mixed |
Closest papers by meaning
- The Emergence of Abstract Thought in Large Language Models Beyond Any Language (NeurIPS 2025 · similarity 0.88)
- The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalities (ICLR 2025 · similarity 0.88)
- PaceLLM: Brain-Inspired Large Language Models for Long-Context Understanding (NeurIPS 2025 · similarity 0.87)
- Far from the Shallow: Brain-Predictive Reasoning Embedding through Residual Disentanglement (NeurIPS 2025 · similarity 0.87)
- How do Large Language Models Handle Multilingualism? (NeurIPS 2024 · similarity 0.87)
- Hierarchical Frequency Tagging Probe (HFTP): A Unified Approach to Investigate Syntactic Structure Representations in Large Language Models and the Human Brain (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.