Research map / Large language models / Language, safety and interpretability
Understanding the Emergence of Seemingly Useless Features in Next-Token Predictors
Mark Rofin, Jalal Naghiyev, Michael Hahn
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| allow pre-caching (non-myopic instead of myopic training) (ablation) | tiny GPT-2 LM · TinyStories v2 | loss: 3.29 -> 2.53 | Improved |
| allow pre-caching (ablation) | tiny GPT-2 LM · TinyStories syntactic probes (POS, dependency) | probe performance | No clear effect |
| allow pre-caching and circuit sharing (ablation) | small Transformer · Majority / Conditioned Majority input-phase features | probe correlation | Improved |
| allow pre-caching (non-myopic) (ablation) | 2-layer Transformer · Conditioned Majority | accuracy: 0.76 -> 0.99 | Improved |
| block pre-caching via myopic training (ablation) | 2-layer Transformer · Majority | accuracy: 1.00 -> 1.00 | No clear effect |
| add direct gradient component for NTP-useful vs NTP-useless Othello cells (ablation) | OthelloGPT · Othello probing | probe performance gap | Improved |
| feature Q(w) (pre-caching degree) vs look-ahead usefulness (ablation) | Gemma 2 2B · look-ahead maps up to 5 tokens | Spearman correlation | Hurt |
| steer high pre-caching features (ablation) | Gemma 2 2B · unconditional generation | code and punctuation content | Improved |
Closest papers by meaning
- Auto-Regressive Next-Token Predictors are Universal Learners (ICML 2024 · similarity 0.88)
- Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction (ICML 2025 · similarity 0.88)
- The Pitfalls of Next-Token Prediction (ICML 2024 · similarity 0.87)
- Emergent Response Planning in LLMs (ICML 2025 · similarity 0.87)
- Looking Beyond the Top-1: Transformers Determine Top Tokens in Order (ICML 2025 · similarity 0.87)
- Emergent Symbolic Mechanisms Support Abstract Reasoning in Large Language Models (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
