Research map / Architectures and efficiency / RNNs and normalization
Evolution of Concepts in Language Model Pre-Training
Xuyang Ge, Wentao Shu, Jiaxing Wu, Yunhua Zhou, Zhengfu He, Xipeng Qiu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase crosscoder dictionary size (ablation) | crosscoder · Pythia activations | explained variance and L0 | Improved |
| train one crosscoder over snapshots instead of per-snapshot SAEs | sparse dictionary learning · Pythia per-snapshot activations | explained variance, L0 | No clear effect |
| crosscoder instead of SAE on final snapshot | sparse dictionary learning · Pythia final snapshot | Pareto frontier | Improved |
| later emergence step | crosscoder features · Pythia-160M | feature complexity score: Pearson r = 0.309 | Improved |
| ablate top-ranked attributed features (tens of features) (ablation) | Pythia-160M crosscoder · SVA, induction, IOI | task metric | Hurt |
| keep only top-ranked attributed features (ablation) | Pythia-160M crosscoder · SVA, induction, IOI | task metric recovery | Improved |
Closest papers by meaning
- Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scale (NeurIPS 2025 · similarity 0.88)
- Analyze Feature Flow to Enhance Interpretation and Steering in Language Models (ICML 2025 · similarity 0.88)
- PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs (ICLR 2025 · similarity 0.87)
- The Rise and Down of Babel Tower: Investigating the Evolution Process of Multilingual Code Large Language Model (ICLR 2025 · similarity 0.86)
- On Linear Representations and Pretraining Data Frequency in Language Models (ICLR 2025 · similarity 0.86)
- Algorithmic progress in language models (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
