Research map / Large language models / Language, safety and interpretability
Towards Atoms of Large Language Models
Chenhui Hu, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use atomic inner product instead of Euclidean inner product | pairwise representation angle analysis · 128 WikiData entities, all layers of GPT-J, Pythia, Llama, Gemma | centroid of pairwise angles: deviates from 90 deg -> restored to 90 deg | Improved |
| use TSAE atoms instead of SAE features | representational units · CounterFact, Gemma2/Llama3.1 | faithfulness R2: 48.8% -> 99.90% | Improved |
| use TSAE atoms instead of SAE features | representational units · CounterFact, Gemma2/Llama3.1 | stability q*: 68.2% -> 99.85% | Improved |
| use TSAE atoms instead of neurons | representational units · CounterFact, Gemma2/Llama3.1 | stability q*: 0.5% -> 99.85% | Improved |
| raise TSAE capacity above a critical threshold for a given data scale (ablation) | TSAE on Gemma2-2B layer 1 · WikiData activations (1.9B samples) | faithfulness R2 | Improved |
| choose SAE capacity heuristically regardless of data scale | standard SAE training · Gemma2-2B | faithfulness R2 | Hurt |
| vary training hyperparameters of TSAE (ablation) | TSAE atoms · Gemma2/Llama3.1 | faithfulness and stability | No clear effect |
| use atoms instead of SAE features | representational units · Gemma2-2B/9B, Llama3.1-8B | monosemanticity score (LLM judge) | Improved |
Closest papers by meaning
- Sparse Autoencoders Do Not Find Canonical Units of Analysis (ICLR 2025 · similarity 0.86)
- A Polar coordinate system represents syntax in large language models (NeurIPS 2024 · similarity 0.86)
- Remarkable Robustness of LLMs: Stages of Inference? (NeurIPS 2025 · similarity 0.86)
- Correlation and Navigation in the Vocabulary Key Representation Space of Language Models (ICLR 2025 · similarity 0.85)
- LatentLens: Revealing Highly Interpretable Visual Tokens in LLMs (ICML 2026 · similarity 0.85)
- Tracing the Representation Geometry of Language Models from Pretraining to Post-training (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
