Research map / Large language models / Language, safety and interpretability
LLM Circuit Analyses Are Consistent Across Training and Scale
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Tracks IOI, Gendered-Pronoun, Greater-Than and SVA circuits across 300B tokens of Pythia training and 70M-2.8B scales, finding task abilities and components emerge at similar token counts and that circuit algorithms are stable over time and scale.
Problem
Most LLM circuit analyses are done on models at the end of pre-training, but deployed models undergo continued training, so it is unclear whether circuit findings persist across training time and model scale.
Approach
Uses attribution-based circuit finding (EAP-IG) at every Pythia checkpoint across model sizes, tracks functional attention-head components (induction, successor, copy suppression, name-mover) and verifies IOI algorithm stability via path patching over 300B tokens.
Key results
- Task performance and key components (induction, name-mover, copy suppression heads) emerge at similar token counts across Pythia scales, with induction heads emerging around 2B tokens.
- IOI algorithm metrics stay high (generally above 50%, copy suppression and name-mover heads above 70%) and stable after circuit formation in Pythia-160m, even though individual heads change.
- Circuit size is positively correlated with model size (Pearson r=0.72 for IOI and SVA, 0.9 for Greater-Than, 0.6 for Gendered-Pronoun).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase training tokens after IOI circuit formation | Pythia-160m IOI circuit · IOI | Circuit algorithm metric stability: 79.11% at 38B Tkn; 76.86% at 300B Tkn | No clear effect |
| Increase model scale | Pythia IOI circuits · IOI | Circuit node count: r=0.72 | Mixed |
| Increase model scale | Pythia Greater-Than circuits · Greater-Than | Circuit node count: r=0.9 | Helped |
| Increase model scale | Pythia Gendered-Pronoun circuits · Gendered-Pronoun | Circuit node count: r=0.6 | Helped |
| Increase model scale | Pythia SVA circuits · SVA | Circuit node count: r=0.72 | Helped |
| Increase model scale beyond the task-specific learning-rate ceiling | Pythia models · Gendered-Pronoun and Greater-Than | Task acquisition rate | No clear effect |
| Scale IOI models from 410M–2.8B to 6.9B or 12B parameters | Pythia IOI models · IOI | Task acquisition rate | Hurt |
| Continue training after functional components emerge | Pythia-160m circuit components · IOI and Greater-Than | Functional component behavior: 4×10^9 tokens; 3×10^10 tokens | Mixed |
| Use larger Pythia models | Pythia circuits · All four tasks | Circuit constituent stability | Helped |
| Use the Pythia-160m IOI algorithm as a reference across model scales | Pythia IOI circuits · IOI | Circuit algorithm similarity: 76.86%; 88.73%; 91.89%; 84.03% | Mixed |
Surprising findings
- Individual attention heads lose their functional roles late in training (e.g., name-mover head (4,6) stops at 3e10 tokens) while task performance does not decrease, suggesting compensating heads.
- Beyond a certain size, additional scale did not speed up task acquisition on several tasks, contrary to some prior scaling findings.
Limitations
- Only simple tasks feasible for small models were studied; more complex tasks may show different trends.
- Only the Pythia model family was analyzed, sharing architecture and training setup across scale.
- Pythia-6.9b and 12b were omitted from circuit finding and component analysis for computational cost.
Experimental setup
Datasets: IOI examples (70) Gendered-Pronoun examples (70) Greater-Than examples (200) SVA synthetic sentences (200) SciQ PIQA ARC Easy WinoGrande OpenWebText
Benchmarks: IOI Gendered-Pronoun Greater-Than Subject-Verb Agreement SciQ PIQA ARC Easy WinoGrande
Metrics: logit difference probability difference circuit faithfulness EWMA Jaccard similarity Pearson correlation of circuit size with model size path patching ratio component functional scores
Builds on: edge attribution patching with integrated gradients path patching Pythia indirect object identification circuit
Closest papers by meaning
- Towards Understanding Fine-Tuning Mechanisms of LLMs via Circuit Analysis (ICML 2025 · similarity 0.89)
- Born a Transformer -- Always a Transformer? On the Effect of Pretraining on Architectural Abilities (NeurIPS 2025 · similarity 0.88)
- Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.88)
- CogBench: a large language model walks into a psychology lab (ICML 2024 · similarity 0.88)
- Predicting Emergent Abilities with Infinite Resolution Evaluation (ICLR 2024 · similarity 0.88)
- Transcoders find interpretable LLM feature circuits (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.