Research map / Architectures and efficiency / RNNs and normalization
Scaling Sparse Feature Circuits For Studying In-Context Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses sparse autoencoders and an extended sparse feature circuits method on Gemma 1 2B to find task-execution and task-detection features underlying in-context learning via task vectors.
Problem
Understanding the mechanism behind in-context learning in LLMs, where task vectors are known but not decomposed into interpretable components, and SAEs have not been shown to explain complex end-to-end circuits.
Approach
Introduces Task Vector Cleaning, an optimization that sparsifies SAE decompositions of task vectors while preserving their steering effect, and adapts Sparse Feature Circuits with token-position categorization, a loss modification over all but the first example, and zero ablation to find task-detection and task-execution circuits for ICL.
Key results
- Task Vector Cleaning reduces active SAE features by about 70% while maintaining the task vector's effect on loss for Gemma 1 2B.
- Circuits of about 500 nodes reach average faithfulness of 0.6 across ICL tasks in Gemma 1 2B.
- Ablating task-detection features substantially reduces activation of corresponding task-execution features; executor features activate on arrow tokens (89.8% of mass) and detectors on output tokens (96.8%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Task Vector Cleaning to the SAE decomposition of task vectors | Gemma 1 2B task vectors · Zero-shot task prompts | active SAE feature count: 70% | Helped |
| Apply Task Vector Cleaning across model scales, architectures, and SAE configurations | Task vectors · Zero-shot task prompts | active SAE feature count: 50-80% | Helped |
| Use a 500-node circuit | Modified Sparse Feature Circuits · ICL tasks | faithfulness: 0.6 | Helped |
| Add task-detection feature directions (ablation) | ICL circuit in Gemma 1 2B · ICL tasks | task-execution feature activation | Helped |
| Ablate features found important for one task (ablation) | Extracted ICL task circuits · Other, unrelated ICL tasks | faithfulness | No clear effect |
| Ablate features found important for one task (ablation) | Extracted ICL task circuits · Closely related tasks, including translation tasks | faithfulness | Hurt |
| Use higher target L0 values for SAEs | Task Vector Cleaning · Task vector decomposition | task-loss effect | Helped |
| Use high L0 constraints that drop the shared English-to-foreign translation feature (ablation) | Task Vector Cleaning · French and Italian translation tasks | task-specific steering effect | Hurt |
Surprising findings
- Naive SAE reconstruction of task vectors yields more than 10 non-zero features on average and reduces the task vector's effect on loss, so task vectors are out-of-distribution for SAEs.
- Person profession and present simple gerund tasks show unusually weak detection-execution connections.
Limitations
- Analysis limited to the simple task-vector setting, a subset of ICL.
- Multiple competing execution and detection features were observed, complicating interpretation.
- Circuit search restricted to layers 10-17, with lower-quality IE approximations in earlier layers.
Experimental setup
Datasets: task vector datasets from Todd et al. (2024)
Benchmarks: Todd et al. (2024) ICL task suite
Metrics: relative loss change after steering faithfulness fraction of active SAE features activation mass
Baselines: original task vector SAE reconstruction ITO (inference-time optimization)
Builds on: Sparse Feature Circuits task vectors gated SAE transcoders inference-time optimization
Closest papers by meaning
- Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective (ICML 2025 · similarity 0.90)
- Capturing In-Context Learning Dynamics with Task Operators (NeurIPS 2026 · similarity 0.89)
- Are Sparse Autoencoders Useful? A Case Study in Sparse Probing (ICML 2025 · similarity 0.89)
- Scaling and evaluating sparse autoencoders (ICLR 2025 · similarity 0.89)
- SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability (ICML 2025 · similarity 0.89)
- Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Control (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.