Research map / Architectures and efficiency / Attention and transformers
Inside the Loop: A Mechanistic Study of Weight-Tied Transformers on Depth-Bound Algorithmic Tasks
Magnus Sesodia, Philip Torr, Christian Schroeder de Witt
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Mechanistic study of weight-tied (looped) transformers on depth-bound algorithmic tasks, examining how the reused block performs sequential computation.
Problem
Algorithmic tasks requiring many sequential steps need deep transformers or long chain-of-thought, both costly; it is unclear how looped transformers, which reuse one block across iterations, actually perform this computation.
Approach
Mechanistically analyzes how a weight-tied looped transformer carries out depth-bound algorithmic computation across loop iterations, rather than only measuring end-task accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use a looped block instead of an untied stack at matched effective depth 32 | transformer on DePo N=16 · DePo N=16, lr 1e-3 | seeds solving task: 0/5 -> 5/5 | Improved |
| lower the learning rate of the untied 32-layer model (ablation) | Untied 32-layer · DePo N=16 | seeds solving task: 0/5 -> 4/5 | Improved |
| lower learning rate further to 1e-4 (ablation) | Looped 4x8 and Untied 32-layer · DePo N=16 | seeds solving task: looped 5/5 -> 1/5; untied 4/5 -> 1/5 | Hurt |
| add intermediate next-token supervision at layers 8, 16, 24 (ablation) | Untied 32-layer · DePo N=16, lr 1e-3 | seeds solving task: 0/5 -> 2/5 | Improved |
| increase effective depth (ablation) | looped and untied models · DePo N=16, lr 1e-3 | seeds solving task | Improved |
| split depth as more iterations (8x4, 4x8, 2x16) rather than fewer (16x2) at effective depth 32 (ablation) | looped transformer · DePo N=16, lr 1e-3 | seeds solving task: at least four of five seeds vs never | Improved |
| use looped model on longer chains | Looped 4x8 vs untied 32-layer · DePo N=32, lr 3e-4 | solved-chain depth: H* = 6, 0, 1 vs 1, 1, 1 | Mixed |
| remove iteration embedding (ablation) | Looped 4x8 · DePo N=16, lr 1e-3 | seeds solving task: 3/4 vs 5/5 seeds | No clear effect |
| add auxiliary intermediate loss to untied model (ablation) | Untied 32-layer · DePo N=16 probes | staircase of hop decodability | Improved |
| train with weight tying (looping) | Looped 4x8 · DePo N=16 H=4 probing | intermediate hop decodability | Improved |
| activation patching of donor state after iteration 2 | Looped 4x8, H=6 · DePo N=16 | share of donor answers: reaches 1.00 | Improved |
Closest papers by meaning
- Stability and Generalization in Looped Transformers (NeurIPS 2026 · similarity 0.87)
- Stabilizing Extrapolation in Looped Transformers via Learned Stochastic Stopping (NeurIPS 2026 · similarity 0.86)
- Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning? (ICML 2024 · similarity 0.86)
- Linear Transformers Implicitly Discover Unified Numerical Algorithms (NeurIPS 2025 · similarity 0.86)
- Sparse Layers are Critical to Scaling Looped Language Models (NeurIPS 2026 · similarity 0.86)
- Simulation of Graph Algorithms with Looped Transformers (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
