AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Inside the Loop: A Mechanistic Study of Weight-Tied Transformers on Depth-Bound Algorithmic Tasks

Magnus Sesodia, Philip Torr, Christian Schroeder de Witt

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Transformer expressivity and learnability · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Mechanistic study of weight-tied (looped) transformers on depth-bound algorithmic tasks, examining how the reused block performs sequential computation.

Problem

Algorithmic tasks requiring many sequential steps need deep transformers or long chain-of-thought, both costly; it is unclear how looped transformers, which reuse one block across iterations, actually perform this computation.

Approach

Mechanistically analyzes how a weight-tied looped transformer carries out depth-bound algorithmic computation across loop iterations, rather than only measuring end-task accuracy.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use a looped block instead of an untied stack at matched effective depth 32transformer on DePo N=16 · DePo N=16, lr 1e-3seeds solving task: 0/5 -> 5/5Improved
lower the learning rate of the untied 32-layer model (ablation)Untied 32-layer · DePo N=16seeds solving task: 0/5 -> 4/5Improved
lower learning rate further to 1e-4 (ablation)Looped 4x8 and Untied 32-layer · DePo N=16seeds solving task: looped 5/5 -> 1/5; untied 4/5 -> 1/5Hurt
add intermediate next-token supervision at layers 8, 16, 24 (ablation)Untied 32-layer · DePo N=16, lr 1e-3seeds solving task: 0/5 -> 2/5Improved
increase effective depth (ablation)looped and untied models · DePo N=16, lr 1e-3seeds solving taskImproved
split depth as more iterations (8x4, 4x8, 2x16) rather than fewer (16x2) at effective depth 32 (ablation)looped transformer · DePo N=16, lr 1e-3seeds solving task: at least four of five seeds vs neverImproved
use looped model on longer chainsLooped 4x8 vs untied 32-layer · DePo N=32, lr 3e-4solved-chain depth: H* = 6, 0, 1 vs 1, 1, 1Mixed
remove iteration embedding (ablation)Looped 4x8 · DePo N=16, lr 1e-3seeds solving task: 3/4 vs 5/5 seedsNo clear effect
add auxiliary intermediate loss to untied model (ablation)Untied 32-layer · DePo N=16 probesstaircase of hop decodabilityImproved
train with weight tying (looping)Looped 4x8 · DePo N=16 H=4 probingintermediate hop decodabilityImproved
activation patching of donor state after iteration 2Looped 4x8, H=6 · DePo N=16share of donor answers: reaches 1.00Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy