Research map / Architectures and efficiency / RNNs and normalization
Step-Resolved Data Attribution for Looped Transformers
Georgios Kaissis, David Mildenberger, Juan Felipe Gomez, Martin J. Menten, Eleni Triantafillou
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use sketch-during-backprop TensorSketch instead of materialising per-example gradients | SDI on looped GPT 135.1M · Single 48GB A6000, m=2048 | feasible batch size: 4 -> 40 | Improved |
| increase sketch dimension m (ablation) | SDI TensorSketch · Looped GPT 135.1M | sketch error scaling: log-log slope -0.489 | Improved |
| scale model width (77M to 1.2B parameters) (ablation) | SDI relative sketch cost · Fixed tau=32 | relative overhead: +116% -> +18% | Improved |
| use SDI step-resolved trajectory instead of scalar TracIn to find a periodic latent circuit | Looped transformer on parity · Parity, length-40 probe | mechanistic proxy accuracy: 100% on held-out alternating inputs; ≈93% on length-40 test | Improved |
| use step-resolved SDI rather than scalar TracIn to expose signal cancellation | Parity pair (20-bit train, 40-bit test) · Parity | influence magnitude: TracIn 0.001 vs sum |SDI| 17.24 | Improved |
| remove top 30% late-step SDI training puzzles (ablation) | Looped transformer on Sudoku · Sudoku hard puzzles, tau=32 | board accuracy: 95.8 -> 91.7 | Hurt |
| remove top 30% late-step SDI training puzzles (ablation) | Looped transformer on Sudoku · Sudoku overall, tau=32 | board accuracy: 98.6 -> 97.8 | Hurt |
| harder training puzzles (more blanks) | Sudoku TracIn self-influence · Sudoku train | median self TracIn: 0.225 -> 0.451 | Improved |
| hard vs easy training puzzles: share of SDI energy in late steps | SDI on Sudoku · Sudoku hard test puzzles | SDI late mass (steps 17-32): 24.3% -> 25.3% | Improved |
| add more test-time loops (ablation) | Looped transformer on Sudoku · Sudoku hard puzzles | board accuracy | Improved |
| add AdamW second-moment preconditioning to the gradient inner product (ablation) | SDI/TracIn on Sudoku · Sudoku ranking stability | Spearman rank correlation: rho 0.979 TracIn, 0.934 SDI | No clear effect |
| recompute SDI with full BPTT instead of truncated BPTT (ablation) | Recursive Nanochat SFT · GSM8K train to test | SDI last-step share: 0.604 at tau=8, 0.606 at tau=16 | Improved |
Closest papers by meaning
- Stability and Generalization in Looped Transformers (NeurIPS 2026 · similarity 0.89)
- Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning? (ICML 2024 · similarity 0.89)
- Stabilizing Extrapolation in Looped Transformers via Learned Stochastic Stopping (NeurIPS 2026 · similarity 0.87)
- LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation (ICLR 2026 · similarity 0.87)
- Transformers Provably Learn Chain-of-Thought Reasoning with Length Generalization (NeurIPS 2025 · similarity 0.87)
- From Shortcut to Induction Head: How Data Diversity Shapes Algorithm Selection in Transformers (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
