Research map / Architectures and efficiency / Pruning and efficient tuning
Trainable Transformer in Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Transformer in Transformer (TinT), a parameter-efficient transformer construction that simulates and fine-tunes a 125M auxiliary transformer (e.g., OPT-125M) within a single forward pass, using prefix-embedding weight storage, stacking/sharding, and approximate (first-order, stop-gradient) backpropagation.
Problem
Prior constructions showing in-context learning as implicit gradient descent require enormous simulator size and only handle tiny auxiliary models (linear or small MLP), so simulating fine-tuning of a realistic pre-trained transformer within a forward pass is intractable.
Approach
Store auxiliary weights in prefix embeddings, stack and shard weights across many attention heads to parallelize linear layers, and use first-order approximations for layer norm and stop-gradients through attention scores and Q/K to make the simulated backward pass cheap, updating only value parameters.
Key results
- On WikiText-103, TinT lowers OPT-125M perplexity by about 0.3-0.7 absolute (e.g., 28.0 to 27.5 at 70% training split), close to explicit fine-tuning (27.4).
- Zero-shot average accuracy over 7 classification tasks rises from 68.6 (OPT-125m) to 72.0 for TinT, versus 73.2 for explicit fine-tuning.
- 32-shot average accuracy rises from 50.5 (OPT-125m) to 76.4 for TinT, exceeding OPT-1.3B (75.3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TinT instead of the untuned auxiliary model | GPT-2 · WikiText-103, 30% training | perplexity: 25.6 -> 25.1 | Helped |
| Use TinT instead of the untuned auxiliary model | GPT-2 · WikiText-103, 50% training | perplexity: 24.9 -> 24.3 | Helped |
| Use TinT instead of the untuned auxiliary model | GPT-2 · WikiText-103, 70% training | perplexity: 24.5 -> 23.8 | Helped |
| Use TinT instead of the untuned auxiliary model | GPT-2 · WikiText-103, 90% training | perplexity: 23.3 -> 22.6 | Helped |
| Use TinT instead of the untuned auxiliary model | OPT-125M · WikiText-103, 30% training | perplexity: 29.6 -> 29.3 | Helped |
| Use TinT instead of the untuned auxiliary model | OPT-125M · WikiText-103, 50% training | perplexity: 28.8 -> 28.4 | Helped |
| Use TinT instead of the untuned auxiliary model | OPT-125M · WikiText-103, 70% training | perplexity: 28.0 -> 27.5 | Helped |
| Use TinT instead of the untuned auxiliary model | OPT-125M · WikiText-103, 90% training | perplexity: 28.0 -> 27.4 | Helped |
| Use TinT instead of the untuned auxiliary model | OPT-125M · 7 classification tasks, zero-shot average | accuracy: 68.6 -> 72.0 | Helped |
| Use TinT instead of the untuned auxiliary model | OPT-125M · 7 classification tasks, 32-shot average | accuracy: 50.5 -> 76.4 | Helped |
| Use TinT rather than explicit fine-tuning | OPT-125M · 7 classification tasks, zero-shot average | accuracy: 73.2 -> 72.0 | Hurt |
| Use TinT rather than explicit fine-tuning | OPT-125M · 7 classification tasks, 32-shot average | accuracy: 74.6 -> 76.4 | Helped |
Surprising findings
- A TinT simulator built on OPT-125M reaches 32-shot accuracy comparable to or above the larger OPT-1.3B, suggesting larger models may implement subroutines of smaller ones.
- Approximations (stop-gradient on attention scores and Q/K, first-order layer norm gradient, updating only value weights) barely hurt performance relative to explicit fine-tuning.
Limitations
- Number of simulated gradient steps limited to 3 or fewer because each step adds depth to the simulator.
- Approximate backpropagation ignores query/key gradients and updates only value parameters; exact simulation would be much larger.
- Insights into existing pre-trained models are limited; possible biases from one-step gradient descent not examined.
Experimental setup
Datasets: WikiText-103 SST-2 MR CR MPQA Amazon Polarity AGNews Subj
Benchmarks: WikiText-103 SST-2 MR CR MPQA Amazon Polarity AGNews Subj
Metrics: perplexity classification accuracy
Baselines: OPT-125M (auxiliary model) explicit fine-tuning of OPT-125M OPT-1.3B dynamic evaluation
Builds on: in-context learning as gradient descent constructions GeLU multiplication trick LoRA dynamic evaluation
Closest papers by meaning
- How Do Nonlinear Transformers Learn and Generalize in In-Context Learning? (ICML 2024 · similarity 0.91)
- Understanding Transformers via N-Gram Statistics (NeurIPS 2024 · similarity 0.90)
- Do Efficient Transformers Really Save Computation? (ICML 2024 · similarity 0.89)
- Bigram Subnetworks: Mapping to Next Tokens in Transformer Language Models (NeurIPS 2025 · similarity 0.89)
- Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models (NeurIPS 2024 · similarity 0.88)
- Understanding the Generalization of In-Context Learning in Transformers: An Empirical Study (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.