Research map / Optimization / Implicit bias and generalization

In-Context Deep Learning via Transformer Models

ICML 2025 · Optimization › Implicit bias and generalization · approach: In-context learning as gradient descent · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that a (2N+4)L-layer ReLU transformer (and a 4L-layer softmax transformer) can implicitly simulate L steps of gradient descent for training an N-layer network via in-context learning, with approximation and convergence guarantees and small synthetic experiments.

Problem

Whether transformers can simulate the training of a deep neural network by gradient descent purely through in-context learning, without parameter updates at inference.

Approach

Derives an explicit gradient-descent expression for N-layer feed-forward networks, then constructs a transformer (with an element-wise multiplication layer) that approximates each GD step term by term using sum-of-ReLU approximations, extending in-context gradient descent from shallow models to deep networks.

Key results

• Constructs a (2N+4)L-layer ReLU transformer that simulates L gradient descent steps of an N-layer ReLU network with approximation error bounds, improving on prior O(N^2 L)-layer constructions.
• Gives a 4L-layer Softmax transformer construction for in-context gradient descent on N-layer networks.
• Experiments on 6-layer networks show ICL R-squared comparable to direct training; the trained transformer does not always match the theoretical construction.

Measured effects reported in the paper

Surprising findings

• The theoretical ICL forward pass needs more FLOPs than direct training, yet the empirically trained transformer reached 6-layer network performance with fewer FLOPs (3.3B vs 7.6B).

Limitations

• Constructed transformer hidden and MLP dimensions are very large, about O(NK^2)+D_w.
• Generalization is limited to varying input and parameter distributions with fixed network width and depth.
• Empirically trained transformers differ from the theoretical construction; softmax construction weights are not explicit.

Experimental setup

Datasets: synthetic Gaussian mixture inputs (d=20) labeled by random 3-, 4-, 6-layer ReLU networks

Benchmarks: synthetic regression on 3-, 4-, 6-layer networks

Metrics: R-squared

Baselines: In-context gradient descent for two-layer networks (prior ICGD construction with Transformer in Transformer (TINT)

Builds on: in-context gradient descent transformers learn in context by gradient descent Transformers in Transformers (TINT)

Closest papers by meaning

• Exact Conversion of In-Context Learning to Model Weights in Linearized-Attention Transformers (ICML 2024 · similarity 0.90)
• How Do Nonlinear Transformers Learn and Generalize in In-Context Learning? (ICML 2024 · similarity 0.90)
• Pretrained Transformer Efficiently Learns Low-Dimensional Target Functions In-Context (NeurIPS 2024 · similarity 0.90)
• Transformers are Minimax Optimal Nonparametric In-Context Learners (NeurIPS 2024 · similarity 0.89)
• In-context Convergence of Transformers (ICML 2024 · similarity 0.89)
• In-context Learning on Function Classes Unveiled for Transformers (ICML 2024 · similarity 0.89)