Research map / Architectures and efficiency / Pruning and efficient tuning

Dynamic Layer Tying for Parameter-Efficient Transformers

ICLR 2024 · Architectures and efficiency › Pruning and efficient tuning · approach: Efficient ViT adaptation · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Uses a deep Q-learning agent that, during a single training run, dynamically decides for each transformer layer whether to train independently or copy and tie weights from an earlier layer, cutting trainable parameters and memory while matching or slightly beating perplexity.

Problem

Transformers are overparameterized, and training large transformers requires many trainable parameters and high memory; prior NAS and weight-sharing methods use fixed patterns or multiple training sessions.

Approach

An RL (deep Q-learning, epsilon-greedy) driver network periodically chooses, during training of the primary transformer, which earlier layer each layer should copy and tie to, yielding a dynamic single-pass NAS with weight tying rather than fixed sharing patterns.

Key results

• On GPT-2, perplexity improves on Wiki-2 (53.57 to 49.37) and 1-billion words (88.35 to 72.35) with mean trainable parameters of 151M-218M versus 1.6B for conventional training.
• Peak training memory for GPT-2 drops from 12,566.66 MB to 4,514.31 MB (about 65% lower), with average memory down 68%.
• On ViT for CIFAR-10, accuracy is 0.995 versus 0.999 for the baseline while using about 22% of parameters (139M end vs 630M).

Measured effects reported in the paper

Surprising findings

• Layer 0 dominates as a replication source throughout training, and the learned topology keeps only a small number of independent layers, with the Pearson correlation of untied layers' FFN weights at least 0.93.
• A static transformer with the same final weight-tying structure, trained from scratch, performs far worse than the full dynamic method, suggesting the method does not find good lottery tickets.

Limitations

• Evaluation limited to language modeling and a preliminary ViT/CIFAR-10 experiment; computer vision and downstream results are only preliminary.
• Training time per epoch is mixed, with the method slower on GPT-2 experiments.
• Some results are marginal or mixed, e.g., conventional training outperforms on Wiki-103 and downstream GLUE results show drops on some tasks.

Experimental setup

Datasets: WikiText-2 WikiText-103 LAMBADA 1-billion words Shakespeare CIFAR-10 GLUE

Benchmarks: WikiText-2 WikiText-103 LAMBADA 1-billion words Shakespeare CIFAR-10 GLUE (SST-2, CoLA, QNLI, MRPC, RTE)

Metrics: perplexity accuracy number of trainable parameters number of independent layers peak and average GPU memory training time per epoch

Baselines: Conventional training Cycle weight-sharing pattern Cycle Rev Sequence pattern Vision Transformer (ViT)

Builds on: DQN Cyclic layer sharing (Reuse/Subformer-style fixed patterns) Bellman equation Q-learning

Closest papers by meaning

• Transformers Provably Implement In-Context Reinforcement Learning with Policy Improvement (NeurIPS 2026 · similarity 0.86)
• Peri-LN: Revisiting Normalization Layer in the Transformer Architecture (ICML 2025 · similarity 0.86)
• Infinite Limits of Multi-head Transformer Dynamics (NeurIPS 2024 · similarity 0.86)
• Small-scale proxies for large-scale Transformer training instabilities (ICLR 2024 · similarity 0.86)
• Simplifying Transformer Blocks (ICLR 2024 · similarity 0.86)
• An Evolved Universal Transformer Memory (ICLR 2025 · similarity 0.85)