Research map / Architectures and efficiency / Pruning and efficient tuning

Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA

ICLR 2025 · Architectures and efficiency › Pruning and efficient tuning · approach: Layer-wise pruning of LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Converts pretrained LLMs (Gemma 2B, TinyLlama 1.1B, Pythia 1B) into parameter-shared Recursive Transformers with layer-wise SVD-initialized LoRA ("Relaxed Recursive"), uptrains them, and simulates continuous depth-wise batching with early exits for throughput gains.

Problem

Parameter sharing across layers has yielded limited success in modern LLMs; deploying LLMs is expensive in memory and compute, and naive layer tying degrades performance.

Approach

Initialize a looped block of shared layers from the pretrained model's layers (Stepwise, Average, or Lower methods), then relax tying with per-loop LoRA modules whose weights are initialized by truncated SVD of residuals between original and shared weights; rank tunes the trade-off between recursive and vanilla models. Adds continuous depth-wise batching for serving.

Key results

• Recursive Gemma 1B (2 blocks, Stepwise init) uptrained on 15B tokens reaches 51.7% average few-shot accuracy vs 38.2% for a reduced-size model, outperforming pretrained TinyLlama 1.1B and Pythia 1B.
• Relaxed Recursive Gemma with LoRA rank 512 and 60B tokens with knowledge distillation reaches 58.4% average accuracy, on par with full Gemma 2B (58.6%) trained on 3T tokens.
• Simulated oracle-exit continuous depth-wise batching yields up to 2.66x generation speedup over vanilla counterparts; recursive Gemma reaches nearly 4x throughput over vanilla Pythia.

Measured effects reported in the paper

Surprising findings

• Uptraining full-size Gemma on SlimPajama lowered few-shot accuracy, suggesting dataset distribution shift can hurt pretrained models.
• Average initialization performed poorly for non-relaxed Gemma, while being best for Relaxed models; Stepwise showed U-shaped performance vs LoRA rank.

Limitations

• Throughput results are hypothetical under oracle exiting, not a practical early-exit algorithm.
• Experiments limited to ~1-2B models; scaling to 7B+ is untested.
• Uptraining on SlimPajama can degrade pretrained model performance on some benchmarks, capping achievable accuracy.

Experimental setup

Datasets: SlimPajama RedPajama PG19 Pile

Benchmarks: LAMBADA HellaSwag PIQA WinoGrande ARC-easy ARC-challenge OpenBookQA SlimPajama perplexity RedPajama perplexity PG19 perplexity

Metrics: few-shot accuracy perplexity throughput (simulated tokens/sec)

Baselines: full-size pretrained models (Gemma 2B, TinyLlama 1.1B, Pythia 1B) reduced-size models pretrained from scratch random initialization of recursive models continuous sequence-wise batching

Builds on: Universal Transformer LoRA ALBERT-style cross-layer sharing (CYCLE strategy) depth up-scaling / layer skipping forward KL distillation

Closest papers by meaning

• Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weights (NeurIPS 2025 · similarity 0.90)
• Separations in the Representational Capabilities of Transformers and Recurrent Architectures (NeurIPS 2024 · similarity 0.89)
• DLP: Dynamic Layerwise Pruning in Large Language Models (ICML 2025 · similarity 0.89)
• Skip a Layer or Loop It? Learning Program-of-Layers in LLMs (ICML 2026 · similarity 0.89)
• Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformers (NeurIPS 2025 · similarity 0.89)
• Sparse Layers are Critical to Scaling Looped Language Models (NeurIPS 2026 · similarity 0.89)