Research map / Architectures and efficiency / RNNs and normalization
LAuReL: Learned Augmented Residual Layer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces LAuReL, a learned generalization of the residual connection (residual weights, low-rank, and previous-activation variants) that replaces the standard skip connection with few extra parameters, tested on ResNet-50/ImageNet and 1B/4B LLM pretraining.
Problem
Residual connections are ubiquitous but fixed; improving model quality usually requires scaling parameters or layers, which raises training and inference cost. The paper asks whether a cheap learned residual can improve quality and footprint.
Approach
Generalize x_{i+1} = f(x_i) + x_i into x_{i+1} = alpha*f(x_i) + g(x_i,...,x_0), with learned scalars (RW), a learned low-rank linear map (LR, W = AB + I), and learned weighted mixing of previous k activations (PA), so the residual stream is augmented with learned components at tiny parameter cost.
Key results
- On ResNet-50/ImageNet-1K, LAuReL (RW+LR, r=16) matches the naive add-a-layer baseline (75.20%) with 2.6x fewer extra parameters (1.68% vs 4.37%).
- On LLM-1B, LAuReL adds 0.012% parameters and improves most downstream tasks, e.g. BoolQ +13.07% relative and GSM8K-PAL +10.28% relative.
- On LLM-4B, LAuReL (r=64) adds ~0.1% parameters and ~1-2% latency, improving BookQA by +20.05% and MGSM by +15.45% relative over baseline, while the 4.56B naive-scaling model is outperformed on most tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the standard skip connection with LAuReL residual weights | ResNet-50 · ImageNet-1K | accuracy@1: 74.95 ± 0.01 -> 75.10 ± 0.10 | Helped |
| Add low-rank residual transform (r=16) to learned residual weights | ResNet-50 · ImageNet-1K | accuracy@1: 75.10 ± 0.10 -> 75.20 ± 0.07 | Helped |
| Add previous activations to learned residual weights and low-rank transform | ResNet-50 · ImageNet-1K | accuracy@1: 75.20 ± 0.07 -> 75.25 ± 0.09 | Helped |
| Use LAuReL residual weights + low rank (r=4) | LLM-1B · GSM8K-CoT | score: 8.34 -> 8.79 | Helped |
| Use LAuReL residual weights + low rank (r=4) | LLM-1B · MBPP | score: 27.00 -> 27.00 | No clear effect |
| Use LAuReL residual weights + low rank (r=64) | LLM-4B · WMT23 | score: 68.32 -> 68.24 | No clear effect |
| Use LAuReL residual weights + low rank (r=64) | LLM-4B · DocVQA | score: 68.28 -> 68.34 | No clear effect |
| Use residual weights only instead of standard residual connections (ablation) | 24-layer small LLM · C4 test | test loss: 3.0159 -> 2.9557 | Helped |
| Add a low-rank residual transform only (r=32) (ablation) | 24-layer small LLM · C4 test | test loss: 3.0159 -> 2.9624 | Helped |
| Add previous activations (k=3) (ablation) | 24-layer small LLM · C4 test | test loss: 3.0159 -> 2.9512 | Helped |
| Combine residual weights with low rank (r=32) (ablation) | 24-layer small LLM · C4 test | test loss: 3.0159 -> 2.9531 | Helped |
| Combine residual weights, low rank (r=32), and previous activations (k=3) (ablation) | 24-layer small LLM · C4 test | test loss: 3.0159 -> 2.9499 | Helped |
Surprising findings
- Adding a naive extra layer to ResNet-50 gave only +0.25% accuracy, and the 40-layer naive scaling baseline did not reliably improve downstream quality over the 40-layer baseline.
- The low-rank variant showed a unimodal dependence on rank r, with gains peaking around r in {16, 32} and dropping for larger r.
Limitations
- Downstream LLM gains are reported from single pre-training runs with limited statistical detail; the PA variant was not tested on the LLMs due to cost.
- Low-rank initialization and rank choice significantly affect results and lack a principled scheme.
- Experiments are limited to a ResNet-50 for one epoch and LLMs trained on proprietary data using TPUs; results are not compared against standard open baselines.
Experimental setup
Datasets: ImageNet-1K C4 LLM pre-training mixture (text, code, translations, webpages, books) multimodal and multilingual pre-training mixture (LLM-4B)
Benchmarks: ImageNet-1K MMLU GSM8K MATH BoolQ TyDi QA Hellaswag HumanEval MBPP MGSM Belebele BookQA
Metrics: accuracy@1 test loss accuracy parameter count step time peak memory latency
Baselines: Baseline + 1 Layer (naive scaling) Baseline-41 (naive scaling LLM) Baseline-28 ResNet-50 baseline DenseNet Highway Networks
Builds on: residual connection LoRA DenseNet Highway Networks
Closest papers by meaning
- Revisiting Residual Connections: Orthogonal Updates for Stable and Efficient Deep Networks (NeurIPS 2025 · similarity 0.87)
- Deep Residual Learning for Image Recognition (CVPR 2016 · similarity 0.87)
- Residual Alignment: Uncovering the Mechanisms of Residual Networks (NeurIPS 2023 · similarity 0.87)
- Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling (ICLR 2023 · similarity 0.87)
- Neural Residual Diffusion Models for Deep Scalable Vision Generation (NeurIPS 2024 · similarity 0.85)
- LEMON: Lossless model expansion (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.